| 2024-10-23 19:53:32,022 INFO MainThread:3156467 [wandb_setup.py:_flush():79] Current SDK version is 0.18.5 | |
| 2024-10-23 19:53:32,022 INFO MainThread:3156467 [wandb_setup.py:_flush():79] Configure stats pid to 3156467 | |
| 2024-10-23 19:53:32,022 INFO MainThread:3156467 [wandb_setup.py:_flush():79] Loading settings from /home/align-anything/.config/wandb/settings | |
| 2024-10-23 19:53:32,022 INFO MainThread:3156467 [wandb_setup.py:_flush():79] Loading settings from /home/align-anything/align-anything/local_scripts/wandb/settings | |
| 2024-10-23 19:53:32,022 INFO MainThread:3156467 [wandb_setup.py:_flush():79] Loading settings from environment variables: {'api_key': '***REDACTED***'} | |
| 2024-10-23 19:53:32,022 INFO MainThread:3156467 [wandb_setup.py:_flush():79] Applying setup settings: {'mode': None, '_disable_service': None} | |
| 2024-10-23 19:53:32,022 WARNING MainThread:3156467 [wandb_setup.py:_flush():79] Could not find program at -m align_anything.trainers.text_image_to_text.rm | |
| 2024-10-23 19:53:32,022 INFO MainThread:3156467 [wandb_setup.py:_flush():79] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m align_anything.trainers.text_image_to_text.rm'} | |
| 2024-10-23 19:53:32,022 INFO MainThread:3156467 [wandb_setup.py:_flush():79] Applying login settings: {} | |
| 2024-10-23 19:53:32,023 INFO MainThread:3156467 [wandb_init.py:_log_setup():534] Logging user logs to /data/align-anything/jiayi/ti2t/rm_10_22_ours_13b_2000/wandb/run-20241023_195332-5p9x2ymn/logs/debug.log | |
| 2024-10-23 19:53:32,023 INFO MainThread:3156467 [wandb_init.py:_log_setup():535] Logging internal logs to /data/align-anything/jiayi/ti2t/rm_10_22_ours_13b_2000/wandb/run-20241023_195332-5p9x2ymn/logs/debug-internal.log | |
| 2024-10-23 19:53:32,023 INFO MainThread:3156467 [wandb_init.py:init():621] calling init triggers | |
| 2024-10-23 19:53:32,023 INFO MainThread:3156467 [wandb_init.py:init():628] wandb.init called with sweep_config: {} | |
| config: {'train_cfgs': {'ds_cfgs': 'ds_z3_config.json', 'epochs': 5.0, 'seed': 42, 'per_device_train_batch_size': 4, 'per_device_eval_batch_size': 4, 'gradient_accumulation_steps': 1, 'gradient_checkpointing': True, 'learning_rate': 3e-05, 'lr_scheduler_type': 'constant_with_warmup', 'lr_warmup_ratio': 0.03, 'weight_decay': 0.0, 'adam_betas': [0.9, 0.95], 'bf16': True, 'fp16': False, 'eval_strategy': 'epoch', 'eval_interval': 10, 'regularization': 0.001, 'freeze_mm_proj': False, 'freeze_vision_tower': True, 'freeze_language_model': False}, 'data_cfgs': {'train_datasets': '/data/align-anything/jiayi/ti2t/refine_gen_wo_res/1021_13b_2000', 'train_template': 'AA_TI2T_Critique', 'train_name': None, 'train_size': None, 'train_split': 'train', 'train_subset': None, 'train_data_files': None, 'train_optional_args': [], 'eval_datasets': '/home/align-anything/datasets/40k_ti2t_filtered', 'eval_template': 'AA_TI2T', 'eval_name': None, 'eval_size': None, 'eval_split': 'train', 'eval_subset': None, 'eval_data_files': None, 'eval_optional_args': []}, 'logger_cfgs': {'log_type': 'wandb', 'log_project': 'align-anything', 'log_run_name': 'rm', 'output_dir': '/data/align-anything/jiayi/ti2t/rm_10_22_ours_13b_2000', 'cache_dir': None, 'save_interval': 598.0}, 'model_cfgs': {'model_name_or_path': '/data/models/llava/llava-1.5-13b-hf', 'trust_remote_code': True, 'model_max_length': 2048}, 'special_tokens': None} | |
| 2024-10-23 19:53:32,023 INFO MainThread:3156467 [wandb_init.py:init():671] starting backend | |
| 2024-10-23 19:53:32,023 INFO MainThread:3156467 [wandb_init.py:init():675] sending inform_init request | |
| 2024-10-23 19:53:32,027 INFO MainThread:3156467 [backend.py:_multiprocessing_setup():104] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |
| 2024-10-23 19:53:32,027 INFO MainThread:3156467 [wandb_init.py:init():688] backend started and connected | |
| 2024-10-23 19:53:32,030 INFO MainThread:3156467 [wandb_init.py:init():783] updated telemetry | |
| 2024-10-23 19:53:32,036 INFO MainThread:3156467 [wandb_init.py:init():816] communicating run to backend with 90.0 second timeout | |
| 2024-10-23 19:53:33,024 INFO MainThread:3156467 [wandb_init.py:init():867] starting run threads in backend | |
| 2024-10-23 19:53:33,160 INFO MainThread:3156467 [wandb_run.py:_console_start():2463] atexit reg | |
| 2024-10-23 19:53:33,160 INFO MainThread:3156467 [wandb_run.py:_redirect():2311] redirect: wrap_raw | |
| 2024-10-23 19:53:33,160 INFO MainThread:3156467 [wandb_run.py:_redirect():2376] Wrapping output streams. | |
| 2024-10-23 19:53:33,160 INFO MainThread:3156467 [wandb_run.py:_redirect():2401] Redirects installed. | |
| 2024-10-23 19:53:33,164 INFO MainThread:3156467 [wandb_init.py:init():911] run started, returning control to user process | |
| 2024-10-24 06:50:25,922 INFO MainThread:3156467 [wandb_run.py:_finish():2158] finishing run 524339208/align-anything/5p9x2ymn | |
| 2024-10-24 06:50:25,925 INFO MainThread:3156467 [wandb_run.py:_atexit_cleanup():2426] got exitcode: 0 | |
| 2024-10-24 06:50:25,926 INFO MainThread:3156467 [wandb_run.py:_restore():2408] restore | |
| 2024-10-24 06:50:25,926 INFO MainThread:3156467 [wandb_run.py:_restore():2414] restore done | |
| 2024-10-24 06:50:29,011 INFO MainThread:3156467 [wandb_run.py:_footer_history_summary_info():3975] rendering history | |
| 2024-10-24 06:50:29,013 INFO MainThread:3156467 [wandb_run.py:_footer_history_summary_info():4007] rendering summary | |
| 2024-10-24 06:50:29,023 INFO MainThread:3156467 [wandb_run.py:_footer_sync_info():3934] logging synced files | |