2024-04-13 09:40:54,416 INFO StreamThr :113 [internal.py:wandb_internal():86] W&B internal server running at pid: 113, started at: 2024-04-13 09:40:54.416039 2024-04-13 09:40:54,420 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status 2024-04-13 09:40:54,846 INFO WriterThread:113 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/run-v5dwwo6y.wandb 2024-04-13 09:40:54,847 DEBUG SenderThread:113 [sender.py:send():379] send: header 2024-04-13 09:40:54,850 DEBUG SenderThread:113 [sender.py:send():379] send: run 2024-04-13 09:40:55,004 INFO SenderThread:113 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files 2024-04-13 09:40:55,005 INFO SenderThread:113 [sender.py:_start_run_threads():1124] run started: v5dwwo6y with start time 1713001254.41705 2024-04-13 09:40:55,012 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: check_version 2024-04-13 09:40:55,012 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: check_version 2024-04-13 09:40:55,101 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: run_start 2024-04-13 09:40:55,112 DEBUG HandlerThread:113 [system_info.py:__init__():26] System info init 2024-04-13 09:40:55,113 DEBUG HandlerThread:113 [system_info.py:__init__():41] System info init done 2024-04-13 09:40:55,113 INFO HandlerThread:113 [system_monitor.py:start():194] Starting system monitor 2024-04-13 09:40:55,113 INFO SystemMonitor:113 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-13 09:40:55,113 INFO SystemMonitor:113 [interfaces.py:start():190] Started cpu monitoring 2024-04-13 09:40:55,114 INFO HandlerThread:113 [system_monitor.py:probe():214] Collecting system info 2024-04-13 09:40:55,114 INFO SystemMonitor:113 [interfaces.py:start():190] Started disk monitoring 2024-04-13 09:40:55,116 INFO SystemMonitor:113 [interfaces.py:start():190] Started gpu monitoring 2024-04-13 09:40:55,117 INFO SystemMonitor:113 [interfaces.py:start():190] Started memory monitoring 2024-04-13 09:40:55,118 INFO SystemMonitor:113 [interfaces.py:start():190] Started network monitoring 2024-04-13 09:40:55,135 DEBUG HandlerThread:113 [system_info.py:probe():150] Probing system 2024-04-13 09:40:55,137 DEBUG HandlerThread:113 [gitlib.py:_init_repo():56] git repository is invalid 2024-04-13 09:40:55,137 DEBUG HandlerThread:113 [system_info.py:probe():198] Probing system done 2024-04-13 09:40:55,137 DEBUG HandlerThread:113 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-04-13T09:40:55.135621', 'startedAt': '2024-04-13T09:40:54.408231', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': '05fce47c7cf1', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.142, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.142, 'min': 0.0, 'max': 0.0}, {'current': 2000.142, 'min': 0.0, 'max': 0.0}, {'current': 2000.142, 'min': 0.0, 'max': 0.0}, {'current': 2000.142, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5567.563323974609}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357559204101562}} 2024-04-13 09:40:55,138 INFO HandlerThread:113 [system_monitor.py:probe():224] Finished collecting system info 2024-04-13 09:40:55,138 INFO HandlerThread:113 [system_monitor.py:probe():227] Publishing system info 2024-04-13 09:40:55,138 DEBUG HandlerThread:113 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment 2024-04-13 09:40:56,007 INFO Thread-12 :113 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/conda-environment.yaml 2024-04-13 09:41:10,152 ERROR HandlerThread:113 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda subprocess.call( File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call return p.wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait return self._wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait raise TimeoutExpired(self.args, timeout) subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds 2024-04-13 09:41:10,155 DEBUG HandlerThread:113 [system_info.py:_save_conda():222] Saving conda packages done 2024-04-13 09:41:10,155 INFO HandlerThread:113 [system_monitor.py:probe():229] Finished publishing system info 2024-04-13 09:41:10,162 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:41:10,162 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: keepalive 2024-04-13 09:41:10,162 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:41:10,162 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: keepalive 2024-04-13 09:41:10,162 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:41:10,163 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: keepalive 2024-04-13 09:41:10,163 DEBUG SenderThread:113 [sender.py:send():379] send: files 2024-04-13 09:41:10,163 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now 2024-04-13 09:41:10,415 INFO wandb-upload_0:113 [upload_job.py:push():131] Uploaded file /tmp/tmpfmmm50lgwandb/57s7cabc-wandb-metadata.json 2024-04-13 09:41:11,010 INFO Thread-12 :113 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-metadata.json 2024-04-13 09:41:11,225 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: python_packages 2024-04-13 09:41:11,226 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: python_packages 2024-04-13 09:41:11,229 DEBUG SenderThread:113 [sender.py:send():379] send: telemetry 2024-04-13 09:41:11,240 DEBUG SenderThread:113 [sender.py:send():379] send: config 2024-04-13 09:41:11,242 DEBUG SenderThread:113 [sender.py:send():379] send: metric 2024-04-13 09:41:11,242 DEBUG SenderThread:113 [sender.py:send():379] send: telemetry 2024-04-13 09:41:11,242 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:41:11,243 DEBUG SenderThread:113 [sender.py:send():379] send: metric 2024-04-13 09:41:11,243 WARNING SenderThread:113 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen) 2024-04-13 09:41:11,243 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:41:11,245 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:41:11,278 DEBUG SenderThread:113 [sender.py:send():379] send: telemetry 2024-04-13 09:41:12,011 INFO Thread-12 :113 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/requirements.txt 2024-04-13 09:41:12,011 INFO Thread-12 :113 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 09:41:14,012 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 09:41:15,860 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:41:20,862 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:41:25,868 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:41:26,017 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/config.yaml 2024-04-13 09:41:26,226 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:41:26,227 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:41:26,227 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:41:31,315 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:41:36,316 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:41:41,227 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:41:41,227 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:41:41,267 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:41:41,329 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:41:46,331 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:41:51,332 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:41:55,118 DEBUG SystemMonitor:113 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-13 09:41:55,120 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:41:56,228 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:41:56,229 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:41:56,229 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:41:56,363 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:42:01,364 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:42:06,365 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:42:11,229 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:42:11,230 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:42:11,230 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:42:12,308 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:42:17,309 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:42:22,311 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:42:25,121 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:42:26,227 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:42:26,228 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:42:26,231 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:42:28,261 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:42:28,481 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:42:28,484 DEBUG SenderThread:113 [sender.py:send():379] send: metric 2024-04-13 09:42:28,484 DEBUG SenderThread:113 [sender.py:send():379] send: metric 2024-04-13 09:42:28,484 DEBUG SenderThread:113 [sender.py:send():379] send: metric 2024-04-13 09:42:28,484 DEBUG SenderThread:113 [sender.py:send():379] send: metric 2024-04-13 09:42:28,484 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:42:28,484 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:42:28,485 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:42:29,045 INFO Thread-12 :113 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:42:33,486 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:42:37,635 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:42:37,637 DEBUG SenderThread:113 [sender.py:send():379] send: metric 2024-04-13 09:42:37,637 DEBUG SenderThread:113 [sender.py:send():379] send: metric 2024-04-13 09:42:37,637 DEBUG SenderThread:113 [sender.py:send():379] send: metric 2024-04-13 09:42:37,637 DEBUG SenderThread:113 [sender.py:send():379] send: metric 2024-04-13 09:42:37,637 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:42:37,637 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:42:37,640 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:42:38,049 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:42:38,884 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:42:40,051 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 09:42:41,228 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:42:41,228 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:42:41,231 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:42:44,308 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:42:49,309 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:42:54,310 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:42:55,122 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:42:56,228 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:42:56,228 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:42:56,268 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:43:00,297 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:43:01,059 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/config.yaml 2024-04-13 09:43:05,407 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:43:10,408 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:43:11,231 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:43:11,231 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:43:11,231 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:43:16,290 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:43:21,291 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:43:25,123 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:43:26,229 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:43:26,229 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:43:26,230 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:43:26,329 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:43:31,331 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:43:36,332 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:43:41,229 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:43:41,229 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:43:41,269 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:43:41,350 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:43:46,351 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:43:51,352 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:43:55,124 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:43:56,228 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:43:56,229 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:43:56,269 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:43:57,116 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:43:57,118 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:43:57,118 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:43:57,119 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:43:57,119 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:43:58,085 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:44:02,121 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:44:06,331 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:44:06,333 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:44:06,333 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:44:06,335 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:44:07,089 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:44:07,574 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:44:08,089 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 09:44:11,229 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:44:11,230 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:44:11,232 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:44:13,349 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:44:18,351 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:44:23,351 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:44:25,125 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:44:26,230 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:44:26,230 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:44:26,252 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:44:29,253 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:44:34,254 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:44:39,255 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:44:41,229 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:44:41,230 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:44:41,269 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:44:44,289 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:44:49,290 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:44:54,291 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:44:55,126 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:44:56,229 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:44:56,229 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:44:56,264 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:45:00,265 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:45:05,266 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:45:10,267 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:45:11,230 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:45:11,231 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:45:11,231 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:45:15,378 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:45:20,379 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:45:25,127 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:45:26,128 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:45:26,174 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:45:26,175 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:45:26,176 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:45:26,176 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:45:26,229 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:45:26,230 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:45:26,233 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:45:27,125 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:45:31,305 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:45:35,395 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:45:35,396 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:45:35,397 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:45:35,399 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:45:36,130 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:45:36,643 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:45:37,130 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 09:45:41,230 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:45:41,231 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:45:41,231 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:45:42,362 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:45:47,363 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:45:52,364 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:45:55,128 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:45:56,230 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:45:56,230 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:45:56,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:45:58,273 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:46:03,274 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:46:08,275 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:46:11,230 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:46:11,231 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:46:11,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:46:13,289 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:46:18,290 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:46:23,291 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:46:25,131 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:46:26,231 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:46:26,231 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:46:26,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:46:28,330 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:46:33,331 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:46:38,332 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:46:41,230 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:46:41,231 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:46:41,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:46:44,318 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:46:49,320 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:46:54,321 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:46:55,012 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:46:55,013 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:46:55,013 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:46:55,014 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:46:55,131 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:46:55,163 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:46:56,230 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:46:56,231 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:46:56,234 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:46:59,337 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:47:04,249 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:47:04,251 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:47:04,251 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:47:04,252 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:47:04,494 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:47:05,167 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:47:07,168 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 09:47:09,495 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:47:11,230 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:47:11,231 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:47:11,234 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:47:15,281 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:47:20,282 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:47:25,132 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:47:26,132 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:47:26,230 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:47:26,231 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:47:26,252 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:47:31,254 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:47:36,256 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:47:41,231 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:47:41,232 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:47:41,261 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:47:41,262 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:47:46,264 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:47:51,265 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:47:55,132 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:47:56,230 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:47:56,231 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:47:56,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:47:56,338 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:48:01,339 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:48:06,339 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:48:11,231 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:48:11,231 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:48:11,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:48:12,305 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:48:17,306 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:48:22,307 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:48:23,959 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:48:23,960 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:48:23,960 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:48:23,961 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:48:24,201 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:48:25,134 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:48:26,231 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:48:26,232 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:48:26,237 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:48:28,277 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:48:33,197 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:48:33,199 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:48:33,199 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:48:33,199 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:48:33,205 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:48:33,435 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:48:35,206 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 09:48:38,436 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:48:41,231 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:48:41,232 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:48:41,232 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:48:44,260 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:48:49,261 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:48:54,262 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:48:55,134 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:48:56,231 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:48:56,232 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:48:56,269 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:48:59,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:49:04,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:49:09,273 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:49:11,232 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:49:11,232 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:49:11,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:49:14,335 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:49:19,336 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:49:24,337 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:49:25,136 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:49:26,232 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:49:26,232 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:49:26,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:49:29,344 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:49:34,345 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:49:39,346 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:49:41,232 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:49:41,233 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:49:41,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:49:45,314 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:49:50,315 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:49:52,235 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:49:52,237 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:49:52,237 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:49:52,237 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:49:52,241 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:49:55,137 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:49:56,138 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:49:56,232 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:49:56,233 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:49:56,235 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:50:01,278 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:50:01,463 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:50:01,465 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:50:01,465 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:50:01,466 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:50:02,245 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:50:03,245 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 09:50:06,700 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:50:11,232 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:50:11,233 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:50:11,233 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:50:12,274 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:50:17,275 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:50:22,276 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:50:25,138 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:50:26,232 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:50:26,233 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:50:26,273 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:50:27,322 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:50:32,323 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:50:37,324 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:50:41,232 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:50:41,233 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:50:41,273 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:50:43,294 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:50:48,295 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:50:53,296 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:50:55,139 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:50:56,233 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:50:56,233 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:50:56,273 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:50:58,305 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:51:03,306 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:51:08,307 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:51:11,233 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:51:11,234 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:51:11,274 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:51:13,316 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:51:18,317 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:51:20,713 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:51:20,714 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:51:20,714 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:51:20,715 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:51:21,281 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:51:23,716 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:51:25,140 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:51:26,233 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:51:26,234 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:51:26,238 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:51:29,298 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:51:29,947 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:51:29,948 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:51:29,948 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:51:29,949 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:51:30,285 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:51:33,287 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 09:51:35,187 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:51:40,188 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:51:41,233 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:51:41,234 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:51:41,236 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:51:45,285 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:51:50,285 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:51:55,146 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:51:56,147 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:51:56,233 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:51:56,234 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:51:56,274 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:52:01,366 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:52:06,367 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:52:11,234 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:52:11,234 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:52:11,274 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:52:12,294 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:52:17,296 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:52:22,296 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:52:25,143 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:52:26,234 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:52:26,234 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:52:26,275 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:52:28,285 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:52:33,286 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:52:38,287 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:52:41,234 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:52:41,234 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:52:41,274 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:52:43,343 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:52:48,344 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:52:49,541 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:52:49,543 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:52:49,543 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:52:49,545 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:52:50,320 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:52:53,546 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:52:55,143 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:52:56,234 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:52:56,235 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:52:56,237 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:52:58,755 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:52:58,756 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:52:58,756 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:52:58,757 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:52:58,757 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:52:59,323 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:53:01,324 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 09:53:03,979 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:53:08,981 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:53:11,234 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:53:11,235 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:53:11,238 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:53:14,354 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:53:19,355 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:53:24,356 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:53:25,145 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:53:26,234 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:53:26,235 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:53:26,275 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:53:30,343 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:53:35,344 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:53:40,345 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:53:41,234 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:53:41,235 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:53:41,275 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:53:46,306 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:53:51,307 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:53:55,146 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:53:56,235 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:53:56,236 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:53:56,276 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:53:57,277 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:54:02,278 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:54:07,279 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:54:11,235 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:54:11,235 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:54:11,275 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:54:12,356 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:54:17,357 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:54:18,063 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:54:18,064 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:54:18,065 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:54:18,065 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:54:18,355 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:54:23,066 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:54:25,147 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:54:26,235 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:54:26,235 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:54:26,239 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:54:27,299 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:54:27,300 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:54:27,300 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:54:27,301 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:54:27,358 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:54:28,687 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:54:29,358 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 09:54:33,688 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:54:38,689 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:54:41,235 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:54:41,236 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:54:41,239 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:54:44,264 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:54:49,265 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:54:54,265 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:54:55,148 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:54:56,235 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:54:56,236 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:54:56,276 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:54:59,300 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:55:04,301 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:55:09,302 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:55:11,235 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:55:11,236 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:55:11,276 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:55:14,342 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:55:19,343 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:55:24,344 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:55:25,149 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:55:26,236 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:55:26,236 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:55:26,276 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:55:29,365 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:55:34,366 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:55:39,367 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:55:41,236 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:55:41,236 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:55:41,276 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:55:45,315 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:55:46,673 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:55:46,674 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:55:46,674 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:55:46,675 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:55:47,390 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:55:50,676 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:55:55,150 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:55:55,893 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:55:55,894 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:55:55,895 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:55:55,895 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:55:55,896 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:55:56,236 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:55:56,236 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:55:56,239 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:55:56,394 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:55:57,394 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 09:56:01,361 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:56:06,362 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:56:11,236 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:56:11,237 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:56:11,277 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:56:12,321 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:56:17,322 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:56:22,323 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:56:25,152 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:56:26,236 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:56:26,237 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:56:26,277 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:56:27,366 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:56:32,367 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:56:37,368 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:56:41,236 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:56:41,237 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:56:41,277 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:56:43,358 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:56:48,359 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:56:53,360 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:56:55,153 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:56:56,236 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:56:56,237 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:56:56,256 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:56:59,257 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:57:04,258 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:57:09,259 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:57:11,237 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:57:11,237 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:57:11,277 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:57:14,375 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:57:15,244 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:57:15,246 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:57:15,246 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:57:15,246 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:57:15,427 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:57:20,247 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:57:24,462 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:57:24,464 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:57:24,464 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:57:24,464 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:57:25,154 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:57:25,430 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:57:26,155 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:57:26,237 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:57:26,238 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:57:26,241 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:57:27,431 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 09:57:31,282 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:57:36,283 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:57:41,237 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:57:41,237 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:57:41,278 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:57:41,286 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:57:46,287 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:57:51,288 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:57:55,155 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:57:56,237 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:57:56,238 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:57:56,278 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:57:56,338 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:58:01,339 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:58:06,340 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:58:11,237 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:58:11,238 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:58:11,278 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:58:11,344 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:58:16,345 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:58:21,346 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:58:25,157 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:58:26,238 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:58:26,238 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:58:26,278 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:58:26,351 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:58:31,351 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:58:36,352 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:58:41,237 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:58:41,238 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:58:41,278 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:58:41,366 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:58:43,929 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:58:43,930 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:58:43,930 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:58:43,931 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:58:44,461 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:58:46,931 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:58:51,932 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:58:53,139 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 09:58:53,140 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 09:58:53,141 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 09:58:53,141 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 09:58:53,464 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 09:58:55,158 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:58:55,465 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 09:58:56,238 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:58:56,238 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:58:56,241 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:58:57,285 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:59:02,286 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:59:07,287 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:59:11,238 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:59:11,238 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:59:11,278 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:59:12,331 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:59:17,332 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:59:22,332 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:59:25,159 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:59:26,238 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:59:26,239 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:59:26,279 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:59:27,368 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:59:32,369 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:59:37,370 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:59:41,238 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:59:41,239 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:59:41,279 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:59:43,315 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:59:48,316 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:59:53,317 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 09:59:55,160 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 09:59:56,238 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 09:59:56,239 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 09:59:56,279 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 09:59:58,350 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:00:03,351 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:00:08,352 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:00:11,239 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:00:11,240 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:00:11,285 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:00:12,494 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:00:12,496 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:00:12,496 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:00:12,496 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:00:13,495 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:00:13,497 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:00:18,498 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:00:21,732 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:00:21,733 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:00:21,733 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:00:21,734 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:00:22,498 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:00:23,498 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:00:23,990 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:00:25,161 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:00:26,239 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:00:26,240 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:00:26,244 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:00:29,333 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:00:34,334 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:00:39,335 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:00:41,239 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:00:41,239 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:00:41,279 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:00:44,348 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:00:49,348 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:00:54,349 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:00:55,162 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:00:56,239 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:00:56,239 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:00:56,280 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:00:59,356 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:01:04,357 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:01:09,358 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:01:11,239 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:01:11,240 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:01:11,259 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:01:15,260 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:01:20,261 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:01:25,163 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:01:26,164 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:01:26,239 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:01:26,240 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:01:26,275 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:01:31,276 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:01:36,277 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:01:41,240 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:01:41,240 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:01:41,280 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:01:41,281 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:01:41,304 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:01:41,304 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:01:41,305 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:01:41,306 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:01:41,528 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:01:46,307 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:01:50,475 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:01:50,476 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:01:50,476 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:01:50,476 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:01:50,531 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:01:51,532 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:01:51,811 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:01:55,164 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:01:56,240 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:01:56,240 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:01:56,243 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:01:57,311 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:02:02,312 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:02:07,313 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:02:11,240 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:02:11,240 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:02:11,280 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:02:12,352 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:02:17,353 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:02:22,354 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:02:25,165 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:02:26,240 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:02:26,241 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:02:26,281 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:02:28,318 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:02:33,319 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:02:38,320 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:02:41,240 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:02:41,241 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:02:41,281 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:02:43,322 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:02:48,323 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:02:53,324 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:02:55,165 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:02:56,240 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:02:56,241 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:02:56,281 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:02:58,356 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:03:03,357 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:03:08,358 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:03:10,067 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:03:10,069 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:03:10,069 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:03:10,069 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:03:10,560 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:03:11,240 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:03:11,241 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:03:11,245 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:03:14,313 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:03:19,300 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:03:19,302 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:03:19,302 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:03:19,302 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:03:19,532 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:03:19,562 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:03:21,563 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:03:24,532 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:03:25,166 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:03:26,241 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:03:26,241 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:03:26,244 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:03:30,368 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:03:35,369 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:03:40,370 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:03:41,241 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:03:41,241 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:03:41,281 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:03:46,348 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:03:51,349 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:03:55,168 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:03:56,241 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:03:56,241 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:03:56,278 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:03:57,279 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:04:02,280 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:04:07,281 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:04:11,241 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:04:11,241 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:04:11,281 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:04:12,301 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:04:17,302 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:04:22,303 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:04:25,169 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:04:26,241 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:04:26,241 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:04:26,282 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:04:27,369 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:04:32,370 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:04:37,371 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:04:38,779 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:04:38,780 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:04:38,780 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:04:38,781 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:04:39,590 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:04:41,241 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:04:41,242 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:04:41,242 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:04:43,307 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:04:48,006 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:04:48,007 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:04:48,007 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:04:48,008 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:04:48,594 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:04:49,227 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:04:49,594 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:04:54,228 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:04:55,170 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:04:56,241 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:04:56,242 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:04:56,245 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:04:59,362 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:05:04,363 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:05:09,364 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:05:11,242 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:05:11,243 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:05:11,283 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:05:15,312 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:05:20,313 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:05:25,174 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:05:26,175 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:05:26,242 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:05:26,242 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:05:26,282 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:05:31,307 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:05:36,308 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:05:41,242 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:05:41,243 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:05:41,283 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:05:41,319 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:05:46,320 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:05:51,321 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:05:55,172 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:05:56,242 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:05:56,243 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:05:56,283 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:05:56,326 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:06:01,327 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:06:06,328 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:06:07,511 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:06:07,512 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:06:07,513 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:06:07,514 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:06:07,623 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:06:11,242 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:06:11,243 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:06:11,247 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:06:12,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:06:16,735 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:06:16,737 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:06:16,737 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:06:16,738 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:06:17,627 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:06:17,627 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:06:18,046 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:06:23,047 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:06:25,173 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:06:26,243 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:06:26,243 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:06:26,247 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:06:28,334 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:06:33,335 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:06:38,336 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:06:41,243 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:06:41,243 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:06:41,283 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:06:44,323 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:06:49,324 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:06:54,325 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:06:55,176 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:06:56,243 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:06:56,243 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:06:56,283 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:06:59,365 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:07:04,366 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:07:09,367 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:07:11,243 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:07:11,244 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:07:11,283 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:07:15,364 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:07:20,365 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:07:25,175 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:07:26,176 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:07:26,243 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:07:26,244 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:07:26,284 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:07:31,359 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:07:36,124 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:07:36,125 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:07:36,125 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:07:36,126 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:07:36,655 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:07:37,127 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:07:41,243 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:07:41,244 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:07:41,247 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:07:42,374 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:07:45,339 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:07:45,340 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:07:45,340 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:07:45,341 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:07:45,658 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:07:47,563 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:07:47,659 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:07:52,564 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:07:55,176 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:07:56,243 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:07:56,244 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:07:56,245 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:07:58,329 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:08:03,330 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:08:08,331 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:08:11,243 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:08:11,244 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:08:11,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:08:14,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:08:19,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:08:24,273 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:08:25,177 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:08:26,243 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:08:26,244 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:08:26,284 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:08:29,317 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:08:34,318 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:08:39,319 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:08:41,244 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:08:41,245 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:08:41,285 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:08:44,363 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:08:49,364 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:08:54,365 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:08:55,178 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:08:56,244 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:08:56,244 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:08:56,284 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:09:00,308 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:09:04,521 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:09:04,522 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:09:04,523 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:09:04,523 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:09:04,689 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:09:05,524 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:09:10,525 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:09:11,244 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:09:11,245 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:09:11,248 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:09:13,728 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:09:13,729 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:09:13,729 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:09:13,730 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:09:14,692 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:09:15,693 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:09:15,954 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:09:20,955 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:09:25,179 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:09:26,179 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:09:26,244 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:09:26,244 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:09:26,248 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:09:31,274 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:09:36,275 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:09:41,244 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:09:41,245 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:09:41,285 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:09:41,288 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:09:46,289 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:09:51,289 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:09:55,180 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:09:56,245 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:09:56,245 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:09:56,285 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:09:56,334 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:10:01,334 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:10:06,335 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:10:11,245 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:10:11,245 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:10:11,285 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:10:11,375 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:10:16,376 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:10:21,377 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:10:25,181 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:10:26,245 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:10:26,245 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:10:26,285 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:10:27,366 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:10:32,367 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:10:33,085 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:10:33,086 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:10:33,086 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:10:33,087 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:10:33,720 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:10:38,088 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:10:41,245 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:10:41,246 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:10:41,249 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:10:42,313 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:10:42,315 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:10:42,315 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:10:42,315 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:10:42,725 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:10:43,542 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:10:44,725 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:10:48,543 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:10:53,543 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:10:55,182 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:10:56,245 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:10:56,246 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:10:56,246 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:10:59,346 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:11:04,347 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:11:09,347 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:11:11,245 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:11:11,246 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:11:11,286 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:11:15,292 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:11:20,293 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:11:25,183 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:11:26,184 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:11:26,245 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:11:26,246 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:11:26,286 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:11:31,342 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:11:36,343 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:11:41,246 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:11:41,246 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:11:41,286 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:11:42,331 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:11:47,332 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:11:52,333 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:11:55,184 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:11:56,246 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:11:56,246 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:11:56,287 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:11:58,305 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:12:01,906 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:12:01,907 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:12:01,908 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:12:01,908 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:12:02,756 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:12:03,909 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:12:08,910 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:12:11,103 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:12:11,104 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:12:11,104 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:12:11,105 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:12:11,246 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:12:11,246 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:12:11,249 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:12:11,759 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:12:12,760 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:12:14,379 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:12:19,380 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:12:24,381 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:12:25,185 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:12:26,246 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:12:26,246 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:12:26,286 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:12:29,381 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:12:34,382 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:12:39,383 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:12:41,246 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:12:41,247 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:12:41,287 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:12:45,359 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:12:50,360 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:12:55,186 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:12:56,187 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:12:56,246 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:12:56,246 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:12:56,278 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:13:01,280 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:13:06,281 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:13:11,246 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:13:11,247 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:13:11,287 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:13:11,370 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:13:16,371 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:13:21,372 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:13:25,187 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:13:26,247 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:13:26,247 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:13:26,287 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:13:27,342 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:13:30,545 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:13:30,546 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:13:30,547 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:13:30,547 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:13:30,788 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:13:32,548 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:13:37,549 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:13:39,764 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:13:39,765 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:13:39,765 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:13:39,766 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:13:39,792 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:13:41,247 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:13:41,247 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:13:41,251 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:13:42,793 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:13:43,345 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:13:48,346 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:13:53,347 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:13:55,187 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:13:56,247 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:13:56,247 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:13:56,287 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:13:59,290 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:14:04,291 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:14:09,292 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:14:11,247 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:14:11,247 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:14:11,287 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:14:14,380 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:14:19,381 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:14:24,382 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:14:25,189 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:14:26,247 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:14:26,248 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:14:26,273 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:14:30,274 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:14:35,275 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:14:40,276 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:14:41,247 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:14:41,248 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:14:41,288 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:14:45,335 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:14:50,336 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:14:55,190 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:14:56,191 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:14:56,247 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:14:56,248 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:14:56,288 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:14:59,085 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:14:59,086 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:14:59,086 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:14:59,087 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:14:59,822 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:15:02,088 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:15:07,088 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:15:08,321 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:15:08,323 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:15:08,323 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:15:08,323 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:15:08,826 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:15:10,826 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:15:11,248 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:15:11,249 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:15:11,249 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:15:12,351 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:15:17,352 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:15:22,353 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:15:25,191 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:15:26,248 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:15:26,248 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:15:26,288 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:15:28,331 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:15:33,332 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:15:38,333 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:15:41,248 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:15:41,249 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:15:41,288 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:15:43,385 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:15:48,386 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:15:53,387 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:15:55,192 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:15:56,248 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:15:56,249 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:15:56,289 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:15:59,311 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:16:04,312 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:16:09,313 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:16:11,248 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:16:11,249 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:16:11,289 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:16:14,324 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:16:19,325 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:16:24,326 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:16:25,194 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:16:26,248 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:16:26,249 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:16:26,289 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:16:27,612 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:16:27,613 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:16:27,613 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:16:27,614 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:16:27,856 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:16:29,614 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:16:34,615 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:16:36,844 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:16:36,845 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:16:36,846 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:16:36,846 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:16:36,860 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:16:38,862 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:16:40,292 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:16:41,248 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:16:41,250 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:16:41,252 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:16:45,386 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:16:50,387 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:16:55,194 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:16:56,195 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:16:56,249 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:16:56,249 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:16:56,289 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:17:01,328 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:17:06,330 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:17:11,249 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:17:11,249 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:17:11,289 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:17:12,290 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:17:17,291 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:17:22,293 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:17:25,195 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:17:26,249 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:17:26,250 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:17:26,290 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:17:27,342 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:17:32,343 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:17:37,344 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:17:41,249 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:17:41,250 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:17:41,290 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:17:43,342 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:17:48,343 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:17:53,344 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:17:55,196 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:17:56,249 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:17:56,250 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:17:56,290 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:17:56,338 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:17:56,340 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:17:56,340 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:17:56,341 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:17:56,896 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:17:59,342 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:18:04,342 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:18:05,561 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:18:05,562 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:18:05,562 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:18:05,563 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:18:05,899 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:18:06,900 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:18:09,833 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:18:11,250 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:18:11,250 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:18:11,251 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:18:15,314 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:18:20,315 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:18:25,197 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:18:26,198 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:18:26,249 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:18:26,250 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:18:26,290 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:18:31,332 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:18:36,333 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:18:41,250 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:18:41,250 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:18:41,290 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:18:42,331 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:18:47,332 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:18:52,333 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:18:55,202 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:18:56,250 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:18:56,250 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:18:56,291 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:18:57,370 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:19:02,371 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:19:07,372 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:19:11,250 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:19:11,251 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:19:11,287 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:19:13,288 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:19:18,289 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:19:23,290 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:19:25,033 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:19:25,035 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:19:25,035 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:19:25,035 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:19:25,200 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:19:25,931 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:19:26,250 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:19:26,251 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:19:26,254 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:19:28,295 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:19:33,296 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:19:34,255 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:19:34,256 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:19:34,256 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:19:34,259 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:19:34,935 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:19:36,935 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:19:38,508 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:19:41,250 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:19:41,251 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:19:41,251 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:19:44,362 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:19:49,363 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:19:54,364 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:19:55,200 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:19:56,251 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:19:56,251 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:19:56,291 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:20:00,323 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:20:05,324 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:20:10,326 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:20:11,251 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:20:11,251 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:20:11,291 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:20:15,391 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:20:20,392 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:20:25,203 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:20:26,204 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:20:26,250 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:20:26,251 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:20:26,291 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:20:31,394 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:20:36,395 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:20:41,251 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:20:41,252 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:20:41,291 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:20:42,367 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:20:47,368 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:20:52,369 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:20:53,469 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:20:53,470 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:20:53,470 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:20:53,471 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:20:53,964 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:20:55,202 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:20:56,251 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:20:56,252 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:20:56,255 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:20:58,318 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:21:02,685 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:21:02,686 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:21:02,687 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:21:02,687 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:21:02,967 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:21:03,913 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:21:04,968 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:21:08,914 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:21:11,251 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:21:11,252 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:21:11,255 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:21:14,359 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:21:19,360 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:21:24,361 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:21:25,204 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:21:26,251 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:21:26,252 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:21:26,292 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:21:30,355 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:21:35,356 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:21:40,357 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:21:41,252 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:21:41,252 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:21:41,292 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:21:46,299 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:21:51,301 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:21:55,205 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:21:56,252 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:21:56,252 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:21:56,292 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:21:56,303 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:22:01,304 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:22:06,306 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:22:11,252 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:22:11,252 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:22:11,286 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:22:12,287 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:22:17,288 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:22:22,152 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:22:22,153 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:22:22,154 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:22:22,154 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:22:22,998 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:22:23,155 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:22:25,206 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:22:26,252 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:22:26,253 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:22:26,253 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:22:28,362 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:22:31,359 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:22:31,360 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:22:31,361 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:22:31,361 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:22:32,002 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:22:33,002 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:22:33,593 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:22:38,594 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:22:41,252 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:22:41,252 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:22:41,255 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:22:44,309 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:22:49,310 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:22:54,311 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:22:55,207 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:22:56,252 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:22:56,252 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:22:56,293 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:23:00,299 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:23:05,300 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:23:10,301 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:23:11,252 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:23:11,253 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:23:11,293 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:23:15,370 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:23:20,371 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:23:25,208 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:23:26,209 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:23:26,252 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:23:26,253 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:23:26,293 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:23:31,353 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:23:36,354 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:23:41,253 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:23:41,253 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:23:41,293 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:23:41,371 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:23:46,372 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:23:50,739 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:23:50,740 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:23:50,740 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:23:50,741 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:23:51,032 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:23:51,741 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:23:55,209 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:23:56,254 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:23:56,254 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:23:56,257 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:23:57,384 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:23:59,974 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:23:59,976 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:23:59,976 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:23:59,976 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:24:00,035 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:24:01,035 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:24:03,207 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:24:08,208 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:24:11,253 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:24:11,253 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:24:11,257 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:24:13,320 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:24:18,321 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:24:23,322 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:24:25,210 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:24:26,253 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:24:26,254 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:24:26,287 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:24:29,288 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:24:34,289 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:24:39,290 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:24:41,253 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:24:41,254 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:24:41,294 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:24:44,359 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:24:49,360 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:24:54,361 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:24:55,212 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:24:56,253 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:24:56,254 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:24:56,294 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:24:59,388 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:25:04,389 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:25:09,390 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:25:11,253 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:25:11,254 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:25:11,294 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:25:15,312 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:25:19,175 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:25:19,177 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:25:19,177 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:25:19,177 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:25:20,064 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:25:21,179 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:25:25,212 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:25:26,213 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:25:26,254 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:25:26,254 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:25:26,257 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:25:28,408 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:25:28,410 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:25:28,410 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:25:28,410 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:25:29,067 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:25:31,068 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:25:31,643 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:25:36,644 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:25:41,254 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:25:41,255 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:25:41,257 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:25:42,377 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:25:47,378 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:25:52,379 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:25:55,213 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:25:56,254 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:25:56,254 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:25:56,294 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:25:58,346 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:26:03,347 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:26:08,348 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:26:11,254 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:26:11,255 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:26:11,295 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:26:14,326 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:26:19,327 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:26:24,328 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:26:25,214 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:26:26,254 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:26:26,255 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:26:26,295 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:26:30,322 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:26:35,323 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:26:40,323 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:26:41,254 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:26:41,255 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:26:41,295 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:26:46,311 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:26:47,582 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:26:47,583 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:26:47,584 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:26:47,584 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:26:48,099 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:26:51,585 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:26:55,215 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:26:56,254 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:26:56,255 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:26:56,258 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:26:56,828 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:26:56,829 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:26:56,830 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:26:56,830 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:26:56,830 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:26:57,103 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:26:59,104 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:27:02,089 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:27:07,090 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:27:11,254 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:27:11,255 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:27:11,258 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:27:12,303 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:27:17,304 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:27:22,305 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:27:25,215 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:27:26,255 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:27:26,255 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:27:26,295 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:27:27,348 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:27:32,349 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:27:37,350 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:27:41,255 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:27:41,255 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:27:41,295 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:27:43,325 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:27:48,326 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:27:53,326 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:27:55,216 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:27:56,255 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:27:56,255 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:27:56,295 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:27:58,382 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:28:03,382 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:28:08,383 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:28:11,255 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:28:11,256 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:28:11,296 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:28:14,341 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:28:15,906 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:28:15,907 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:28:15,907 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:28:15,908 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:28:16,132 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:28:19,908 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:28:24,909 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:28:25,146 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:28:25,147 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:28:25,147 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:28:25,147 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:28:25,217 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:28:26,136 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:28:26,255 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:28:26,256 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:28:26,259 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:28:27,137 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:28:30,322 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:28:35,323 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:28:40,324 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:28:41,255 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:28:41,256 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:28:41,297 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:28:46,304 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:28:51,305 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:28:55,218 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:28:56,255 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:28:56,256 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:28:56,296 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:28:56,369 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:29:01,370 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:29:06,371 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:29:11,256 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:29:11,256 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:29:11,296 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:29:12,333 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:29:17,334 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:29:22,335 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:29:25,219 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:29:26,256 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:29:26,256 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:29:26,296 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:29:27,362 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:29:32,362 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:29:37,363 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:29:41,256 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:29:41,257 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:29:41,297 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:29:43,317 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:29:44,542 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:29:44,543 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:29:44,543 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:29:44,544 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:29:45,167 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:29:48,545 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:29:53,546 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:29:53,752 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:29:53,753 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:29:53,754 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:29:53,754 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:29:54,170 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:29:55,171 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:29:55,220 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:29:56,256 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:29:56,257 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:29:56,257 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:29:59,389 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:30:04,390 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:30:09,391 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:30:11,256 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:30:11,257 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:30:11,297 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:30:15,306 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:30:20,307 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:30:25,220 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:30:26,221 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:30:26,256 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:30:26,257 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:30:26,297 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:30:31,397 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:30:36,398 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:30:41,256 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:30:41,257 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:30:41,297 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:30:42,327 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:30:47,328 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:30:52,329 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:30:55,222 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:30:56,257 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:30:56,257 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:30:56,297 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:30:57,331 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:31:02,333 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:31:07,333 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:31:11,257 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:31:11,257 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:31:11,297 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:31:12,378 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:31:12,796 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:31:12,797 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:31:12,798 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:31:12,798 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:31:13,204 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:31:17,800 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:31:22,026 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:31:22,027 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:31:22,027 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:31:22,028 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:31:22,207 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:31:23,207 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:31:23,295 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:31:25,223 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:31:26,257 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:31:26,258 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:31:26,261 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:31:28,342 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:31:33,343 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:31:38,344 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:31:41,257 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:31:41,258 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:31:41,299 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:31:44,300 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:31:49,301 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:31:54,302 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:31:55,224 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:31:56,257 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:31:56,258 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:31:56,299 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:31:59,314 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:32:04,315 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:32:09,316 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:32:11,257 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:32:11,258 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:32:11,298 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:32:14,362 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:32:19,363 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:32:24,364 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:32:25,228 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:32:26,257 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:32:26,258 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:32:26,298 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:32:29,369 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:32:34,370 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:32:39,371 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:32:41,258 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:32:41,258 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:32:41,281 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:32:41,352 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:32:41,353 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:32:41,353 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:32:41,354 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:32:42,239 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:32:45,355 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:32:50,356 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:32:50,569 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:32:50,571 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:32:50,571 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:32:50,572 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:32:51,243 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:32:51,243 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:32:55,226 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:32:56,227 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:32:56,258 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:32:56,258 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:32:56,261 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:33:01,307 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:33:06,308 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:33:11,258 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:33:11,258 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:33:11,299 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:33:11,389 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:33:16,390 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:33:21,391 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:33:25,229 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:33:26,258 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:33:26,258 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:33:26,299 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:33:26,391 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:33:31,392 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:33:36,393 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:33:41,258 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:33:41,259 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:33:41,299 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:33:42,321 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:33:47,322 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:33:52,323 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:33:55,228 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:33:56,258 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:33:56,259 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:33:56,282 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:33:58,283 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:34:03,283 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:34:08,284 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:34:09,760 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:34:09,761 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:34:09,761 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:34:09,762 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:34:10,273 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:34:11,259 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:34:11,259 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:34:11,262 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:34:13,351 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:34:18,352 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:34:18,989 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:34:18,991 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:34:18,991 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:34:18,992 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:34:19,276 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:34:21,277 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:34:24,228 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:34:25,229 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:34:26,259 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:34:26,260 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:34:26,260 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:34:29,388 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:34:34,389 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:34:39,390 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:34:41,259 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:34:41,259 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:34:41,299 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:34:45,345 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:34:50,345 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:34:55,229 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:34:56,230 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:34:56,259 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:34:56,259 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:34:56,300 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:35:01,358 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:35:06,359 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:35:11,259 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:35:11,260 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:35:11,300 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:35:12,323 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:35:17,324 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:35:22,325 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:35:25,231 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:35:26,259 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:35:26,259 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:35:26,298 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:35:28,299 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:35:33,300 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:35:38,301 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:35:38,776 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:35:38,777 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:35:38,777 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:35:38,778 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:35:39,307 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:35:41,259 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:35:41,260 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:35:41,263 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:35:43,335 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:35:48,011 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:35:48,013 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:35:48,013 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:35:48,014 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:35:48,311 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:35:49,277 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:35:49,312 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:35:54,278 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:35:55,232 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:35:56,259 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:35:56,260 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:35:56,264 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:35:59,338 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:36:04,339 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:36:09,339 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:36:11,260 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:36:11,260 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:36:11,300 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:36:14,343 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:36:19,344 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:36:24,345 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:36:25,234 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:36:26,260 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:36:26,260 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:36:26,300 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:36:29,369 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:36:34,370 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:36:39,371 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:36:41,260 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:36:41,260 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:36:41,300 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:36:44,377 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:36:49,378 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:36:54,379 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:36:55,234 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:36:56,260 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:36:56,260 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:36:56,300 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:37:00,363 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:37:05,364 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:37:07,279 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:37:07,280 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:37:07,280 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:37:07,281 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:37:07,342 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:37:11,260 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:37:11,261 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:37:11,263 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:37:11,350 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:37:16,351 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:37:16,507 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:37:16,509 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:37:16,509 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:37:16,509 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:37:17,345 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:37:18,345 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:37:21,763 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:37:25,235 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:37:26,260 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:37:26,261 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:37:26,262 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:37:27,348 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:37:32,349 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:37:37,350 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:37:41,260 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:37:41,261 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:37:41,301 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:37:42,379 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:37:47,380 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:37:52,381 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:37:55,237 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:37:56,261 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:37:56,261 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:37:56,301 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:37:58,331 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:38:03,331 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:38:08,332 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:38:11,261 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:38:11,261 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:38:11,301 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:38:14,327 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:38:19,328 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:38:24,329 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:38:25,238 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:38:26,261 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:38:26,261 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:38:26,301 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:38:29,382 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:38:34,382 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:38:35,814 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:38:35,815 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:38:35,816 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:38:35,816 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:38:36,374 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:38:39,817 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:38:41,261 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:38:41,262 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:38:41,264 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:38:45,037 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:38:45,038 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:38:45,038 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:38:45,039 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:38:45,039 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:38:45,377 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:38:46,377 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:38:50,260 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:38:55,240 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:38:56,241 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:38:56,261 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:38:56,262 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:38:56,264 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:39:01,336 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:39:06,337 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:39:11,261 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:39:11,261 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:39:11,302 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:39:12,312 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:39:17,313 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:39:22,313 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:39:25,241 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:39:26,261 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:39:26,262 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:39:26,302 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:39:27,327 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:39:32,328 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:39:37,328 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:39:41,262 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:39:41,262 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:39:41,302 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:39:42,353 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:39:47,354 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:39:52,355 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:39:55,242 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:39:56,262 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:39:56,262 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:39:56,302 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:39:57,382 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:40:02,383 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:40:04,388 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:40:04,389 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:40:04,389 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:40:04,390 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:40:04,407 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:40:07,391 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:40:11,262 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:40:11,262 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:40:11,265 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:40:13,342 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:40:13,601 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:40:13,602 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:40:13,602 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:40:13,602 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:40:14,411 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:40:16,412 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:40:18,827 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:40:23,828 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:40:25,243 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:40:26,262 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:40:26,262 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:40:26,265 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:40:29,364 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:40:34,365 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:40:39,366 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:40:41,262 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:40:41,263 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:40:41,303 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:40:44,376 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:40:49,377 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:40:54,378 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:40:55,245 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:40:56,262 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:40:56,263 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:40:56,303 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:41:00,330 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:41:05,331 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:41:10,332 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:41:11,262 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:41:11,263 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:41:11,303 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:41:15,409 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:41:20,410 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:41:25,246 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:41:26,247 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:41:26,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:41:26,270 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:41:26,311 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:41:31,390 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:41:32,521 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:41:32,522 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:41:32,522 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:41:32,523 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:41:33,440 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:41:36,524 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:41:41,267 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:41:41,268 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:41:41,269 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:41:41,733 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:41:41,734 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:41:41,735 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:41:41,735 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:41:41,735 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:41:42,443 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:41:44,444 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:41:46,968 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:41:51,969 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:41:55,246 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:41:56,268 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:41:56,269 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:41:56,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:41:57,351 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:42:02,352 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:42:07,353 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:42:11,267 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:42:11,268 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:42:11,308 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:42:13,346 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:42:18,347 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:42:23,348 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:42:25,248 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:42:26,268 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:42:26,268 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:42:26,308 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:42:29,314 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:42:34,315 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:42:39,316 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:42:41,268 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:42:41,268 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:42:41,308 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:42:44,332 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:42:49,333 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:42:54,333 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:42:55,249 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:42:56,268 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:42:56,268 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:42:56,308 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:43:00,312 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:43:00,869 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:43:00,871 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:43:00,871 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:43:00,871 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:43:01,474 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:43:05,872 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:43:10,105 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:43:10,106 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:43:10,106 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:43:10,107 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:43:10,478 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:43:11,268 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:43:11,269 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:43:11,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:43:11,388 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:43:12,479 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:43:16,389 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:43:21,389 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:43:25,251 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:43:26,268 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:43:26,269 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:43:26,309 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:43:27,331 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:43:32,332 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:43:37,333 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:43:41,268 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:43:41,269 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:43:41,309 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:43:43,312 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:43:48,313 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:43:53,314 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:43:55,252 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:43:56,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:43:56,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:43:56,271 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:43:58,324 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:44:03,325 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:44:08,326 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:44:11,268 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:44:11,269 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:44:11,309 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:44:13,381 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:44:18,381 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:44:23,382 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:44:25,253 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:44:26,269 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:44:26,270 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:44:26,311 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:44:29,330 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:44:29,603 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:44:29,604 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:44:29,605 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:44:29,605 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:44:30,509 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:44:34,606 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:44:38,833 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:44:38,834 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:44:38,834 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:44:38,835 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:44:39,513 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:44:40,069 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:44:40,513 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:44:41,269 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:44:41,269 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:44:41,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:44:45,408 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:44:50,409 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:44:55,254 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:44:56,255 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:44:56,269 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:44:56,269 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:44:56,310 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:45:01,343 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:45:06,344 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:45:11,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:45:11,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:45:11,271 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:45:12,330 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:45:17,331 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:45:22,332 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:45:25,256 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:45:26,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:45:26,271 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:45:26,311 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:45:27,337 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:45:32,338 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:45:37,339 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:45:41,269 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:45:41,270 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:45:41,310 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:45:42,380 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:45:47,381 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:45:52,382 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:45:55,260 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:45:56,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:45:56,270 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:45:56,311 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:45:58,353 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:45:58,617 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:45:58,619 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:45:58,619 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:45:58,619 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:45:59,541 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:46:03,620 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:46:07,844 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:46:07,845 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:46:07,845 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:46:07,845 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:46:08,545 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:46:09,076 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:46:10,545 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:46:11,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:46:11,271 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:46:11,273 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:46:14,328 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:46:19,328 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:46:24,329 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:46:25,258 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:46:26,269 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:46:26,270 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:46:26,310 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:46:29,343 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:46:34,344 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:46:39,345 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:46:41,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:46:41,271 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:46:41,311 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:46:44,394 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:46:49,395 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:46:54,396 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:46:55,260 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:46:56,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:46:56,270 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:46:56,308 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:47:00,310 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:47:05,311 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:47:10,312 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:47:11,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:47:11,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:47:11,272 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:47:15,381 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:47:20,381 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:47:25,260 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:47:26,261 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:47:26,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:47:26,270 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:47:26,310 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:47:27,309 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:47:27,310 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:47:27,311 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:47:27,311 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:47:27,574 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:47:31,312 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:47:36,313 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:47:36,532 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:47:36,533 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:47:36,533 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:47:36,534 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:47:36,578 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:47:38,579 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:47:41,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:47:41,271 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:47:41,273 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:47:41,375 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:47:46,375 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:47:51,376 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:47:55,261 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:47:56,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:47:56,271 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:47:56,309 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:47:57,310 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:48:02,311 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:48:07,312 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:48:11,270 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:48:11,271 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:48:11,311 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:48:12,336 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:48:17,337 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:48:22,338 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:48:25,262 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:48:26,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:48:26,272 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:48:26,311 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:48:27,387 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:48:32,388 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:48:37,389 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:48:41,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:48:41,271 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:48:41,311 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:48:43,342 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:48:48,343 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:48:53,344 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:48:55,263 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:48:55,635 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:48:55,636 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:48:55,637 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:48:55,637 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:48:56,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:48:56,271 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:48:56,274 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:48:56,607 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:48:59,335 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:49:04,335 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:49:04,863 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:49:04,864 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:49:04,864 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:49:04,864 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:49:05,611 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:49:06,611 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:49:10,124 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:49:11,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:49:11,272 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:49:11,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:49:15,363 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:49:20,364 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:49:25,263 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:49:26,264 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:49:26,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:49:26,271 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:49:26,312 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:49:31,345 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:49:36,347 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:49:41,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:49:41,272 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:49:41,312 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:49:42,345 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:49:47,346 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:49:52,346 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:49:55,264 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:49:56,271 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:49:56,272 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:49:56,312 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:49:57,376 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:50:02,377 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:50:07,378 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:50:11,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:50:11,272 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:50:11,312 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:50:12,596 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:50:17,597 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:50:22,598 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:50:24,628 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:50:24,629 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:50:24,630 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:50:24,630 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:50:24,639 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:50:25,265 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:50:26,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:50:26,273 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:50:26,273 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:50:28,344 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:50:33,345 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:50:33,865 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:50:33,867 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:50:33,867 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:50:33,868 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:50:34,642 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:50:36,643 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/output.log 2024-04-13 10:50:39,307 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:50:41,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:50:41,272 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:50:41,276 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:50:44,388 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:50:49,389 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:50:54,390 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:50:55,266 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:50:56,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:50:56,273 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:50:56,313 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:51:00,335 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:51:05,336 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:51:10,337 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:51:11,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:51:11,273 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:51:11,313 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:51:15,355 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:51:20,356 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:51:25,267 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:51:26,267 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:51:26,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:51:26,273 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:51:26,313 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:51:31,332 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:51:36,333 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:51:41,272 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:51:41,273 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:51:41,313 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:51:41,338 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:51:46,339 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:51:51,340 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:51:54,049 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:51:54,050 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:51:54,050 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:51:54,050 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:51:54,672 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:51:55,268 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:51:56,273 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: stop_status 2024-04-13 10:51:56,274 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: internal_messages 2024-04-13 10:51:56,274 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: stop_status 2024-04-13 10:51:56,354 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:52:01,355 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report 2024-04-13 10:52:03,263 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:52:03,265 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:52:03,265 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:52:03,266 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:52:03,512 DEBUG SenderThread:113 [sender.py:send():379] send: telemetry 2024-04-13 10:52:03,512 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:52:03,514 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: partial_history 2024-04-13 10:52:03,515 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:52:03,515 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:52:03,516 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:52:03,516 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:52:03,517 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:52:03,517 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:52:03,518 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: pause 2024-04-13 10:52:03,518 INFO HandlerThread:113 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-13 10:52:03,518 INFO HandlerThread:113 [system_monitor.py:finish():203] Stopping system monitor 2024-04-13 10:52:03,518 DEBUG SystemMonitor:113 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-13 10:52:03,518 DEBUG SystemMonitor:113 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-13 10:52:03,519 INFO HandlerThread:113 [interfaces.py:finish():202] Joined cpu monitor 2024-04-13 10:52:03,519 INFO HandlerThread:113 [interfaces.py:finish():202] Joined disk monitor 2024-04-13 10:52:03,519 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:52:03,520 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:52:03,523 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:52:03,523 DEBUG SenderThread:113 [sender.py:send():379] send: history 2024-04-13 10:52:03,523 DEBUG SenderThread:113 [sender.py:send_request():406] send_request: summary_record 2024-04-13 10:52:03,524 INFO SenderThread:113 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end 2024-04-13 10:52:03,534 INFO HandlerThread:113 [interfaces.py:finish():202] Joined gpu monitor 2024-04-13 10:52:03,534 INFO HandlerThread:113 [interfaces.py:finish():202] Joined memory monitor 2024-04-13 10:52:03,534 INFO HandlerThread:113 [interfaces.py:finish():202] Joined network monitor 2024-04-13 10:52:03,535 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: resume 2024-04-13 10:52:03,535 INFO HandlerThread:113 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-13 10:52:03,535 INFO HandlerThread:113 [system_monitor.py:start():194] Starting system monitor 2024-04-13 10:52:03,535 INFO SystemMonitor:113 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-13 10:52:03,536 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: pause 2024-04-13 10:52:03,536 INFO HandlerThread:113 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-04-13 10:52:03,536 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:52:03,536 INFO SystemMonitor:113 [interfaces.py:start():190] Started cpu monitoring 2024-04-13 10:52:03,536 INFO HandlerThread:113 [system_monitor.py:finish():203] Stopping system monitor 2024-04-13 10:52:03,537 INFO SystemMonitor:113 [interfaces.py:start():190] Started disk monitoring 2024-04-13 10:52:03,537 DEBUG SystemMonitor:113 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-04-13 10:52:03,537 DEBUG SystemMonitor:113 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-04-13 10:52:03,537 DEBUG SystemMonitor:113 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-04-13 10:52:03,538 INFO HandlerThread:113 [interfaces.py:finish():202] Joined cpu monitor 2024-04-13 10:52:03,540 INFO HandlerThread:113 [interfaces.py:finish():202] Joined disk monitor 2024-04-13 10:52:03,540 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: resume 2024-04-13 10:52:03,540 INFO HandlerThread:113 [handler.py:handle_request_resume():699] starting system metrics thread 2024-04-13 10:52:03,540 INFO HandlerThread:113 [system_monitor.py:start():194] Starting system monitor 2024-04-13 10:52:03,540 INFO SystemMonitor:113 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-04-13 10:52:03,541 INFO SystemMonitor:113 [interfaces.py:start():190] Started cpu monitoring 2024-04-13 10:52:03,541 DEBUG SenderThread:113 [sender.py:send():379] send: stats 2024-04-13 10:52:03,542 INFO SystemMonitor:113 [interfaces.py:start():190] Started disk monitoring 2024-04-13 10:52:03,543 INFO SystemMonitor:113 [interfaces.py:start():190] Started gpu monitoring 2024-04-13 10:52:03,543 INFO SystemMonitor:113 [interfaces.py:start():190] Started memory monitoring 2024-04-13 10:52:03,543 INFO SystemMonitor:113 [interfaces.py:start():190] Started network monitoring 2024-04-13 10:52:03,675 INFO Thread-12 :113 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240413_094054-v5dwwo6y/files/wandb-summary.json 2024-04-13 10:52:06,542 DEBUG HandlerThread:113 [handler.py:handle_request():146] handle_request: status_report