2024-02-26 12:56:46,867 INFO StreamThr :3250 [internal.py:wandb_internal():86] W&B internal server running at pid: 3250, started at: 2024-02-26 12:56:46.863894 2024-02-26 12:56:46,894 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status 2024-02-26 12:56:51,750 INFO WriterThread:3250 [datastore.py:open_for_write():87] open: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko/wandb/run-20240226_125646-ilpu1hh1/run-ilpu1hh1.wandb 2024-02-26 12:56:51,755 DEBUG SenderThread:3250 [sender.py:send():382] send: header 2024-02-26 12:56:51,766 DEBUG SenderThread:3250 [sender.py:send():382] send: run 2024-02-26 12:56:52,426 INFO SenderThread:3250 [dir_watcher.py:__init__():211] watching files in: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko/wandb/run-20240226_125646-ilpu1hh1/files 2024-02-26 12:56:52,426 INFO SenderThread:3250 [sender.py:_start_run_threads():1136] run started: ilpu1hh1 with start time 1708952206.865337 2024-02-26 12:56:52,427 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:56:52,431 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: check_version 2024-02-26 12:56:52,431 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: check_version 2024-02-26 12:56:52,582 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: run_start 2024-02-26 12:56:52,595 DEBUG HandlerThread:3250 [system_info.py:__init__():27] System info init 2024-02-26 12:56:52,596 DEBUG HandlerThread:3250 [system_info.py:__init__():42] System info init done 2024-02-26 12:56:52,596 INFO HandlerThread:3250 [system_monitor.py:start():194] Starting system monitor 2024-02-26 12:56:52,596 INFO SystemMonitor:3250 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-26 12:56:52,597 INFO HandlerThread:3250 [system_monitor.py:probe():214] Collecting system info 2024-02-26 12:56:52,599 INFO SystemMonitor:3250 [interfaces.py:start():190] Started cpu monitoring 2024-02-26 12:56:52,600 INFO SystemMonitor:3250 [interfaces.py:start():190] Started disk monitoring 2024-02-26 12:56:52,602 INFO SystemMonitor:3250 [interfaces.py:start():190] Started gpu monitoring 2024-02-26 12:56:52,604 INFO SystemMonitor:3250 [interfaces.py:start():190] Started memory monitoring 2024-02-26 12:56:52,605 INFO SystemMonitor:3250 [interfaces.py:start():190] Started network monitoring 2024-02-26 12:56:52,605 DEBUG HandlerThread:3250 [system_info.py:probe():151] Probing system 2024-02-26 12:56:52,627 DEBUG HandlerThread:3250 [gitlib.py:_init_repo():56] git repository is invalid 2024-02-26 12:56:52,627 DEBUG HandlerThread:3250 [system_info.py:probe():199] Probing system done 2024-02-26 12:56:52,630 DEBUG HandlerThread:3250 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-02-26T12:56:52.606063', 'startedAt': '2024-02-26T12:56:46.812320', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Gemma-7B-Finetuning-databricks-dolly-15k-ko.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1F6i05tAvH1iXwoMf62NCwP1cRC3gLIVJ', 'host': '28abd6843fb6', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2000.2, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.2, 'min': 0.0, 'max': 0.0}, {'current': 2000.2, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 78.1898422241211, 'used': 42.800540924072266}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.674781799316406}} 2024-02-26 12:56:52,630 INFO HandlerThread:3250 [system_monitor.py:probe():224] Finished collecting system info 2024-02-26 12:56:52,630 INFO HandlerThread:3250 [system_monitor.py:probe():227] Publishing system info 2024-02-26 12:56:52,648 INFO HandlerThread:3250 [system_monitor.py:probe():229] Finished publishing system info 2024-02-26 12:56:52,661 DEBUG SenderThread:3250 [sender.py:send():382] send: files 2024-02-26 12:56:52,662 INFO SenderThread:3250 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now 2024-02-26 12:56:52,692 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: python_packages 2024-02-26 12:56:52,692 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: python_packages 2024-02-26 12:56:52,700 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 12:56:52,701 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 12:56:52,724 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 12:56:53,037 DEBUG SenderThread:3250 [sender.py:send():382] send: telemetry 2024-02-26 12:56:53,038 DEBUG SenderThread:3250 [sender.py:send():382] send: config 2024-02-26 12:56:53,039 DEBUG SenderThread:3250 [sender.py:send():382] send: metric 2024-02-26 12:56:53,039 DEBUG SenderThread:3250 [sender.py:send():382] send: telemetry 2024-02-26 12:56:53,039 DEBUG SenderThread:3250 [sender.py:send():382] send: metric 2024-02-26 12:56:53,040 WARNING SenderThread:3250 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) 2024-02-26 12:56:53,040 DEBUG SenderThread:3250 [sender.py:send():382] send: telemetry 2024-02-26 12:56:53,431 INFO Thread-12 :3250 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko/wandb/run-20240226_125646-ilpu1hh1/files/requirements.txt 2024-02-26 12:56:53,439 INFO Thread-12 :3250 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko/wandb/run-20240226_125646-ilpu1hh1/files/wandb-metadata.json 2024-02-26 12:56:53,854 INFO wandb-upload_0:3250 [upload_job.py:push():131] Uploaded file /tmp/tmpacc0xlywwandb/tc7menn9-wandb-metadata.json 2024-02-26 12:56:58,041 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:57:03,042 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:57:03,701 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: pause 2024-02-26 12:57:03,702 INFO HandlerThread:3250 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-02-26 12:57:03,702 INFO HandlerThread:3250 [system_monitor.py:finish():203] Stopping system monitor 2024-02-26 12:57:03,703 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined cpu monitor 2024-02-26 12:57:03,707 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined disk monitor 2024-02-26 12:57:03,704 DEBUG SystemMonitor:3250 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-26 12:57:03,709 DEBUG SystemMonitor:3250 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-02-26 12:57:03,709 DEBUG SystemMonitor:3250 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-02-26 12:57:03,717 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined gpu monitor 2024-02-26 12:57:03,718 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined memory monitor 2024-02-26 12:57:03,718 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined network monitor 2024-02-26 12:57:03,718 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 12:57:07,700 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 12:57:07,701 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 12:57:07,708 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 12:57:09,009 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:57:14,010 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:57:19,016 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:57:19,481 INFO Thread-12 :3250 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko/wandb/run-20240226_125646-ilpu1hh1/files/config.yaml 2024-02-26 12:57:22,701 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 12:57:22,701 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 12:57:22,741 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 12:57:25,023 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:57:30,024 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:57:35,025 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:57:37,701 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 12:57:37,701 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 12:57:37,741 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 12:57:40,927 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:57:45,928 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:57:50,929 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:57:52,701 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 12:57:52,701 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 12:57:52,741 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 12:57:56,024 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:58:01,024 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:58:06,025 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:58:07,701 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 12:58:07,702 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 12:58:07,742 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 12:58:11,040 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:58:16,041 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:58:21,042 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:58:22,701 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 12:58:22,706 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 12:58:22,742 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 12:58:26,946 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:58:31,947 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:58:36,948 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:58:37,701 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 12:58:37,702 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 12:58:37,742 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 12:58:41,988 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:58:46,989 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:58:51,990 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:58:52,702 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 12:58:52,702 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 12:58:52,742 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 12:58:57,012 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:59:02,013 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:59:07,014 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:59:07,702 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 12:59:07,702 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 12:59:07,742 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 12:59:12,946 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:59:17,948 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:59:22,702 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 12:59:22,702 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 12:59:22,742 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 12:59:23,947 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:59:28,948 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:59:33,949 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:59:37,702 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 12:59:37,703 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 12:59:37,742 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 12:59:38,990 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:59:43,991 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:59:48,992 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:59:52,702 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 12:59:52,703 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 12:59:52,743 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 12:59:54,935 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 12:59:59,935 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:00:04,936 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:00:07,702 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:00:07,703 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:00:07,743 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:00:10,006 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:00:15,007 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:00:20,008 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:00:22,702 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:00:22,703 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:00:22,743 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:00:25,041 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:00:30,042 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:00:35,043 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:00:37,702 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:00:37,703 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:00:37,743 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:00:41,040 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:00:46,042 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:00:51,043 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:00:52,703 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:00:52,703 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:00:52,743 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:00:57,006 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:01:02,007 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:01:07,009 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:01:07,703 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:01:07,704 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:01:07,743 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:01:12,028 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:01:17,030 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:01:22,031 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:01:22,703 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:01:22,704 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:01:22,744 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:01:27,996 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:01:32,997 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:01:37,703 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:01:37,704 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:01:37,744 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:01:38,968 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:01:43,969 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:01:48,970 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:01:52,703 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:01:52,704 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:01:52,744 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:01:54,956 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:01:59,957 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:02:04,958 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:02:07,703 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:02:07,704 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:02:07,744 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:02:10,021 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:02:15,022 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:02:20,023 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:02:22,704 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:02:22,704 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:02:22,744 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:02:25,980 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:02:30,982 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:02:35,983 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:02:37,704 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:02:37,705 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:02:37,744 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:02:41,971 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:02:46,972 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:02:51,973 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:02:52,704 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:02:52,705 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:02:52,745 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:02:57,049 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:03:02,050 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:03:07,051 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:03:07,704 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:03:07,705 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:03:07,745 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:03:13,048 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:03:18,049 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:03:22,704 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:03:22,705 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:03:22,745 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:03:23,966 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:03:28,972 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:03:29,475 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: resume 2024-02-26 13:03:29,476 INFO HandlerThread:3250 [handler.py:handle_request_resume():699] starting system metrics thread 2024-02-26 13:03:29,477 INFO HandlerThread:3250 [system_monitor.py:start():194] Starting system monitor 2024-02-26 13:03:29,478 INFO SystemMonitor:3250 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-26 13:03:29,480 INFO SystemMonitor:3250 [interfaces.py:start():190] Started cpu monitoring 2024-02-26 13:03:29,481 INFO SystemMonitor:3250 [interfaces.py:start():190] Started disk monitoring 2024-02-26 13:03:29,484 INFO SystemMonitor:3250 [interfaces.py:start():190] Started gpu monitoring 2024-02-26 13:03:29,493 INFO SystemMonitor:3250 [interfaces.py:start():190] Started memory monitoring 2024-02-26 13:03:29,494 INFO SystemMonitor:3250 [interfaces.py:start():190] Started network monitoring 2024-02-26 13:03:29,516 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: pause 2024-02-26 13:03:29,516 INFO HandlerThread:3250 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-02-26 13:03:29,516 INFO HandlerThread:3250 [system_monitor.py:finish():203] Stopping system monitor 2024-02-26 13:03:29,521 DEBUG SystemMonitor:3250 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-26 13:03:29,521 DEBUG SystemMonitor:3250 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-02-26 13:03:29,521 DEBUG SystemMonitor:3250 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-02-26 13:03:29,522 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined cpu monitor 2024-02-26 13:03:29,522 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined disk monitor 2024-02-26 13:03:29,528 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined gpu monitor 2024-02-26 13:03:29,529 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined memory monitor 2024-02-26 13:03:29,529 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined network monitor 2024-02-26 13:03:29,530 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:03:32,538 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: resume 2024-02-26 13:03:32,546 INFO HandlerThread:3250 [handler.py:handle_request_resume():699] starting system metrics thread 2024-02-26 13:03:32,546 INFO HandlerThread:3250 [system_monitor.py:start():194] Starting system monitor 2024-02-26 13:03:32,560 INFO SystemMonitor:3250 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-26 13:03:32,568 INFO SystemMonitor:3250 [interfaces.py:start():190] Started cpu monitoring 2024-02-26 13:03:32,579 INFO SystemMonitor:3250 [interfaces.py:start():190] Started disk monitoring 2024-02-26 13:03:32,593 INFO SystemMonitor:3250 [interfaces.py:start():190] Started gpu monitoring 2024-02-26 13:03:32,593 INFO SystemMonitor:3250 [interfaces.py:start():190] Started memory monitoring 2024-02-26 13:03:32,602 INFO SystemMonitor:3250 [interfaces.py:start():190] Started network monitoring 2024-02-26 13:03:34,531 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:03:35,951 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: pause 2024-02-26 13:03:35,953 INFO HandlerThread:3250 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-02-26 13:03:35,953 INFO HandlerThread:3250 [system_monitor.py:finish():203] Stopping system monitor 2024-02-26 13:03:35,953 DEBUG SystemMonitor:3250 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-26 13:03:35,953 DEBUG SystemMonitor:3250 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-02-26 13:03:35,954 DEBUG SystemMonitor:3250 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-02-26 13:03:35,956 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined cpu monitor 2024-02-26 13:03:35,960 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined disk monitor 2024-02-26 13:03:35,974 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined gpu monitor 2024-02-26 13:03:35,974 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined memory monitor 2024-02-26 13:03:35,974 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined network monitor 2024-02-26 13:03:35,975 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:03:37,704 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:03:37,705 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:03:37,711 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:03:40,012 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:03:45,013 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:03:49,103 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: resume 2024-02-26 13:03:49,104 INFO HandlerThread:3250 [handler.py:handle_request_resume():699] starting system metrics thread 2024-02-26 13:03:49,105 INFO HandlerThread:3250 [system_monitor.py:start():194] Starting system monitor 2024-02-26 13:03:49,105 INFO SystemMonitor:3250 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-26 13:03:49,106 INFO SystemMonitor:3250 [interfaces.py:start():190] Started cpu monitoring 2024-02-26 13:03:49,107 INFO SystemMonitor:3250 [interfaces.py:start():190] Started disk monitoring 2024-02-26 13:03:49,110 INFO SystemMonitor:3250 [interfaces.py:start():190] Started gpu monitoring 2024-02-26 13:03:49,113 INFO SystemMonitor:3250 [interfaces.py:start():190] Started memory monitoring 2024-02-26 13:03:49,117 INFO SystemMonitor:3250 [interfaces.py:start():190] Started network monitoring 2024-02-26 13:03:49,144 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: pause 2024-02-26 13:03:49,144 INFO HandlerThread:3250 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-02-26 13:03:49,144 INFO HandlerThread:3250 [system_monitor.py:finish():203] Stopping system monitor 2024-02-26 13:03:49,145 DEBUG SystemMonitor:3250 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-26 13:03:49,145 DEBUG SystemMonitor:3250 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-02-26 13:03:49,145 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined cpu monitor 2024-02-26 13:03:49,145 DEBUG SystemMonitor:3250 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-02-26 13:03:49,146 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined disk monitor 2024-02-26 13:03:49,152 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined gpu monitor 2024-02-26 13:03:49,153 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined memory monitor 2024-02-26 13:03:49,153 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined network monitor 2024-02-26 13:03:49,153 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:03:50,154 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:03:52,704 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:03:52,705 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:03:52,711 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:03:53,654 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: resume 2024-02-26 13:03:53,658 INFO HandlerThread:3250 [handler.py:handle_request_resume():699] starting system metrics thread 2024-02-26 13:03:53,658 INFO HandlerThread:3250 [system_monitor.py:start():194] Starting system monitor 2024-02-26 13:03:53,663 INFO SystemMonitor:3250 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-26 13:03:53,663 INFO SystemMonitor:3250 [interfaces.py:start():190] Started cpu monitoring 2024-02-26 13:03:53,664 INFO SystemMonitor:3250 [interfaces.py:start():190] Started disk monitoring 2024-02-26 13:03:53,666 INFO SystemMonitor:3250 [interfaces.py:start():190] Started gpu monitoring 2024-02-26 13:03:53,672 INFO SystemMonitor:3250 [interfaces.py:start():190] Started memory monitoring 2024-02-26 13:03:53,675 INFO SystemMonitor:3250 [interfaces.py:start():190] Started network monitoring 2024-02-26 13:03:54,884 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: pause 2024-02-26 13:03:54,885 INFO HandlerThread:3250 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-02-26 13:03:54,885 INFO HandlerThread:3250 [system_monitor.py:finish():203] Stopping system monitor 2024-02-26 13:03:54,885 DEBUG SystemMonitor:3250 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-26 13:03:54,886 DEBUG SystemMonitor:3250 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-02-26 13:03:54,886 DEBUG SystemMonitor:3250 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-02-26 13:03:54,887 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined cpu monitor 2024-02-26 13:03:54,888 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined disk monitor 2024-02-26 13:03:54,900 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined gpu monitor 2024-02-26 13:03:54,900 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined memory monitor 2024-02-26 13:03:54,901 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined network monitor 2024-02-26 13:03:54,901 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:03:55,902 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:04:00,903 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:04:05,904 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:04:07,705 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:04:07,705 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:04:07,712 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:04:10,987 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:04:15,989 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:04:20,989 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:04:22,705 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:04:22,705 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:04:22,745 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:04:26,969 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:04:31,484 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: resume 2024-02-26 13:04:31,487 INFO HandlerThread:3250 [handler.py:handle_request_resume():699] starting system metrics thread 2024-02-26 13:04:31,488 INFO HandlerThread:3250 [system_monitor.py:start():194] Starting system monitor 2024-02-26 13:04:31,496 INFO SystemMonitor:3250 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-26 13:04:31,501 INFO SystemMonitor:3250 [interfaces.py:start():190] Started cpu monitoring 2024-02-26 13:04:31,506 INFO SystemMonitor:3250 [interfaces.py:start():190] Started disk monitoring 2024-02-26 13:04:31,512 INFO SystemMonitor:3250 [interfaces.py:start():190] Started gpu monitoring 2024-02-26 13:04:31,519 INFO SystemMonitor:3250 [interfaces.py:start():190] Started memory monitoring 2024-02-26 13:04:31,521 INFO SystemMonitor:3250 [interfaces.py:start():190] Started network monitoring 2024-02-26 13:04:31,526 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: pause 2024-02-26 13:04:31,526 INFO HandlerThread:3250 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-02-26 13:04:31,526 INFO HandlerThread:3250 [system_monitor.py:finish():203] Stopping system monitor 2024-02-26 13:04:31,527 DEBUG SystemMonitor:3250 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-26 13:04:31,527 DEBUG SystemMonitor:3250 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-02-26 13:04:31,532 DEBUG SystemMonitor:3250 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-02-26 13:04:31,527 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined cpu monitor 2024-02-26 13:04:31,536 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined disk monitor 2024-02-26 13:04:31,543 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined gpu monitor 2024-02-26 13:04:31,544 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined memory monitor 2024-02-26 13:04:31,544 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined network monitor 2024-02-26 13:04:31,545 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:04:32,545 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:04:37,546 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:04:37,705 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:04:37,706 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:04:37,712 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:04:43,007 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:04:48,008 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:04:52,705 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:04:52,706 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:04:52,746 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:04:53,036 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:04:58,037 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:05:03,039 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:05:07,705 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:05:07,706 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:05:07,746 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:05:08,961 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:05:13,962 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:05:18,963 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:05:22,705 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:05:22,706 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:05:22,747 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:05:24,034 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:05:25,219 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: resume 2024-02-26 13:05:25,224 INFO HandlerThread:3250 [handler.py:handle_request_resume():699] starting system metrics thread 2024-02-26 13:05:25,225 INFO HandlerThread:3250 [system_monitor.py:start():194] Starting system monitor 2024-02-26 13:05:25,227 INFO SystemMonitor:3250 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-26 13:05:25,233 INFO SystemMonitor:3250 [interfaces.py:start():190] Started cpu monitoring 2024-02-26 13:05:25,238 INFO SystemMonitor:3250 [interfaces.py:start():190] Started disk monitoring 2024-02-26 13:05:25,248 INFO SystemMonitor:3250 [interfaces.py:start():190] Started gpu monitoring 2024-02-26 13:05:25,254 INFO SystemMonitor:3250 [interfaces.py:start():190] Started memory monitoring 2024-02-26 13:05:25,254 INFO SystemMonitor:3250 [interfaces.py:start():190] Started network monitoring 2024-02-26 13:05:25,269 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: pause 2024-02-26 13:05:25,269 INFO HandlerThread:3250 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-02-26 13:05:25,269 INFO HandlerThread:3250 [system_monitor.py:finish():203] Stopping system monitor 2024-02-26 13:05:25,270 DEBUG SystemMonitor:3250 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-26 13:05:25,276 DEBUG SystemMonitor:3250 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-02-26 13:05:25,270 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined cpu monitor 2024-02-26 13:05:25,278 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined disk monitor 2024-02-26 13:05:25,278 DEBUG SystemMonitor:3250 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-02-26 13:05:25,282 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined gpu monitor 2024-02-26 13:05:25,283 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined memory monitor 2024-02-26 13:05:25,283 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined network monitor 2024-02-26 13:05:25,284 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:05:29,285 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:05:34,286 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:05:37,706 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:05:37,706 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:05:37,712 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:05:39,972 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:05:44,973 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:05:49,974 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:05:52,706 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:05:52,707 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:05:52,746 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:05:54,982 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:05:59,983 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:06:04,984 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:06:07,706 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:06:07,707 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:06:07,746 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:06:10,036 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:06:15,037 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:06:20,038 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:06:22,706 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:06:22,707 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:06:22,747 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:06:25,990 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:06:30,991 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:06:35,992 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:06:37,706 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:06:37,707 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:06:37,747 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:06:41,969 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:06:46,970 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:06:51,971 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:06:52,706 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:06:52,707 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:06:52,747 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:06:57,055 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:07:02,056 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:07:07,057 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:07:07,707 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:07:07,707 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:07:07,747 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:07:13,000 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:07:18,001 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:07:22,707 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:07:22,707 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:07:22,747 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:07:23,040 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:07:28,041 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:07:33,042 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:07:37,707 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:07:37,708 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:07:37,747 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:07:38,961 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:07:43,962 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:07:48,964 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:07:51,004 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: resume 2024-02-26 13:07:51,005 INFO HandlerThread:3250 [handler.py:handle_request_resume():699] starting system metrics thread 2024-02-26 13:07:51,006 INFO HandlerThread:3250 [system_monitor.py:start():194] Starting system monitor 2024-02-26 13:07:51,006 INFO SystemMonitor:3250 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-26 13:07:51,007 INFO SystemMonitor:3250 [interfaces.py:start():190] Started cpu monitoring 2024-02-26 13:07:51,009 INFO SystemMonitor:3250 [interfaces.py:start():190] Started disk monitoring 2024-02-26 13:07:51,012 INFO SystemMonitor:3250 [interfaces.py:start():190] Started gpu monitoring 2024-02-26 13:07:51,015 INFO SystemMonitor:3250 [interfaces.py:start():190] Started memory monitoring 2024-02-26 13:07:51,015 INFO SystemMonitor:3250 [interfaces.py:start():190] Started network monitoring 2024-02-26 13:07:51,044 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: pause 2024-02-26 13:07:51,044 INFO HandlerThread:3250 [handler.py:handle_request_pause():708] stopping system metrics thread 2024-02-26 13:07:51,044 INFO HandlerThread:3250 [system_monitor.py:finish():203] Stopping system monitor 2024-02-26 13:07:51,045 DEBUG SystemMonitor:3250 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-26 13:07:51,045 DEBUG SystemMonitor:3250 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-02-26 13:07:51,045 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined cpu monitor 2024-02-26 13:07:51,046 DEBUG SystemMonitor:3250 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-02-26 13:07:51,048 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined disk monitor 2024-02-26 13:07:51,054 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined gpu monitor 2024-02-26 13:07:51,054 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined memory monitor 2024-02-26 13:07:51,055 INFO HandlerThread:3250 [interfaces.py:finish():202] Joined network monitor 2024-02-26 13:07:51,055 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:07:52,707 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:07:52,708 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:07:52,713 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:07:53,019 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: resume 2024-02-26 13:07:53,019 INFO HandlerThread:3250 [handler.py:handle_request_resume():699] starting system metrics thread 2024-02-26 13:07:53,019 INFO HandlerThread:3250 [system_monitor.py:start():194] Starting system monitor 2024-02-26 13:07:53,024 INFO SystemMonitor:3250 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-02-26 13:07:53,025 INFO SystemMonitor:3250 [interfaces.py:start():190] Started cpu monitoring 2024-02-26 13:07:53,026 INFO SystemMonitor:3250 [interfaces.py:start():190] Started disk monitoring 2024-02-26 13:07:53,028 INFO SystemMonitor:3250 [interfaces.py:start():190] Started gpu monitoring 2024-02-26 13:07:53,033 INFO SystemMonitor:3250 [interfaces.py:start():190] Started memory monitoring 2024-02-26 13:07:53,034 INFO SystemMonitor:3250 [interfaces.py:start():190] Started network monitoring 2024-02-26 13:07:54,089 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:07:59,090 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:08:04,091 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:08:07,707 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:08:07,708 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:08:07,713 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:08:09,982 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:08:14,983 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:08:19,984 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:08:22,707 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:08:22,708 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:08:22,748 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:08:25,023 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:08:30,024 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:08:35,025 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:08:37,707 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:08:37,708 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:08:37,748 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:08:41,004 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:08:46,005 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:08:51,006 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:08:52,713 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:08:52,719 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:08:52,748 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:08:53,035 DEBUG SystemMonitor:3250 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-02-26 13:08:53,056 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:08:56,058 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:08:56,265 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: partial_history 2024-02-26 13:08:56,270 DEBUG SenderThread:3250 [sender.py:send():382] send: metric 2024-02-26 13:08:56,271 DEBUG SenderThread:3250 [sender.py:send():382] send: metric 2024-02-26 13:08:56,271 DEBUG SenderThread:3250 [sender.py:send():382] send: metric 2024-02-26 13:08:56,271 DEBUG SenderThread:3250 [sender.py:send():382] send: metric 2024-02-26 13:08:56,271 DEBUG SenderThread:3250 [sender.py:send():382] send: history 2024-02-26 13:08:56,272 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: summary_record 2024-02-26 13:08:56,278 INFO SenderThread:3250 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-26 13:08:56,829 INFO Thread-12 :3250 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko/wandb/run-20240226_125646-ilpu1hh1/files/wandb-summary.json 2024-02-26 13:09:01,281 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:09:06,286 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:09:07,708 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:09:07,708 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:09:07,719 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:09:12,006 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:09:17,014 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:09:17,927 INFO Thread-12 :3250 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko/wandb/run-20240226_125646-ilpu1hh1/files/config.yaml 2024-02-26 13:09:22,447 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:09:22,708 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:09:22,708 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:09:22,748 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:09:23,037 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:09:28,038 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:09:33,039 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:09:37,708 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:09:37,709 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:09:37,749 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:09:39,039 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:09:42,744 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: partial_history 2024-02-26 13:09:42,747 DEBUG SenderThread:3250 [sender.py:send():382] send: history 2024-02-26 13:09:42,747 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: summary_record 2024-02-26 13:09:42,755 INFO SenderThread:3250 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-26 13:09:42,981 INFO Thread-12 :3250 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko/wandb/run-20240226_125646-ilpu1hh1/files/wandb-summary.json 2024-02-26 13:09:44,758 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:09:49,759 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:09:52,736 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:09:52,739 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:09:52,741 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:09:53,048 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:09:55,050 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:10:00,051 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:10:05,051 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:10:07,736 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:10:07,737 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:10:07,737 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:10:10,072 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:10:15,072 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:10:20,077 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:10:22,736 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:10:22,737 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:10:22,738 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:10:23,039 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:10:26,040 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:10:28,737 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: partial_history 2024-02-26 13:10:28,739 DEBUG SenderThread:3250 [sender.py:send():382] send: history 2024-02-26 13:10:28,740 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: summary_record 2024-02-26 13:10:28,750 INFO SenderThread:3250 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-26 13:10:29,141 INFO Thread-12 :3250 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko/wandb/run-20240226_125646-ilpu1hh1/files/wandb-summary.json 2024-02-26 13:10:31,755 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:10:36,759 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:10:37,736 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:10:37,738 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:10:37,739 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:10:42,090 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:10:47,091 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:10:52,091 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:10:52,737 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:10:52,739 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:10:52,742 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:10:53,041 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:10:58,042 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:11:03,043 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:11:07,738 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:11:07,738 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:11:07,739 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:11:09,002 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:11:10,274 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: partial_history 2024-02-26 13:11:10,279 DEBUG SenderThread:3250 [sender.py:send():382] send: history 2024-02-26 13:11:10,280 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: summary_record 2024-02-26 13:11:10,287 INFO SenderThread:3250 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-26 13:11:10,290 INFO Thread-12 :3250 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko/wandb/run-20240226_125646-ilpu1hh1/files/wandb-summary.json 2024-02-26 13:11:14,290 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:11:19,290 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:11:22,738 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:11:22,738 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:11:22,739 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:11:23,091 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:11:25,093 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:11:30,094 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:11:35,095 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:11:37,738 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:11:37,739 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:11:37,739 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:11:41,030 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:11:46,031 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:11:51,031 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:11:52,065 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: partial_history 2024-02-26 13:11:52,069 DEBUG SenderThread:3250 [sender.py:send():382] send: history 2024-02-26 13:11:52,069 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: summary_record 2024-02-26 13:11:52,076 INFO SenderThread:3250 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-26 13:11:52,433 INFO Thread-12 :3250 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko/wandb/run-20240226_125646-ilpu1hh1/files/wandb-summary.json 2024-02-26 13:11:52,737 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:11:52,738 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:11:52,739 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:11:53,041 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:11:56,044 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:12:01,045 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:12:06,046 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:12:07,738 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:12:07,738 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:12:07,739 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:12:11,065 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:12:16,066 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:12:21,067 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:12:22,738 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:12:22,739 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:12:22,739 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:12:23,042 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:12:27,043 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:12:32,045 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:12:37,046 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:12:37,738 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:12:37,738 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:12:37,739 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:12:42,082 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:12:47,083 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:12:52,084 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:12:52,737 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:12:52,743 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:12:52,743 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:12:53,063 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:12:58,071 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:12:59,657 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: partial_history 2024-02-26 13:12:59,659 DEBUG SenderThread:3250 [sender.py:send():382] send: history 2024-02-26 13:12:59,659 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: summary_record 2024-02-26 13:12:59,674 INFO SenderThread:3250 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-26 13:13:00,631 INFO Thread-12 :3250 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko/wandb/run-20240226_125646-ilpu1hh1/files/wandb-summary.json 2024-02-26 13:13:03,678 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:13:05,424 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: partial_history 2024-02-26 13:13:05,427 DEBUG SenderThread:3250 [sender.py:send():382] send: metric 2024-02-26 13:13:05,428 DEBUG SenderThread:3250 [sender.py:send():382] send: metric 2024-02-26 13:13:05,434 DEBUG SenderThread:3250 [sender.py:send():382] send: metric 2024-02-26 13:13:05,436 DEBUG SenderThread:3250 [sender.py:send():382] send: metric 2024-02-26 13:13:05,436 DEBUG SenderThread:3250 [sender.py:send():382] send: metric 2024-02-26 13:13:05,436 DEBUG SenderThread:3250 [sender.py:send():382] send: history 2024-02-26 13:13:05,438 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: summary_record 2024-02-26 13:13:05,446 INFO SenderThread:3250 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-02-26 13:13:05,644 INFO Thread-12 :3250 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS-Ko/wandb/run-20240226_125646-ilpu1hh1/files/wandb-summary.json 2024-02-26 13:13:07,738 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:13:07,739 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:13:07,740 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:13:08,985 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:13:13,986 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:13:18,987 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report 2024-02-26 13:13:22,738 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: internal_messages 2024-02-26 13:13:22,739 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: stop_status 2024-02-26 13:13:22,739 DEBUG SenderThread:3250 [sender.py:send_request():409] send_request: stop_status 2024-02-26 13:13:23,044 DEBUG SenderThread:3250 [sender.py:send():382] send: stats 2024-02-26 13:13:24,044 DEBUG HandlerThread:3250 [handler.py:handle_request():146] handle_request: status_report