| 2024-02-26 10:10:59,410 INFO StreamThr :5808 [internal.py:wandb_internal():86] W&B internal server running at pid: 5808, started at: 2024-02-26 10:10:59.405118 | |
| 2024-02-26 10:10:59,432 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status | |
| 2024-02-26 10:11:01,960 INFO WriterThread:5808 [datastore.py:open_for_write():87] open: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/run-z4vsrt4l.wandb | |
| 2024-02-26 10:11:01,966 DEBUG SenderThread:5808 [sender.py:send():382] send: header | |
| 2024-02-26 10:11:01,976 DEBUG SenderThread:5808 [sender.py:send():382] send: run | |
| 2024-02-26 10:11:02,447 INFO SenderThread:5808 [dir_watcher.py:__init__():211] watching files in: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files | |
| 2024-02-26 10:11:02,447 INFO SenderThread:5808 [sender.py:_start_run_threads():1136] run started: z4vsrt4l with start time 1708942259.407921 | |
| 2024-02-26 10:11:02,462 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-02-26 10:11:02,463 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: check_version | |
| 2024-02-26 10:11:02,575 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-02-26 10:11:02,585 DEBUG HandlerThread:5808 [system_info.py:__init__():27] System info init | |
| 2024-02-26 10:11:02,585 DEBUG HandlerThread:5808 [system_info.py:__init__():42] System info init done | |
| 2024-02-26 10:11:02,585 INFO HandlerThread:5808 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 10:11:02,585 INFO SystemMonitor:5808 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 10:11:02,586 INFO HandlerThread:5808 [system_monitor.py:probe():214] Collecting system info | |
| 2024-02-26 10:11:02,586 INFO SystemMonitor:5808 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 10:11:02,587 INFO SystemMonitor:5808 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 10:11:02,588 INFO SystemMonitor:5808 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 10:11:02,589 INFO SystemMonitor:5808 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 10:11:02,590 INFO SystemMonitor:5808 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 10:11:02,599 DEBUG HandlerThread:5808 [system_info.py:probe():151] Probing system | |
| 2024-02-26 10:11:02,605 DEBUG HandlerThread:5808 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-02-26 10:11:02,605 DEBUG HandlerThread:5808 [system_info.py:probe():199] Probing system done | |
| 2024-02-26 10:11:02,605 DEBUG HandlerThread:5808 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-02-26T10:11:02.599344', 'startedAt': '2024-02-26T10:10:59.351815', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Gemma-7B-Finetuning-Insurance.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1kan-kI251WUZEPoyknzcoQ3vjRSC1xpP', 'host': '30a0ffea74aa', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 1, 'cpu_count_logical': 2, 'cpu_freq': {'current': 2000.148, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.148, 'min': 0.0, 'max': 0.0}, {'current': 2000.148, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 78.1898422241211, 'used': 42.80147171020508}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 12.674789428710938}} | |
| 2024-02-26 10:11:02,605 INFO HandlerThread:5808 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-02-26 10:11:02,606 INFO HandlerThread:5808 [system_monitor.py:probe():227] Publishing system info | |
| 2024-02-26 10:11:02,626 INFO HandlerThread:5808 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-02-26 10:11:02,632 DEBUG SenderThread:5808 [sender.py:send():382] send: files | |
| 2024-02-26 10:11:02,633 INFO SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now | |
| 2024-02-26 10:11:02,659 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-02-26 10:11:02,659 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: python_packages | |
| 2024-02-26 10:11:02,660 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:11:02,661 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:11:02,667 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:11:02,915 DEBUG SenderThread:5808 [sender.py:send():382] send: telemetry | |
| 2024-02-26 10:11:02,915 DEBUG SenderThread:5808 [sender.py:send():382] send: config | |
| 2024-02-26 10:11:02,917 DEBUG SenderThread:5808 [sender.py:send():382] send: metric | |
| 2024-02-26 10:11:02,922 DEBUG SenderThread:5808 [sender.py:send():382] send: telemetry | |
| 2024-02-26 10:11:02,923 DEBUG SenderThread:5808 [sender.py:send():382] send: metric | |
| 2024-02-26 10:11:02,923 WARNING SenderThread:5808 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) | |
| 2024-02-26 10:11:02,923 DEBUG SenderThread:5808 [sender.py:send():382] send: telemetry | |
| 2024-02-26 10:11:03,260 INFO wandb-upload_0:5808 [upload_job.py:push():131] Uploaded file /tmp/tmpae1t1xluwandb/ddsashsh-wandb-metadata.json | |
| 2024-02-26 10:11:03,452 INFO Thread-12 :5808 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/requirements.txt | |
| 2024-02-26 10:11:03,453 INFO Thread-12 :5808 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-metadata.json | |
| 2024-02-26 10:11:04,924 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:11:09,925 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:11:14,926 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:11:17,660 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:11:17,661 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:11:17,661 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:11:20,845 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:11:25,845 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:11:30,853 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:11:31,500 INFO Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/config.yaml | |
| 2024-02-26 10:11:32,660 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:11:32,661 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:11:32,700 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:11:36,834 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:11:41,835 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:11:46,836 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:11:47,660 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:11:47,661 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:11:47,700 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:11:51,860 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:11:56,861 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:12:01,862 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:12:02,590 DEBUG SystemMonitor:5808 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 10:12:02,592 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:12:02,660 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:12:02,661 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:12:02,700 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:12:07,820 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:12:12,821 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:12:17,660 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:12:17,661 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:12:17,701 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:12:17,888 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:12:22,889 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:12:27,895 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:12:32,593 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:12:32,661 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:12:32,661 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:12:32,701 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:12:33,800 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:12:38,800 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:12:43,801 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:12:47,661 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:12:47,661 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:12:47,701 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:12:48,881 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:12:53,882 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:12:58,883 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:13:02,594 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:13:02,660 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:13:02,661 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:13:02,701 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:13:04,840 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:13:09,840 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:13:12,172 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 10:13:12,175 DEBUG SenderThread:5808 [sender.py:send():382] send: metric | |
| 2024-02-26 10:13:12,177 DEBUG SenderThread:5808 [sender.py:send():382] send: metric | |
| 2024-02-26 10:13:12,177 DEBUG SenderThread:5808 [sender.py:send():382] send: metric | |
| 2024-02-26 10:13:12,177 DEBUG SenderThread:5808 [sender.py:send():382] send: metric | |
| 2024-02-26 10:13:12,178 DEBUG SenderThread:5808 [sender.py:send():382] send: history | |
| 2024-02-26 10:13:12,178 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 10:13:12,184 INFO SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 10:13:12,689 INFO Thread-12 :5808 [dir_watcher.py:_on_file_created():271] file/dir created: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-summary.json | |
| 2024-02-26 10:13:15,187 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:13:17,691 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:13:17,692 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:13:17,728 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:13:20,832 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:13:25,833 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:13:30,834 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:13:32,594 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:13:32,673 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:13:32,674 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:13:32,674 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:13:35,872 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:13:36,754 INFO Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/config.yaml | |
| 2024-02-26 10:13:41,174 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:13:46,175 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:13:47,672 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:13:47,673 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:13:47,674 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:13:51,811 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:13:56,812 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:14:01,813 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:14:02,595 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:14:02,672 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:14:02,673 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:14:02,675 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:14:06,866 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:14:11,867 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:14:16,868 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:14:17,672 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:14:17,674 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:14:17,675 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:14:21,919 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:14:26,919 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:14:31,923 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:14:32,596 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:14:32,672 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:14:32,673 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:14:32,674 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:14:37,907 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:14:42,908 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:14:47,673 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:14:47,674 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:14:47,675 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:14:48,813 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 10:14:48,816 DEBUG SenderThread:5808 [sender.py:send():382] send: history | |
| 2024-02-26 10:14:48,817 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 10:14:48,821 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:14:48,828 INFO SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 10:14:48,916 INFO Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-summary.json | |
| 2024-02-26 10:14:53,839 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:14:58,840 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:15:02,597 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:15:02,673 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:15:02,674 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:15:02,676 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:15:03,943 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:15:08,944 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:15:13,945 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:15:17,673 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:15:17,675 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:15:17,675 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:15:19,896 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:15:24,897 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:15:29,899 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:15:32,597 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:15:32,673 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:15:32,674 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:15:32,675 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:15:34,904 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:15:39,907 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:15:44,908 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:15:47,673 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:15:47,674 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:15:47,675 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:15:50,908 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:15:55,912 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:16:00,913 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:16:02,598 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:16:02,673 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:16:02,675 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:16:02,675 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:16:05,915 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:16:10,250 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 10:16:10,253 DEBUG SenderThread:5808 [sender.py:send():382] send: history | |
| 2024-02-26 10:16:10,253 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 10:16:10,260 INFO SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 10:16:11,151 INFO Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-summary.json | |
| 2024-02-26 10:16:11,262 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:16:16,263 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:16:17,709 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:16:17,718 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:16:17,720 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:16:21,883 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:16:26,885 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:16:31,886 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:16:32,599 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:16:32,709 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:16:32,710 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:16:32,749 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:16:37,850 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:16:42,851 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:16:47,709 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:16:47,710 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:16:47,750 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:16:47,944 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:16:52,945 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:16:57,945 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:17:02,600 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:17:02,710 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:17:02,712 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:17:02,751 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:17:03,931 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:17:08,932 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:17:13,933 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:17:17,711 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:17:17,711 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:17:17,751 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:17:19,866 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:17:24,866 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:17:29,868 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:17:32,601 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:17:32,711 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:17:32,711 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:17:32,751 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:17:34,947 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:17:39,948 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:17:44,949 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:17:47,711 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:17:47,712 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:17:47,751 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:17:50,933 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:17:55,934 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:18:00,934 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:18:02,602 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:18:02,711 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:18:02,712 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:18:02,751 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:18:05,937 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:18:08,893 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 10:18:08,898 DEBUG SenderThread:5808 [sender.py:send():382] send: history | |
| 2024-02-26 10:18:08,898 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 10:18:08,905 INFO SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 10:18:09,454 INFO Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-summary.json | |
| 2024-02-26 10:18:11,909 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:18:16,908 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:18:17,711 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:18:17,712 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:18:17,720 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:18:21,912 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:18:26,913 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:18:31,918 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:18:32,602 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:18:32,711 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:18:32,712 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:18:32,752 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:18:37,899 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:18:42,900 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:18:47,711 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:18:47,712 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:18:47,752 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:18:48,880 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:18:53,873 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:18:58,874 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:19:02,603 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:19:02,712 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:19:02,712 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:19:02,752 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:19:04,874 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:19:09,875 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:19:14,875 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:19:17,712 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:19:17,712 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:19:17,752 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:19:19,949 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:19:24,950 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:19:29,951 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:19:32,604 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:19:32,712 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:19:32,712 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:19:32,752 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:19:35,901 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:19:40,902 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:19:45,903 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:19:47,712 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:19:47,713 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:19:47,752 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:19:50,950 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:19:55,951 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:20:00,464 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 10:20:00,470 DEBUG SenderThread:5808 [sender.py:send():382] send: history | |
| 2024-02-26 10:20:00,470 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 10:20:00,477 INFO SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 10:20:00,698 INFO Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-summary.json | |
| 2024-02-26 10:20:01,478 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:20:02,605 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:20:05,970 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:20:05,970 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:20:06,128 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:20:07,183 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:20:12,184 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:20:17,189 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:20:20,925 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:20:20,928 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:20:20,965 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:20:23,116 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:20:28,117 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:20:32,606 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:20:33,607 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:20:35,925 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:20:35,926 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:20:35,965 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:20:39,079 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:20:44,080 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:20:49,081 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:20:50,925 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:20:50,926 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:20:50,966 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:20:54,169 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:20:59,170 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:21:02,607 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:21:04,610 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:21:05,925 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:21:05,926 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:21:05,966 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:21:10,168 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:21:15,168 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:21:20,169 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:21:20,925 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:21:20,926 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:21:20,966 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:21:26,132 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:21:30,342 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 10:21:30,343 DEBUG SenderThread:5808 [sender.py:send():382] send: history | |
| 2024-02-26 10:21:30,344 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 10:21:30,349 INFO SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 10:21:30,906 INFO Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-summary.json | |
| 2024-02-26 10:21:31,351 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:21:32,608 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:21:36,012 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:21:36,014 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:21:36,014 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:21:37,213 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:21:38,703 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 10:21:38,706 DEBUG SenderThread:5808 [sender.py:send():382] send: metric | |
| 2024-02-26 10:21:38,707 DEBUG SenderThread:5808 [sender.py:send():382] send: metric | |
| 2024-02-26 10:21:38,707 DEBUG SenderThread:5808 [sender.py:send():382] send: metric | |
| 2024-02-26 10:21:38,708 DEBUG SenderThread:5808 [sender.py:send():382] send: metric | |
| 2024-02-26 10:21:38,708 DEBUG SenderThread:5808 [sender.py:send():382] send: metric | |
| 2024-02-26 10:21:38,709 DEBUG SenderThread:5808 [sender.py:send():382] send: history | |
| 2024-02-26 10:21:38,710 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 10:21:38,729 INFO SenderThread:5808 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 10:21:38,924 INFO Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/wandb-summary.json | |
| 2024-02-26 10:21:42,731 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:21:47,732 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:21:50,553 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 10:21:50,554 INFO HandlerThread:5808 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 10:21:50,554 INFO HandlerThread:5808 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 10:21:50,557 DEBUG SystemMonitor:5808 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 10:21:50,565 DEBUG SystemMonitor:5808 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 10:21:50,557 INFO HandlerThread:5808 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 10:21:50,567 INFO HandlerThread:5808 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 10:21:50,576 INFO HandlerThread:5808 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 10:21:50,576 INFO HandlerThread:5808 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 10:21:50,577 INFO HandlerThread:5808 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 10:21:50,578 DEBUG SenderThread:5808 [sender.py:send():382] send: stats | |
| 2024-02-26 10:21:51,012 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 10:21:51,013 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 10:21:51,014 DEBUG SenderThread:5808 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 10:21:53,235 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:21:58,240 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 10:21:58,560 DEBUG HandlerThread:5808 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 10:21:58,561 INFO HandlerThread:5808 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 10:21:58,561 INFO HandlerThread:5808 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 10:21:58,561 INFO SystemMonitor:5808 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 10:21:58,563 INFO SystemMonitor:5808 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 10:21:58,567 INFO SystemMonitor:5808 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 10:21:58,568 INFO SystemMonitor:5808 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 10:21:58,574 INFO SystemMonitor:5808 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 10:21:58,574 INFO SystemMonitor:5808 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 10:21:58,975 INFO Thread-12 :5808 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/gdrive/MyDrive/LLM/Gemma-7B-Finetuning-JCS/wandb/run-20240226_101059-z4vsrt4l/files/config.yaml | |