2024-12-26 22:48:36,731 INFO StreamThr :840649 [internal.py:wandb_internal():85] W&B internal server running at pid: 840649, started at: 2024-12-26 22:48:36.730694 2024-12-26 22:48:36,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status 2024-12-26 22:48:36,741 INFO WriterThread:840649 [datastore.py:open_for_write():87] open: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/run-fta6xses.wandb 2024-12-26 22:48:36,742 DEBUG SenderThread:840649 [sender.py:send():379] send: header 2024-12-26 22:48:36,766 DEBUG SenderThread:840649 [sender.py:send():379] send: run 2024-12-26 22:48:36,989 INFO SenderThread:840649 [dir_watcher.py:__init__():211] watching files in: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files 2024-12-26 22:48:36,989 INFO SenderThread:840649 [sender.py:_start_run_threads():1188] run started: fta6xses with start time 1735253316.732709 2024-12-26 22:48:37,012 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: check_version 2024-12-26 22:48:37,012 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: check_version 2024-12-26 22:48:37,086 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: run_start 2024-12-26 22:48:37,116 DEBUG HandlerThread:840649 [system_info.py:__init__():26] System info init 2024-12-26 22:48:37,117 DEBUG HandlerThread:840649 [system_info.py:__init__():41] System info init done 2024-12-26 22:48:37,117 INFO HandlerThread:840649 [system_monitor.py:start():194] Starting system monitor 2024-12-26 22:48:37,117 INFO SystemMonitor:840649 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-12-26 22:48:37,117 INFO HandlerThread:840649 [system_monitor.py:probe():214] Collecting system info 2024-12-26 22:48:37,118 INFO SystemMonitor:840649 [interfaces.py:start():188] Started cpu monitoring 2024-12-26 22:48:37,118 INFO SystemMonitor:840649 [interfaces.py:start():188] Started disk monitoring 2024-12-26 22:48:37,119 INFO SystemMonitor:840649 [interfaces.py:start():188] Started gpu monitoring 2024-12-26 22:48:37,120 INFO SystemMonitor:840649 [interfaces.py:start():188] Started memory monitoring 2024-12-26 22:48:37,122 INFO SystemMonitor:840649 [interfaces.py:start():188] Started network monitoring 2024-12-26 22:48:37,183 DEBUG HandlerThread:840649 [system_info.py:probe():152] Probing system 2024-12-26 22:48:37,186 DEBUG HandlerThread:840649 [system_info.py:_probe_git():137] Probing git 2024-12-26 22:48:37,207 DEBUG HandlerThread:840649 [system_info.py:_probe_git():145] Probing git done 2024-12-26 22:48:37,207 DEBUG HandlerThread:840649 [system_info.py:probe():200] Probing system done 2024-12-26 22:48:37,207 DEBUG HandlerThread:840649 [system_monitor.py:probe():223] {'os': 'Linux-6.8.0-48-generic-x86_64-with-glibc2.39', 'python': '3.10.0', 'heartbeatAt': '2024-12-26T22:48:37.183149', 'startedAt': '2024-12-26T22:48:36.715544', 'docker': None, 'cuda': None, 'args': ('--model_family', 'llama', '--apply_instruct_masks', '--token_scaled_loss', '--seq_parallel_size', '8', '--report_to', 'wandb', '--do_train', '--model_name_or_path', '/scratch3/workspace/ctpham_umass_edu-ft/models/Llama-3-8B-ProLong-512k-Base/', '--config_name', '/scratch3/workspace/ctpham_umass_edu-ft/models/Llama-3-8B-ProLong-512k-Base/', '--tokenizer_name', '/scratch3/workspace/ctpham_umass_edu-ft/models/Llama-3-8B-ProLong-512k-Base/', '--run_name', 'balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_', '--output_dir', '/scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_', '--config_overrides_json', '', '--gradient_accumulation_steps', '2', '--per_device_train_batch_size', '1', '--bf16', '--learning_rate', '1e-6', '--min_lr_ratio', '0.1', '--lr_scheduler_type', 'cosine', '--max_grad_norm', '1.0', '--adam_beta1', '0.9', '--adam_beta2', '0.95', '--weight_decay', '0.1', '--warmup_ratio', '0.05', '--optim', 'adamw_torch', '--logging_steps', '1', '--log_level', 'info', '--save_steps', '200', '--dataloader_num_workers', '1', '--disable_tqdm', 'true', '--use_fast_tokenizer', 'false', '--remove_unused_columns', 'false', '--ddp_find_unused_parameters', 'false', '--fsdp', 'auto_wrap offload', '--gradient_checkpointing', '--tokenized_mds_train', '/work/pi_miyyer_umass_edu/ctpham/BookClaim-dev/data/ft/bookclaim_balanced_pack_complete', '--cuda_empty_cache', '--num_train_epochs', '1'), 'state': 'running', 'program': '/work/pi_miyyer_umass_edu/ctpham/BookClaim-dev/prolong-final/finetune.py', 'codePathLocal': 'finetune.py', 'codePath': 'prolong-final/finetune.py', 'git': {'remote': 'https://github.com/chtmp223/BookGen-dev.git', 'commit': '438e68ae65a707227d28baa427c5f6fdb0932c7b'}, 'email': 'ctpham@umass.edu', 'root': '/work/pi_miyyer_umass_edu/ctpham/BookClaim-dev', 'host': 'gpu021', 'username': 'ctpham_umass_edu', 'executable': '/scratch3/workspace/ctpham_umass_edu-ft/envs/prolong-final/bin/python3.10', 'cpu_count': 112, 'cpu_count_logical': 112, 'cpu_freq': {'current': 935.5062678571428, 'min': 800.0, 'max': 3800.0}, 'cpu_freq_per_core': [{'current': 776.892, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 783.257, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 785.598, 'min': 800.0, 'max': 3800.0}, {'current': 791.295, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 783.585, 'min': 800.0, 'max': 3800.0}, {'current': 900.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2000.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1500.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1984.654, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2400.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1868.877, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1100.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1100.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 785.28, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 785.936, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 784.409, 'min': 800.0, 'max': 3800.0}, {'current': 2500.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 783.236, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}], 'disk': {'/': {'total': 438.487850189209, 'used': 16.266197204589844}}, 'gpu': 'NVIDIA A100-SXM4-80GB', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}], 'memory': {'total': 2015.3287200927734}} 2024-12-26 22:48:37,207 INFO HandlerThread:840649 [system_monitor.py:probe():224] Finished collecting system info 2024-12-26 22:48:37,207 INFO HandlerThread:840649 [system_monitor.py:probe():227] Publishing system info 2024-12-26 22:48:37,207 DEBUG HandlerThread:840649 [system_info.py:_save_conda():209] Saving list of conda packages installed into the current environment 2024-12-26 22:48:37,991 INFO Thread-12 :840649 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/conda-environment.yaml 2024-12-26 22:48:44,491 DEBUG HandlerThread:840649 [system_info.py:_save_conda():224] Saving conda packages done 2024-12-26 22:48:44,494 INFO HandlerThread:840649 [system_monitor.py:probe():229] Finished publishing system info 2024-12-26 22:48:44,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:48:44,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-26 22:48:44,501 DEBUG SenderThread:840649 [sender.py:send():379] send: files 2024-12-26 22:48:44,501 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2024-12-26 22:48:44,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: python_packages 2024-12-26 22:48:44,772 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: python_packages 2024-12-26 22:48:44,772 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:48:44,773 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:48:44,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:48:44,907 INFO wandb-upload_0:840649 [upload_job.py:push():130] Uploaded file /tmp/tmpbzfroc7jwandb/qc3elvv1-wandb-metadata.json 2024-12-26 22:48:44,930 DEBUG SenderThread:840649 [sender.py:send():379] send: telemetry 2024-12-26 22:48:44,931 DEBUG SenderThread:840649 [sender.py:send():379] send: config 2024-12-26 22:48:44,931 DEBUG SenderThread:840649 [sender.py:send():379] send: telemetry 2024-12-26 22:48:44,931 DEBUG SenderThread:840649 [sender.py:send():379] send: metric 2024-12-26 22:48:44,931 DEBUG SenderThread:840649 [sender.py:send():379] send: telemetry 2024-12-26 22:48:44,931 DEBUG SenderThread:840649 [sender.py:send():379] send: metric 2024-12-26 22:48:44,931 WARNING SenderThread:840649 [sender.py:send_metric():1405] Seen metric with glob (shouldn't happen) 2024-12-26 22:48:44,931 DEBUG SenderThread:840649 [sender.py:send():379] send: telemetry 2024-12-26 22:48:44,931 DEBUG SenderThread:840649 [sender.py:send():379] send: telemetry 2024-12-26 22:48:44,931 DEBUG SenderThread:840649 [sender.py:send():379] send: config 2024-12-26 22:48:44,993 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/conda-environment.yaml 2024-12-26 22:48:44,994 INFO Thread-12 :840649 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:48:44,994 INFO Thread-12 :840649 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-metadata.json 2024-12-26 22:48:44,994 INFO Thread-12 :840649 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/requirements.txt 2024-12-26 22:48:45,773 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:48:46,773 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:48:46,995 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:48:47,773 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:48:47,933 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:48:48,773 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:48:49,773 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:48:50,773 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:48:51,774 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:48:52,774 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:48:53,773 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:48:53,774 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:48:54,774 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:48:55,774 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:48:56,774 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:48:57,774 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:48:58,774 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:48:58,967 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:48:59,004 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:48:59,772 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:48:59,772 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:48:59,812 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:00,774 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:01,774 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:02,774 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:03,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:04,773 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:49:04,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:05,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:06,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:07,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:08,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:09,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:49:09,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:10,012 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/config.yaml 2024-12-26 22:49:10,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:11,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:12,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:13,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:14,772 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:49:14,772 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:49:14,812 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:15,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:15,921 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:49:16,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:17,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:18,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:19,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:20,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:21,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:49:21,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:22,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:23,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:24,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:25,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:26,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:26,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:49:27,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:28,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:29,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:49:29,625 DEBUG SenderThread:840649 [sender.py:send():379] send: metric 2024-12-26 22:49:29,625 DEBUG SenderThread:840649 [sender.py:send():379] send: metric 2024-12-26 22:49:29,626 DEBUG SenderThread:840649 [sender.py:send():379] send: metric 2024-12-26 22:49:29,626 DEBUG SenderThread:840649 [sender.py:send():379] send: metric 2024-12-26 22:49:29,626 DEBUG SenderThread:840649 [sender.py:send():379] send: metric 2024-12-26 22:49:29,626 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:49:29,626 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:49:29,631 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:49:30,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:49:30,029 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:49:30,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:30,037 INFO Thread-12 :840649 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:49:31,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:31,038 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:49:32,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:32,105 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:49:33,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:34,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:35,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:36,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:37,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:37,122 DEBUG SystemMonitor:840649 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-12-26 22:49:37,124 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:49:37,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:49:38,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:39,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:40,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:41,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:42,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:43,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:43,034 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:49:44,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:44,046 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/config.yaml 2024-12-26 22:49:44,773 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:49:44,774 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:49:45,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:46,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:47,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:48,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:49,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:49:49,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:50,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:51,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:52,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:53,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:54,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:55,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:49:55,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:56,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:57,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:58,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:59,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:49:59,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:49:59,777 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:50:00,034 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:00,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:50:00,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:50:00,674 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:50:00,674 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:50:00,676 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:50:01,071 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:50:01,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:02,083 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:03,072 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:50:03,083 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:04,083 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:05,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:50:05,083 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:06,083 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:07,083 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:07,126 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:50:08,083 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:09,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:10,083 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:11,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:50:11,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:12,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:13,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:14,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:14,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:50:14,777 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:50:15,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:16,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:17,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:50:17,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:18,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:19,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:20,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:21,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:22,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:50:22,085 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:23,085 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:24,085 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:25,085 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:26,085 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:27,085 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:28,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:50:28,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:29,092 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:29,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:50:29,786 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:50:30,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:31,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:32,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:33,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:50:33,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:33,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:50:33,350 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:50:33,350 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:50:33,356 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:50:34,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:34,103 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:50:35,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:36,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:36,104 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:50:37,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:37,130 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:50:38,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:38,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:50:39,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:40,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:41,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:42,096 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:43,096 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:44,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:50:44,096 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:44,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:50:44,779 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:50:45,233 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:46,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:47,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:48,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:49,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:49,232 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:50:50,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:51,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:52,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:53,230 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:54,230 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:55,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:50:55,230 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:56,230 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:57,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:58,235 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:59,239 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:50:59,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:50:59,784 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:51:00,238 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:00,242 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:51:01,236 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:02,236 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:02,632 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:51:02,633 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:51:02,633 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:51:02,635 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:51:03,134 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:51:03,236 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:04,242 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:05,241 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:06,136 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:51:06,235 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:51:06,241 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:07,135 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:51:07,241 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:08,241 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:09,241 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:10,241 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:11,241 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:12,236 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:51:12,242 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:13,242 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:14,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:14,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:51:14,781 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:51:15,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:16,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:17,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:18,236 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:51:18,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:19,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:20,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:21,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:22,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:23,236 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:51:23,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:24,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:25,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:26,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:27,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:28,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:51:28,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:29,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:29,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:51:29,782 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:51:30,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:30,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:51:30,496 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:51:30,496 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:51:30,501 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:51:31,163 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:51:31,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:32,164 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:51:32,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:33,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:33,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:51:34,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:35,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:36,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:37,139 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:51:37,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:38,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:39,141 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:51:39,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:40,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:41,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:42,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:43,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:44,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:51:44,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:44,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:51:44,782 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:51:45,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:46,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:47,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:48,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:49,239 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:51:49,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:50,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:51,319 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:52,319 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:53,319 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:54,319 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:55,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:51:55,319 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:56,328 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:57,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:58,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:59,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:51:59,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:51:59,789 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:52:00,242 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:52:00,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:01,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:01,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:52:01,342 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:52:01,342 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:52:01,345 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:52:02,193 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:52:02,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:03,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:04,194 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:52:04,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:05,242 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:52:05,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:06,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:07,143 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:52:07,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:08,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:09,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:10,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:11,242 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:52:11,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:12,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:13,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:14,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:14,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:52:14,786 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:52:15,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:16,242 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:52:16,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:17,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:18,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:19,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:20,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:21,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:22,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:52:22,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:23,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:24,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:25,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:26,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:27,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:52:27,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:28,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:29,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:29,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:52:29,790 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:52:30,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:31,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:52:31,214 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:52:31,215 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:52:31,219 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:52:31,228 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:52:31,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:32,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:52:32,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:33,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:34,230 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:52:34,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:35,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:36,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:37,147 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:52:37,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:38,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:52:38,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:39,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:40,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:41,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:42,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:43,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:52:43,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:44,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:44,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:52:44,790 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:52:45,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:46,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:47,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:48,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:52:48,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:49,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:50,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:51,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:52,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:53,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:54,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:52:54,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:55,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:56,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:57,629 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:58,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:59,247 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:52:59,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:52:59,794 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:52:59,795 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:53:00,632 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:01,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:53:01,259 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:53:01,259 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:53:01,263 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:53:01,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:02,256 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:53:02,257 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:53:02,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:03,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:04,299 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:53:04,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:05,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:06,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:07,151 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:53:07,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:08,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:09,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:10,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:53:10,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:11,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:12,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:13,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:14,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:14,794 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:53:14,795 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:53:15,246 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:53:15,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:16,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:17,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:18,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:19,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:20,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:21,246 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:53:21,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:22,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:23,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:24,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:25,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:26,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:27,246 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:53:27,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:28,686 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:29,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:29,796 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:53:29,796 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:53:30,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:31,681 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:32,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:53:32,678 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:33,678 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:34,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:53:34,120 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:53:34,120 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:53:34,126 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:53:34,295 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:53:34,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:35,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:36,297 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:53:36,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:37,155 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:53:37,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:38,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:53:38,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:39,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:40,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:41,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:42,683 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:43,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:53:43,683 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:44,683 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:44,796 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:53:44,797 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:53:45,683 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:46,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:47,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:48,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:49,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:53:49,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:50,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:51,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:52,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:53,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:54,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:53:54,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:55,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:56,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:57,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:58,731 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:59,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:53:59,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:53:59,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:53:59,803 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:54:00,732 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:01,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:02,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:54:02,487 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:54:02,488 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:54:02,489 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:54:02,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:03,326 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:54:03,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:04,326 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:54:04,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:54:04,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:05,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:06,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:07,158 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:54:07,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:08,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:09,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:10,161 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:54:10,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:11,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:12,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:13,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:14,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:54:14,799 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:54:14,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:15,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:54:15,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:16,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:17,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:18,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:19,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:20,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:54:20,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:21,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:22,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:23,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:24,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:25,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:54:25,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:26,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:27,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:28,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:29,799 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:54:29,803 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:54:29,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:30,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:54:30,921 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:31,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:32,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:54:32,123 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:54:32,123 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:54:32,124 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:54:32,349 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:54:32,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:33,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:34,351 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:54:34,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:35,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:54:35,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:36,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:37,162 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:54:37,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:38,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:39,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:40,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:41,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:54:41,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:42,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:43,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:44,799 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:54:44,799 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:54:44,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:45,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:46,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:47,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:54:47,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:48,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:49,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:50,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:51,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:52,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:54:52,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:53,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:54,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:55,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:56,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:57,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:54:57,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:58,925 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:54:59,804 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:54:59,805 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:54:59,928 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:00,926 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:01,925 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:02,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:55:02,925 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:03,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:55:03,521 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:55:03,521 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:55:03,523 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:55:03,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:04,384 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:55:04,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:05,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:06,386 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:55:06,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:07,166 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:55:07,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:08,168 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:55:08,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:09,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:10,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:11,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:12,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:13,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:55:13,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:14,804 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:55:14,805 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:55:14,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:15,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:16,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:17,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:18,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:55:18,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:19,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:20,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:21,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:22,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:23,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:24,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:55:24,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:25,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:26,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:27,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:28,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:29,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:55:29,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:55:29,810 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:55:29,946 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:30,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:31,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:32,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:33,949 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:34,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:55:34,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:35,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:55:35,177 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:55:35,178 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:55:35,179 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:55:35,412 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:55:35,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:36,413 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:55:36,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:37,170 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:55:37,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:38,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:39,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:40,173 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:55:40,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:41,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:42,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:43,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:44,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:55:44,807 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:55:44,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:45,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:55:45,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:46,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:47,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:48,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:49,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:50,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:55:50,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:51,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:52,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:53,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:54,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:55,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:56,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:55:56,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:57,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:58,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:55:59,814 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:55:59,814 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:55:59,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:00,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:01,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:56:01,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:02,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:03,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:04,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:05,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:56:05,125 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:56:05,125 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:56:05,126 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:56:05,446 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:56:05,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:06,447 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:56:06,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:56:06,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:07,174 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:56:07,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:08,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:09,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:10,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:11,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:12,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:56:12,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:13,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:14,811 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:56:14,811 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:56:14,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:15,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:16,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:17,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:18,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:56:18,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:19,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:20,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:21,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:22,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:23,958 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:24,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:56:24,958 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:25,958 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:26,958 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:27,958 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:28,958 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:29,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:56:29,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:56:29,825 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:56:29,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:30,972 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:31,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:32,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:33,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:34,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:56:34,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:35,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:56:35,185 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:56:35,185 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:56:35,186 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:56:35,478 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:56:35,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:36,479 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:56:36,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:37,179 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:56:37,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:38,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:39,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:40,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:56:40,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:41,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:42,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:43,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:44,814 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:56:44,815 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:56:44,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:45,595 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:56:45,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:46,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:47,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:48,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:49,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:50,596 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:56:50,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:51,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:52,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:53,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:54,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:55,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:56,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:56:56,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:57,972 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:58,965 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:56:59,828 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:56:59,828 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:56:59,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:00,977 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:01,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:57:01,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:02,588 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:57:02,589 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:57:02,589 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:57:02,590 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:57:03,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:03,506 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:57:04,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:04,507 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:57:05,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:06,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:07,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:07,183 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:57:07,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:57:08,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:09,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:10,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:11,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:12,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:12,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:57:13,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:14,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:14,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:57:14,816 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:57:15,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:16,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:17,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:17,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:57:18,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:19,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:20,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:21,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:22,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:23,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:23,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:57:24,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:25,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:26,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:27,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:28,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:28,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:57:29,008 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:29,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:57:29,821 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:57:30,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:31,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:32,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:33,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:33,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:57:33,988 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:57:33,990 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:57:33,990 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:57:33,991 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:57:34,399 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:34,533 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:57:35,395 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:36,395 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:36,535 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:57:37,186 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:57:37,396 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:38,396 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:39,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:57:39,396 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:40,396 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:41,396 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:42,396 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:43,396 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:44,396 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:44,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:57:44,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:57:44,818 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:57:45,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:46,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:47,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:48,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:49,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:49,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:57:50,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:51,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:52,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:53,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:54,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:55,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:55,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:57:56,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:57,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:58,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:59,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:57:59,821 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:57:59,822 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:58:00,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:00,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:58:01,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:01,686 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:58:01,688 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:58:01,688 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:58:01,690 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:58:02,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:02,558 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:58:02,558 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:58:03,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:04,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:05,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:06,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:06,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:58:07,190 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:58:07,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:08,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:09,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:10,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:11,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:12,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:12,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:58:13,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:14,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:14,821 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:58:14,822 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:58:15,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:16,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:17,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:17,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:58:18,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:19,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:20,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:21,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:22,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:23,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:23,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:58:24,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:25,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:26,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:27,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:28,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:28,623 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:58:29,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:29,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:58:29,503 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:58:29,503 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:58:29,505 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:58:29,585 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:58:29,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:58:29,916 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:58:30,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:30,586 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:58:31,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:32,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:33,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:34,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:34,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:58:35,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:36,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:37,194 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:58:37,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:38,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:39,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:40,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:58:40,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:41,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:42,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:43,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:44,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:44,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:58:44,912 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:58:45,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:45,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:58:46,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:47,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:48,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:49,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:50,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:51,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:51,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:58:52,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:53,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:54,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:55,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:56,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:56,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:58:57,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:58,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:59,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:58:59,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:58:59,915 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:59:00,471 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:59:00,472 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:59:00,472 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:59:00,473 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:59:00,614 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:59:00,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:01,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:01,895 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:59:02,615 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:59:02,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:03,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:04,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:05,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:06,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:07,196 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:59:07,198 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:59:07,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:08,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:09,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:10,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:11,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:12,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:12,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:59:13,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:14,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:14,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:59:14,912 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:59:15,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:16,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:17,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:17,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:59:18,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:19,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:20,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:21,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:22,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:23,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:59:23,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:24,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:25,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:26,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:27,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:28,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:59:28,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:29,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:29,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:59:29,914 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:59:30,908 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:31,900 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:32,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:33,900 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:59:33,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:34,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 22:59:34,470 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 22:59:34,470 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 22:59:34,472 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 22:59:34,646 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 22:59:34,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:35,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:36,647 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 22:59:36,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:37,201 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 22:59:37,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:38,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:39,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:59:39,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:40,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:41,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:42,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:43,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:44,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:59:44,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:44,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:59:44,915 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 22:59:45,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:46,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:48,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:49,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:49,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:59:50,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:51,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:52,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:53,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:54,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:54,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:59:55,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:56,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:57,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:58,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:59,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 22:59:59,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 22:59:59,927 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 22:59:59,931 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:00:00,142 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:01,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:02,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:03,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:04,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:04,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:00:05,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:05,679 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:00:05,680 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:00:05,680 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:00:05,682 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:00:06,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:06,673 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:00:06,673 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:00:07,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:07,205 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:00:08,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:09,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:10,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:10,207 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:00:11,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:12,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:13,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:14,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:14,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:00:14,919 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:00:15,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:16,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:00:16,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:17,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:18,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:19,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:20,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:21,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:00:21,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:22,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:23,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:24,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:25,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:26,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:00:26,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:27,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:28,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:29,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:29,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:00:29,919 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:00:30,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:31,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:32,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:00:32,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:33,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:34,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:35,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:36,155 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:37,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:00:37,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:37,210 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:00:38,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:38,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:00:38,266 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:00:38,267 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:00:38,268 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:00:38,697 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:00:39,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:40,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:40,699 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:00:41,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:42,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:43,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:00:43,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:44,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:44,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:00:44,919 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:00:45,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:46,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:47,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:48,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:49,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:00:49,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:50,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:51,382 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:52,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:53,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:54,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:00:54,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:55,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:56,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:57,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:58,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:59,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:00:59,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:00:59,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:00:59,919 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:01:00,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:01,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:02,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:03,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:04,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:01:04,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:05,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:06,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:07,212 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:01:07,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:08,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:08,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:01:08,722 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:01:08,723 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:01:08,726 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:01:09,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:09,720 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:01:09,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:01:10,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:10,721 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:01:11,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:12,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:13,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:14,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:14,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:01:14,919 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:01:15,066 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:01:15,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:16,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:17,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:18,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:19,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:20,096 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:01:20,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:21,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:22,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:23,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:24,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:25,097 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:01:25,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:26,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:27,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:28,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:29,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:29,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:01:29,919 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:01:30,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:31,086 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:01:31,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:32,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:33,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:34,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:35,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:36,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:01:36,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:37,212 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:01:37,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:37,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:01:37,603 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:01:37,603 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:01:37,605 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:01:37,753 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:01:38,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:38,754 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:01:39,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:40,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:41,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:42,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:01:42,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:43,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:44,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:44,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:01:44,920 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:01:45,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:46,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:47,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:48,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:01:48,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:49,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:50,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:51,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:52,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:53,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:01:53,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:54,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:55,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:56,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:57,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:58,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:01:58,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:59,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:01:59,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:01:59,920 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:02:00,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:01,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:02,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:03,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:04,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:02:04,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:05,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:06,319 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:07,219 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:02:07,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:08,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:09,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:02:09,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:10,234 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:02:10,235 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:02:10,235 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:02:10,237 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:02:10,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:10,784 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:02:11,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:12,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:12,786 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:02:13,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:14,386 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:14,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:02:14,920 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:02:15,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:02:15,386 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:16,386 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:17,386 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:18,386 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:19,386 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:20,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:02:20,386 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:21,386 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:22,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:23,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:24,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:25,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:02:25,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:26,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:27,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:28,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:29,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:29,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:02:29,920 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:02:30,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:31,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:02:31,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:32,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:33,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:34,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:35,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:36,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:02:36,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:37,219 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:02:37,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:38,496 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:39,497 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:40,106 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:02:40,107 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:02:40,108 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:02:40,110 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:02:40,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:40,812 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:02:40,813 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:02:41,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:02:41,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:42,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:43,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:44,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:44,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:02:44,920 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:02:45,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:46,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:47,124 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:02:47,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:48,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:49,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:50,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:51,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:52,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:53,124 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:02:53,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:54,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:55,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:56,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:57,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:58,124 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:02:58,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:59,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:02:59,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:02:59,920 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:03:00,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:01,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:02,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:03,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:04,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:03:04,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:05,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:06,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:07,228 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:03:07,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:08,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:09,235 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:03:09,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:10,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:11,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:12,476 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:03:12,477 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:03:12,478 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:03:12,483 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:03:12,843 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:03:13,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:14,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:03:14,538 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:14,845 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:03:14,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:03:14,920 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:03:15,538 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:16,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:17,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:18,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:19,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:20,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:03:20,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:21,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:22,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:23,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:24,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:25,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:03:25,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:26,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:27,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:28,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:29,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:29,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:03:29,921 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:03:30,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:30,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:03:31,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:32,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:33,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:34,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:35,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:36,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:03:36,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:37,229 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:03:37,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:38,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:39,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:40,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:41,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:03:41,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:42,131 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:03:42,132 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:03:42,132 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:03:42,133 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:03:42,875 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:03:42,875 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:03:43,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:44,369 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:44,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:03:44,921 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:03:45,369 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:46,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:03:46,369 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:47,369 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:48,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:49,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:50,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:51,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:03:51,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:52,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:53,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:54,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:55,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:56,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:03:56,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:57,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:58,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:59,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:03:59,921 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:03:59,921 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:04:00,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:01,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:02,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:04:02,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:03,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:04,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:05,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:06,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:07,229 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:04:07,235 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:04:07,372 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:08,372 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:09,372 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:10,373 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:11,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:12,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:13,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:04:13,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:04:13,290 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:04:13,291 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:04:13,292 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:04:13,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:13,904 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:04:14,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:14,905 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:04:14,921 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:04:14,922 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:04:15,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:16,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:17,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:18,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:04:18,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:19,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:20,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:21,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:22,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:23,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:04:23,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:24,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:25,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:26,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:27,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:28,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:04:28,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:29,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:29,921 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:04:29,921 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:04:30,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:31,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:32,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:33,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:34,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:04:34,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:35,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:36,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:37,233 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:04:37,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:38,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:39,240 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:04:39,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:40,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:41,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:42,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:43,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:43,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:04:43,871 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:04:43,872 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:04:43,873 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:04:43,937 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:04:44,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:44,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:04:44,921 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:04:44,922 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:04:44,938 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:04:45,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:46,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:47,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:48,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:49,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:50,142 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:04:50,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:51,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:52,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:53,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:54,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:55,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:56,142 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:04:56,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:57,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:58,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:59,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:04:59,921 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:04:59,922 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:05:00,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:01,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:02,141 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:05:02,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:03,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:04,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:05,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:06,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:07,142 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:05:07,237 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:05:07,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:08,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:09,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:10,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:11,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:12,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:05:12,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:13,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:14,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:05:14,487 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:05:14,488 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:05:14,488 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:05:14,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:14,967 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:05:15,024 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:05:15,024 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:05:15,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:16,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:16,969 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:05:17,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:18,142 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:05:18,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:19,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:20,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:21,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:22,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:23,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:24,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:05:24,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:25,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:26,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:27,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:28,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:29,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:30,024 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:05:30,024 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:05:30,175 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:05:30,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:31,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:32,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:33,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:34,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:35,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:36,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:05:36,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:37,241 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:05:37,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:38,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:39,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:40,908 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:41,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:05:41,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:42,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:43,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:44,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:45,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:05:45,025 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:05:45,740 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:05:45,741 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:05:45,742 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:05:45,743 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:05:45,997 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:05:46,166 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:46,168 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:05:46,998 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:05:47,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:48,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:49,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:50,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:51,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:52,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:52,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:05:53,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:54,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:55,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:56,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:57,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:58,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:05:58,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:05:59,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:00,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:06:00,026 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:06:00,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:01,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:02,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:03,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:03,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:06:04,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:05,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:06,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:07,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:07,245 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:06:08,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:08,247 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:06:09,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:10,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:11,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:12,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:13,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:14,168 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:06:14,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:15,030 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:06:15,031 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:06:15,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:16,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:16,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:06:16,171 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:06:16,172 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:06:16,173 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:06:17,026 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:06:17,026 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:06:17,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:18,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:19,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:19,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:06:20,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:21,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:22,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:23,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:24,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:25,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:06:25,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:26,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:27,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:28,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:29,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:30,031 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:06:30,031 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:06:30,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:30,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:06:31,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:32,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:33,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:34,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:35,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:36,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:06:36,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:37,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:37,249 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:06:38,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:39,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:40,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:41,165 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:06:41,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:42,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:43,184 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:44,185 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:45,031 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:06:45,032 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:06:45,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:46,168 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:06:46,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:47,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:47,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:06:47,192 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:06:47,192 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:06:47,193 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:06:48,058 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:06:48,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:49,059 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:06:49,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:50,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:51,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:52,167 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:06:52,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:53,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:54,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:55,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:56,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:57,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:58,167 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:06:58,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:06:59,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:00,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:07:00,033 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:07:00,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:01,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:02,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:03,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:04,168 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:07:04,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:05,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:06,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:07,191 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:07,253 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:07:08,191 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:09,191 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:09,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:07:10,191 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:11,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:12,192 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:13,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:14,193 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:15,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:07:15,040 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:07:15,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:15,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:07:16,199 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:17,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:17,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:07:17,214 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:07:17,214 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:07:17,215 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:07:18,081 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:07:18,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:19,083 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:07:19,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:20,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:20,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:07:21,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:22,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:23,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:24,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:25,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:26,173 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:07:26,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:27,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:28,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:29,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:30,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:07:30,037 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:07:30,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:31,174 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:07:31,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:32,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:33,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:34,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:35,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:36,175 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:07:36,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:37,258 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:07:37,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:38,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:39,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:40,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:41,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:42,174 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:07:42,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:43,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:44,403 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:45,042 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:07:45,043 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:07:45,407 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:46,410 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:47,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:07:47,407 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:48,408 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:49,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:07:49,217 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:07:49,217 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:07:49,219 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:07:49,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:50,110 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:07:50,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:51,111 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:07:51,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:52,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:53,178 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:07:53,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:54,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:55,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:56,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:57,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:58,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:07:59,178 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:07:59,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:00,042 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:08:00,043 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:08:00,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:01,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:02,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:03,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:04,179 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:08:04,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:05,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:06,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:07,261 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:08:07,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:08,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:09,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:08:09,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:10,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:11,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:12,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:13,628 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:14,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:15,050 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:08:15,054 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:08:15,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:08:15,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:16,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:17,635 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:18,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:19,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:19,959 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:08:19,961 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:08:19,961 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:08:19,963 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:08:20,138 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:08:20,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:21,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:08:21,139 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:08:21,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:22,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:23,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:24,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:25,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:26,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:08:26,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:27,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:28,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:29,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:30,045 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:08:30,046 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:08:30,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:31,184 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:08:31,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:32,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:33,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:34,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:35,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:36,184 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:08:36,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:37,265 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:08:37,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:38,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:39,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:40,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:41,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:42,184 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:08:42,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:43,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:44,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:45,046 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:08:45,046 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:08:45,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:46,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:47,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:08:47,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:48,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:49,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:50,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:51,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:08:51,518 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:08:51,518 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:08:51,520 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:08:51,926 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:52,171 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:08:52,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:08:52,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:53,172 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:08:53,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:54,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:55,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:56,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:57,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:58,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:08:58,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:08:59,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:00,046 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:09:00,046 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:09:00,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:01,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:02,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:03,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:04,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:09:04,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:05,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:06,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:07,269 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:09:07,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:08,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:09,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:09:09,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:10,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:11,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:12,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:13,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:14,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:15,048 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:09:15,049 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:09:15,206 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:09:15,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:16,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:17,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:18,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:18,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:09:18,849 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:09:18,849 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:09:18,850 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:09:19,196 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:09:19,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:20,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:20,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:09:21,197 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:09:21,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:22,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:23,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:24,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:25,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:26,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:09:26,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:27,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:28,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:29,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:30,049 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:09:30,049 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:09:30,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:31,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:32,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:09:32,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:33,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:34,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:35,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:36,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:37,191 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:09:37,273 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:09:37,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:38,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:39,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:40,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:41,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:42,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:43,192 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:09:43,687 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:44,680 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:45,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:09:45,069 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:09:45,680 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:46,684 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:47,681 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:48,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:09:48,681 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:49,228 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:09:49,229 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:09:49,230 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:09:49,231 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:09:49,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:50,222 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:09:50,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:51,223 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:09:51,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:52,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:53,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:54,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:09:54,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:55,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:56,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:57,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:58,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:09:59,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:00,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:10:00,055 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:10:00,158 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:10:00,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:01,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:02,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:03,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:04,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:05,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:10:05,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:06,772 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:07,278 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:10:07,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:08,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:09,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:10,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:10:10,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:11,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:12,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:13,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:14,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:15,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:10:15,070 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:10:15,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:16,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:10:16,778 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:17,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:18,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:19,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:20,604 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:10:20,605 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:10:20,605 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:10:20,607 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:10:21,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:21,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:10:21,259 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:10:21,260 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:10:22,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:23,015 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:24,015 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:25,015 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:26,015 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:27,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:27,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:10:28,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:29,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:30,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:30,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:10:30,060 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:10:31,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:32,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:33,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:33,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:10:34,075 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:35,073 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:36,073 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:37,073 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:37,281 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:10:38,073 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:38,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:10:39,073 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:40,073 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:41,073 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:42,074 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:43,074 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:44,074 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:44,198 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:10:45,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:10:45,060 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:10:45,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:46,074 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:47,074 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:48,075 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:49,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:49,208 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:10:50,083 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:51,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:52,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:53,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:53,185 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:10:53,187 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:10:53,187 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:10:53,189 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:10:53,289 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:10:54,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:54,228 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:10:55,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:55,291 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:10:56,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:57,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:58,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:10:59,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:00,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:11:00,060 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:11:00,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:00,141 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:11:01,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:02,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:03,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:04,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:05,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:05,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:11:06,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:07,106 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:07,284 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:11:08,106 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:09,106 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:10,106 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:10,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:11:11,106 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:12,107 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:13,107 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:14,107 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:15,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:11:15,060 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:11:15,107 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:16,107 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:16,178 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:11:17,107 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:18,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:19,112 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:20,108 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:21,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:21,214 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:11:22,108 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:23,108 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:23,574 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:11:23,575 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:11:23,576 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:11:23,577 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:11:24,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:24,319 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:11:25,108 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:25,320 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:11:26,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:27,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:27,207 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:11:28,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:29,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:30,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:11:30,060 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:11:30,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:31,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:32,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:33,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:33,207 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:11:34,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:35,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:36,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:37,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:37,289 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:11:38,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:38,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:11:39,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:40,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:41,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:42,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:43,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:44,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:44,208 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:11:45,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:11:45,060 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:11:45,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:46,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:47,117 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:48,117 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:49,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:49,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:11:50,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:51,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:52,119 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:52,119 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:11:52,120 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:11:52,120 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:11:52,121 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:11:52,345 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:11:53,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:53,346 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:11:54,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:55,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:55,161 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:11:56,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:57,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:58,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:11:59,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:00,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:12:00,061 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:12:00,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:00,198 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:12:01,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:02,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:03,122 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:04,122 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:05,122 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:05,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:12:06,122 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:07,122 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:07,292 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:12:08,122 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:09,122 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:10,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:10,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:12:11,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:12,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:13,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:14,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:15,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:12:15,061 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:12:15,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:16,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:16,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:12:17,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:18,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:19,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:20,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:21,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:21,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:12:22,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:23,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:24,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:24,604 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:12:24,605 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:12:24,606 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:12:24,608 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:12:25,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:25,377 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:12:25,378 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:12:26,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:26,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:12:27,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:28,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:29,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:30,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:12:30,061 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:12:30,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:31,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:32,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:32,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:12:33,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:34,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:35,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:36,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:37,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:37,293 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:12:37,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:12:38,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:39,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:40,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:41,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:42,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:43,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:43,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:12:44,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:45,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:12:45,061 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:12:45,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:46,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:47,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:48,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:49,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:49,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:12:50,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:51,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:52,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:53,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:54,133 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:55,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:55,140 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:12:55,141 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:12:55,142 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:12:55,142 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:12:55,144 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:12:55,416 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:12:56,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:57,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:58,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:12:58,418 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:12:59,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:00,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:13:00,061 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:13:00,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:00,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:13:01,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:02,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:03,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:04,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:05,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:05,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:13:06,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:07,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:07,297 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:13:08,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:09,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:10,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:10,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:13:11,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:12,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:13,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:14,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:15,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:13:15,061 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:13:15,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:16,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:16,178 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:13:17,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:18,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:19,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:20,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:21,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:21,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:13:22,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:23,142 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:24,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:25,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:26,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:26,573 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:13:26,574 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:13:26,575 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:13:26,575 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:13:26,576 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:13:27,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:27,447 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:13:28,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:28,448 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:13:29,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:30,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:13:30,062 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:13:30,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:31,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:32,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:32,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:13:33,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:34,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:35,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:36,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:37,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:37,301 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:13:37,308 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:13:38,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:39,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:40,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:41,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:42,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:43,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:43,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:13:44,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:45,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:13:45,061 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:13:45,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:46,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:47,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:48,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:49,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:49,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:13:50,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:51,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:52,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:53,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:54,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:54,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:13:55,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:56,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:57,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:57,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:13:57,190 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:13:57,190 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:13:57,191 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:13:57,474 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:13:58,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:13:59,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:00,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:14:00,062 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:14:00,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:00,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:14:00,477 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:14:01,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:02,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:03,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:04,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:05,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:05,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:14:06,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:07,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:07,303 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:14:08,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:09,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:10,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:10,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:14:11,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:12,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:13,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:14,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:15,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:14:15,062 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:14:15,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:16,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:16,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:14:17,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:18,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:19,155 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:20,161 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:21,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:21,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:14:22,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:23,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:23,179 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:14:23,180 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:14:23,181 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:14:23,182 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:14:23,499 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:14:24,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:25,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:26,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:26,501 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:14:27,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:27,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:14:28,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:29,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:30,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:14:30,062 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:14:30,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:31,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:32,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:33,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:33,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:14:34,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:35,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:36,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:37,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:37,307 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:14:38,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:38,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:14:39,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:40,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:41,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:42,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:43,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:44,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:44,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:14:45,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:14:45,062 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:14:45,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:46,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:47,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:48,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:49,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:49,308 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:14:50,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:51,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:52,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:52,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:14:52,958 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:14:52,959 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:14:52,961 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:14:53,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:53,524 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:14:54,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:54,526 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:14:55,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:14:55,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:56,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:57,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:58,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:14:59,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:00,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:15:00,062 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:15:00,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:15:00,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:01,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:02,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:03,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:04,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:05,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:15:05,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:06,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:07,311 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:15:07,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:08,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:09,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:10,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:15:10,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:11,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:12,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:13,377 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:14,377 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:15,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:15:15,062 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:15:15,377 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:16,193 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:15:16,391 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:17,389 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:18,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:19,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:20,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:21,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:15:21,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:22,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:15:22,293 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:15:22,294 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:15:22,295 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:15:22,555 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:15:22,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:23,438 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:24,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:24,557 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:15:25,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:26,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:27,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:15:27,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:28,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:29,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:30,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:15:30,062 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:15:30,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:31,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:32,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:33,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:15:33,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:34,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:35,442 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:36,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:37,316 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:15:37,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:38,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:15:38,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:39,442 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:40,442 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:41,442 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:42,442 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:43,442 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:44,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:15:44,442 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:45,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:15:45,062 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:15:45,442 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:46,442 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:47,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:48,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:49,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:15:49,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:50,452 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:51,451 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:52,451 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:53,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:53,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:15:53,476 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:15:53,476 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:15:53,478 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:15:53,585 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:15:54,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:54,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:15:55,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:56,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:56,587 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:15:57,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:58,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:15:59,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:00,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:16:00,062 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:16:00,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:16:00,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:01,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:02,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:03,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:04,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:05,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:16:05,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:06,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:07,320 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:16:07,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:08,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:09,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:10,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:16:10,451 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:11,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:12,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:13,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:14,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:15,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:16:15,063 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:16:15,451 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:16,228 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:16:16,451 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:17,451 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:18,451 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:19,451 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:20,451 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:21,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:16:21,451 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:22,451 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:23,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:24,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:25,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:26,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:16:26,453 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:26,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:16:26,931 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:16:26,931 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:16:26,932 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:16:27,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:27,627 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:16:28,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:28,627 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:16:29,453 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:30,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:16:30,063 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:16:30,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:31,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:16:31,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:32,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:33,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:34,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:35,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:36,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:16:36,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:37,323 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:16:37,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:38,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:39,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:40,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:41,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:16:41,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:42,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:43,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:44,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:45,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:16:45,063 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:16:45,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:46,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:47,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:16:47,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:48,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:49,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:50,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:51,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:52,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:16:52,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:53,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:54,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:55,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:56,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:57,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:16:57,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:58,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:16:58,091 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:16:58,092 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:16:58,093 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:16:58,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:16:58,658 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:16:59,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:00,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:17:00,063 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:17:00,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:00,659 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:17:01,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:02,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:03,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:17:03,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:04,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:05,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:06,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:07,327 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:17:07,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:08,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:17:08,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:09,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:10,580 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:11,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:12,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:13,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:14,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:17:14,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:15,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:17:15,063 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:17:15,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:16,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:17,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:18,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:19,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:20,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:17:20,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:21,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:22,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:23,593 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:24,585 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:25,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:17:25,587 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:26,587 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:27,588 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:28,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:28,800 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:17:28,802 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:17:28,802 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:17:28,803 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:17:29,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:29,688 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:17:30,066 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:17:30,067 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:17:30,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:30,689 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:17:31,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:17:31,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:32,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:33,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:34,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:35,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:36,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:17:36,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:37,331 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:17:37,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:38,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:39,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:40,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:41,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:17:41,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:42,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:43,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:44,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:45,066 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:17:45,067 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:17:45,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:46,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:17:46,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:47,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:48,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:49,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:50,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:51,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:52,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:17:52,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:53,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:54,595 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:55,594 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:56,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:57,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:17:57,595 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:58,600 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:59,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:17:59,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:17:59,707 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:17:59,707 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:17:59,708 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:17:59,713 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:18:00,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:18:00,154 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:18:00,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:01,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:02,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:18:02,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:02,715 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:18:03,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:04,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:05,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:06,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:07,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:18:07,336 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:18:07,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:08,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:09,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:10,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:11,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:12,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:13,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:18:13,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:14,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:15,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:18:15,150 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:18:15,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:16,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:17,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:18,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:18:18,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:19,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:20,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:21,600 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:22,600 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:23,600 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:24,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:18:24,600 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:25,600 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:26,600 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:27,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:28,605 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:29,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:18:29,605 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:30,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:18:30,154 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:18:30,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:31,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:32,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:33,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:18:33,120 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:18:33,120 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:18:33,121 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:18:33,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:33,744 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:18:34,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:34,746 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:18:35,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:18:35,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:36,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:37,339 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:18:37,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:38,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:39,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:40,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:18:40,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:41,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:42,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:43,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:44,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:45,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:18:45,155 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:18:45,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:46,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:18:46,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:47,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:48,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:49,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:50,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:51,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:18:51,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:52,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:53,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:54,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:55,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:56,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:57,336 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:18:57,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:58,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:18:59,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:00,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:19:00,158 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:19:00,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:01,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:02,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:19:02,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:19:02,518 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:19:02,518 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:19:02,519 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:19:02,786 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:19:02,928 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:03,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:04,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:04,788 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:19:05,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:06,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:07,343 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:19:07,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:19:07,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:08,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:09,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:10,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:11,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:12,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:13,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:19:13,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:14,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:15,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:19:15,158 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:19:15,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:16,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:17,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:18,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:19,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:19:19,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:20,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:21,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:22,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:23,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:24,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:19:24,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:25,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:26,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:27,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:28,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:29,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:19:29,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:30,161 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:19:30,162 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:19:30,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:31,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:19:31,153 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:19:31,153 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:19:31,154 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:19:31,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:31,822 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:19:32,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:32,823 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:19:33,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:34,561 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:19:34,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:35,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:36,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:37,347 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:19:37,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:38,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:39,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:40,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:19:40,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:41,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:43,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:43,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:44,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:45,161 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:19:45,162 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:19:45,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:46,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:19:46,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:47,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:48,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:49,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:50,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:51,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:19:51,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:52,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:53,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:54,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:55,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:56,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:19:56,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:57,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:58,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:19:59,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:00,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:20:00,162 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:20:00,452 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:20:00,454 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:20:00,455 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:20:00,456 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:20:00,855 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:20:00,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:01,867 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:02,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:20:02,857 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:20:02,867 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:03,867 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:04,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:05,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:06,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:07,350 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:20:07,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:08,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:20:08,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:09,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:10,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:11,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:12,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:13,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:20:13,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:14,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:15,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:20:15,163 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:20:15,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:16,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:17,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:18,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:19,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:20:19,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:20,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:21,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:22,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:23,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:24,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:25,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:20:25,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:26,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:27,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:28,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:29,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:30,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:20:30,166 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:20:30,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:31,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:20:31,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:32,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:20:32,070 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:20:32,070 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:20:32,072 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:20:32,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:32,886 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:20:32,887 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:20:33,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:34,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:35,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:36,567 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:20:36,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:37,354 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:20:37,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:38,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:39,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:40,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:41,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:20:41,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:42,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:43,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:44,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:45,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:20:45,163 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:20:45,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:46,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:47,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:20:47,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:48,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:49,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:50,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:51,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:52,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:20:52,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:53,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:54,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:55,883 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:56,887 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:57,576 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:20:57,895 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:58,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:20:59,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:00,047 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:21:00,049 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:21:00,050 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:21:00,051 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:21:00,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:21:00,537 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:21:00,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:00,908 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:21:00,909 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:21:01,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:02,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:03,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:21:03,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:04,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:05,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:06,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:07,360 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:21:07,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:08,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:09,362 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:21:09,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:10,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:11,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:12,974 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:13,973 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:14,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:21:14,973 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:15,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:21:15,203 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:21:15,973 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:16,973 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:17,973 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:18,973 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:19,973 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:20,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:21:20,973 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:21,973 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:22,973 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:23,973 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:24,985 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:25,573 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:21:25,977 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:26,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:27,982 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:28,986 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:29,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:30,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:21:30,203 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:21:30,416 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:21:30,417 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:21:30,417 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:21:30,419 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:21:30,936 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:21:30,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:31,420 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:21:31,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:32,942 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:21:32,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:33,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:34,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:35,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:36,576 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:21:36,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:37,362 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:21:37,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:38,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:39,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:40,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:41,984 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:42,576 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:21:43,008 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:44,007 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:45,007 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:45,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:21:45,204 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:21:46,007 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:47,007 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:47,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:21:48,007 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:49,007 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:50,007 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:51,007 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:52,007 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:53,007 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:53,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:21:54,007 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:55,014 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:56,014 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:57,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:58,022 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:21:58,592 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:21:59,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:00,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:00,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:22:00,205 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:22:00,442 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:22:00,444 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:22:00,444 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:22:00,446 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:22:00,970 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:22:01,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:02,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:02,971 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:22:03,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:04,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:04,581 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:22:05,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:06,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:07,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:07,366 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:22:08,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:09,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:10,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:10,369 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:22:11,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:12,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:13,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:14,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:15,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:15,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:22:15,205 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:22:16,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:16,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:22:17,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:18,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:19,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:20,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:21,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:21,581 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:22:22,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:23,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:24,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:25,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:26,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:26,583 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:22:27,066 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:28,066 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:29,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:30,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:30,207 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:22:30,208 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:22:30,524 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:22:30,526 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:22:30,526 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:22:30,528 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:22:30,994 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:22:31,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:31,583 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:22:32,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:32,996 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:22:33,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:34,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:35,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:36,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:36,583 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:22:37,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:37,370 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:22:38,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:39,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:40,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:41,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:42,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:42,583 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:22:43,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:44,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:45,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:45,206 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:22:45,207 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:22:46,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:47,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:47,584 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:22:48,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:49,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:50,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:51,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:52,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:53,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:53,584 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:22:54,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:55,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:56,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:57,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:58,075 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:22:58,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:22:59,075 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:00,076 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:00,210 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:23:00,210 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:23:01,075 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:01,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:23:01,766 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:23:01,767 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:23:01,768 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:23:02,034 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:23:02,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:03,035 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:23:03,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:03,808 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:23:04,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:05,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:06,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:07,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:07,374 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:23:08,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:09,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:09,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:23:10,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:11,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:12,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:13,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:14,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:14,587 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:23:15,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:15,210 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:23:15,210 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:23:16,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:17,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:18,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:19,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:20,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:20,587 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:23:21,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:22,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:23,200 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:24,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:25,200 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:25,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:23:26,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:27,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:28,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:28,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:23:28,674 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:23:28,675 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:23:28,676 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:23:29,063 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:23:29,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:30,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:30,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:23:30,212 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:23:31,065 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:23:31,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:31,383 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:23:32,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:33,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:34,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:35,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:36,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:36,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:23:37,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:37,379 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:23:38,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:39,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:40,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:41,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:41,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:23:42,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:43,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:44,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:45,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:45,210 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:23:45,211 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:23:46,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:47,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:47,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:23:48,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:49,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:50,206 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:51,206 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:52,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:52,595 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:23:53,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:54,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:55,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:56,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:57,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:57,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:23:57,905 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:23:57,906 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:23:57,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:23:57,908 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:23:58,095 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:23:58,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:23:59,096 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:23:59,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:00,210 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:24:00,211 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:24:00,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:01,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:02,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:03,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:03,594 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:24:04,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:05,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:06,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:07,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:07,382 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:24:08,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:09,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:09,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:24:10,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:11,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:12,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:13,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:14,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:14,595 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:24:15,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:24:15,211 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:24:15,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:16,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:17,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:18,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:19,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:20,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:20,595 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:24:21,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:22,327 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:23,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:24,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:25,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:25,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:24:26,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:27,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:27,896 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:24:27,897 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:24:27,897 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:24:27,899 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:24:28,123 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:24:28,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:29,124 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:24:29,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:30,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:24:30,211 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:24:30,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:31,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:31,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:24:32,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:33,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:34,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:35,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:36,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:36,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:24:37,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:37,386 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:24:38,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:39,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:40,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:41,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:42,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:42,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:24:43,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:44,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:45,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:24:45,212 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:24:45,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:46,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:47,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:47,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:24:48,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:49,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:50,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:51,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:52,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:53,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:53,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:24:54,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:55,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:56,345 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:57,345 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:58,345 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:24:58,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:24:59,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:00,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:25:00,221 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:25:00,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:01,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:02,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:25:02,158 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:25:02,158 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:25:02,159 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:25:02,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:03,158 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:25:03,159 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:25:03,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:04,199 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:25:04,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:05,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:06,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:07,390 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:25:07,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:08,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:09,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:25:09,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:10,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:11,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:12,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:13,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:14,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:14,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:25:15,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:25:15,216 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:25:15,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:16,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:17,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:18,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:19,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:20,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:20,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:25:21,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:22,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:23,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:24,574 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:25,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:25,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:25:26,573 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:27,573 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:28,573 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:29,240 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:25:29,241 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:25:29,241 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:25:29,242 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:25:29,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:30,182 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:25:30,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:25:30,216 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:25:30,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:31,183 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:25:31,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:25:31,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:32,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:33,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:34,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:35,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:36,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:25:36,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:37,394 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:25:37,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:38,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:39,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:40,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:41,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:25:41,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:42,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:43,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:44,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:45,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:25:45,216 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:25:45,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:46,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:47,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:25:47,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:48,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:49,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:50,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:51,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:52,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:25:52,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:53,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:54,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:55,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:56,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:57,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:25:57,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:58,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:25:59,179 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:25:59,181 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:25:59,181 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:25:59,183 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:25:59,208 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:25:59,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:00,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:26:00,216 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:26:00,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:01,210 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:26:01,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:02,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:03,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:26:03,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:04,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:05,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:06,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:07,399 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:26:07,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:08,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:09,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:26:09,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:10,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:11,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:12,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:13,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:14,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:26:14,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:15,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:26:15,216 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:26:15,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:16,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:17,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:18,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:19,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:26:19,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:20,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:21,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:22,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:23,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:24,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:26:24,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:25,678 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:26,683 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:27,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:28,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:29,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:26:29,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:30,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:26:30,217 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:26:30,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:26:30,375 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:26:30,376 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:26:30,378 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:26:30,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:31,247 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:26:31,247 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:26:31,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:32,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:33,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:34,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:35,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:26:35,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:36,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:37,402 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:26:37,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:38,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:39,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:40,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:41,405 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:26:41,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:42,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:43,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:44,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:45,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:26:45,217 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:26:45,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:46,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:26:46,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:47,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:48,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:49,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:50,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:51,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:52,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:26:52,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:53,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:54,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:55,796 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:56,794 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:57,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:26:57,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:58,796 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:26:59,796 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:00,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:27:00,220 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:27:00,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:00,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:27:00,867 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:27:00,867 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:27:00,869 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:27:01,275 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:27:01,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:02,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:03,277 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:27:03,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:27:03,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:04,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:05,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:06,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:07,406 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:27:07,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:08,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:09,408 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:27:09,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:10,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:11,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:12,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:13,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:14,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:27:14,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:15,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:27:15,220 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:27:15,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:16,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:17,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:18,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:19,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:27:19,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:20,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:21,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:22,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:23,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:24,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:27:24,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:25,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:26,913 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:27,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:28,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:29,629 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:27:29,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:30,222 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:27:30,222 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:27:30,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:27:30,495 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:27:30,496 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:27:30,497 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:27:30,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:31,315 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:27:31,315 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:27:31,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:32,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:33,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:34,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:35,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:27:35,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:36,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:37,410 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:27:37,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:38,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:39,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:40,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:41,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:27:41,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:42,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:43,992 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:44,990 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:45,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:27:45,221 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:27:45,990 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:46,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:27:46,990 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:47,990 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:48,991 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:49,991 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:50,991 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:51,991 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:52,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:27:52,991 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:54,006 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:54,995 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:55,996 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:57,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:57,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:27:57,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:59,002 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:27:59,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:27:59,874 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:27:59,875 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:27:59,877 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:28:00,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:00,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:28:00,295 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:28:00,352 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:28:01,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:01,353 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:28:02,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:03,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:03,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:28:04,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:05,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:06,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:07,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:07,414 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:28:08,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:09,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:09,416 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:28:10,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:11,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:12,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:13,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:14,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:14,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:28:15,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:28:15,292 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:28:15,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:16,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:17,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:18,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:19,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:19,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:28:20,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:21,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:22,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:23,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:24,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:25,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:25,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:28:26,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:27,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:28,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:29,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:30,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:28:30,302 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:28:30,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:31,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:31,388 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:28:32,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:33,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:33,542 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:28:33,544 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:28:33,544 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:28:33,546 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:28:34,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:34,385 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:28:35,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:35,386 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:28:36,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:36,583 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:28:37,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:37,418 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:28:38,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:39,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:40,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:41,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:41,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:28:42,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:43,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:44,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:45,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:28:45,295 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:28:45,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:46,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:46,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:28:47,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:48,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:49,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:50,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:51,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:52,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:52,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:28:53,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:54,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:55,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:56,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:57,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:58,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:28:58,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:28:59,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:00,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:29:00,303 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:29:00,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:01,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:02,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:03,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:03,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:29:03,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:29:03,870 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:29:03,871 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:29:03,872 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:29:04,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:04,416 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:29:05,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:05,417 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:29:06,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:07,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:07,423 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:29:08,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:09,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:09,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:29:10,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:11,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:12,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:13,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:14,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:14,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:29:15,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:29:15,298 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:29:15,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:16,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:17,317 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:18,317 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:19,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:19,628 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:29:20,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:21,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:22,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:23,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:24,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:25,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:25,628 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:29:26,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:27,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:28,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:29,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:30,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:29:30,309 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:29:30,345 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:31,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:31,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:29:32,328 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:33,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:33,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:29:33,911 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:29:33,911 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:29:33,913 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:29:34,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:34,452 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:29:35,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:35,453 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:29:36,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:36,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:29:37,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:37,426 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:29:38,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:39,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:40,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:41,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:42,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:42,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:29:43,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:44,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:45,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:29:45,304 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:29:45,344 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:46,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:47,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:48,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:48,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:29:49,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:50,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:51,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:52,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:53,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:53,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:29:54,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:55,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:56,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:57,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:58,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:29:58,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:29:59,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:00,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:30:00,318 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:30:00,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:01,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:02,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:03,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:03,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:30:04,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:04,575 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:30:04,577 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:30:04,578 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:30:04,579 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:30:05,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:05,482 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:30:05,483 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:30:06,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:07,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:07,430 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:30:08,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:09,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:09,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:30:10,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:11,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:12,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:13,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:14,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:14,633 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:30:15,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:30:15,310 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:30:15,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:16,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:17,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:18,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:19,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:19,633 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:30:20,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:21,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:22,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:23,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:24,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:24,634 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:30:25,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:26,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:27,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:28,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:29,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:30,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:30:30,310 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:30:30,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:30,407 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:30:31,365 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:32,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:33,362 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:34,363 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:35,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:35,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:30:36,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:36,595 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:30:36,597 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:30:36,597 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:30:36,599 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:30:37,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:37,433 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:30:37,518 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:30:37,518 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:30:38,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:39,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:40,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:41,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:41,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:30:42,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:43,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:44,361 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:45,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:30:45,310 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:30:45,361 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:46,361 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:46,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:30:47,361 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:48,361 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:49,361 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:50,362 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:51,361 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:51,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:30:52,361 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:53,361 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:54,361 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:55,362 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:56,362 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:57,362 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:57,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:30:58,362 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:30:59,362 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:00,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:31:00,310 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:31:00,362 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:01,373 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:02,369 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:02,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:31:03,369 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:04,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:05,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:06,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:06,997 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:31:06,999 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:31:06,999 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:31:07,001 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:31:07,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:07,438 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:31:07,542 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:31:07,543 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:31:08,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:08,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:31:09,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:10,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:11,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:12,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:13,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:13,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:31:14,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:15,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:31:15,310 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:31:15,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:16,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:17,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:18,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:19,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:19,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:31:20,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:21,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:22,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:23,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:24,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:24,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:31:25,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:26,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:27,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:28,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:29,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:30,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:31:30,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:31:30,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:30,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:31:31,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:32,446 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:33,442 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:34,442 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:35,451 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:35,756 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:31:36,445 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:37,441 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:31:37,445 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:38,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:31:38,091 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:31:38,092 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:31:38,093 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:31:38,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:38,574 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:31:39,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:39,575 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:31:40,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:41,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:31:41,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:42,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:43,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:44,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:45,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:31:45,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:31:45,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:46,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:31:46,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:47,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:48,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:49,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:50,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:51,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:51,752 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:31:52,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:53,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:54,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:55,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:56,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:56,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:31:57,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:58,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:31:59,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:00,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:32:00,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:32:00,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:01,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:02,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:02,755 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:32:03,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:04,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:05,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:06,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:07,443 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:32:07,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:08,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:32:08,160 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:32:08,160 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:32:08,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:32:08,161 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:32:08,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:08,604 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:32:09,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:09,606 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:32:10,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:11,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:12,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:13,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:13,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:32:14,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:15,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:32:15,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:32:15,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:16,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:17,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:18,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:19,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:19,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:32:20,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:21,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:22,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:23,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:24,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:24,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:32:25,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:26,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:27,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:28,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:29,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:30,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:32:30,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:32:30,397 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:32:30,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:31,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:32,574 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:33,575 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:34,580 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:35,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:35,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:32:36,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:37,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:32:37,316 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:32:37,317 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:32:37,318 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:32:37,447 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:32:37,637 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:32:37,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:38,732 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:39,638 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:32:39,732 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:40,732 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:41,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:32:41,732 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:42,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:43,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:44,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:45,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:32:45,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:32:45,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:46,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:32:46,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:47,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:48,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:49,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:50,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:51,734 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:51,761 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:32:52,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:53,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:54,734 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:55,734 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:56,734 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:57,734 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:57,760 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:32:58,734 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:32:59,734 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:00,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:33:00,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:33:00,734 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:01,734 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:02,734 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:03,734 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:03,760 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:33:04,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:05,736 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:06,736 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:07,452 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:33:07,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:08,736 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:09,453 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:33:09,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:10,006 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:33:10,008 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:33:10,008 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:33:10,010 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:33:10,666 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:33:10,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:11,667 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:33:11,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:12,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:13,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:14,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:14,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:33:15,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:33:15,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:33:15,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:16,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:17,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:18,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:19,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:19,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:33:20,738 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:21,738 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:22,738 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:23,738 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:24,738 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:24,764 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:33:25,738 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:26,738 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:27,738 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:28,738 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:29,738 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:30,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:33:30,312 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:33:30,402 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:33:30,738 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:31,739 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:32,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:33,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:34,745 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:35,745 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:35,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:33:36,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:37,456 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:33:37,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:38,635 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:33:38,637 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:33:38,637 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:33:38,639 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:33:38,693 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:33:39,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:39,694 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:33:40,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:41,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:41,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:33:42,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:43,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:44,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:45,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:45,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:33:45,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:33:46,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:46,767 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:33:47,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:48,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:49,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:50,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:51,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:51,767 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:33:52,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:53,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:54,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:55,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:56,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:56,768 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:33:57,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:58,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:33:59,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:00,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:00,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:34:00,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:34:01,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:02,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:02,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:34:03,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:04,186 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:05,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:06,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:07,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:07,460 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:34:08,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:34:08,059 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:34:08,060 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:34:08,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:34:08,061 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:34:08,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:08,723 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:34:09,210 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:09,724 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:34:10,210 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:11,210 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:12,210 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:13,210 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:13,772 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:34:14,210 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:15,210 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:15,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:34:15,312 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:34:16,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:17,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:18,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:19,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:19,772 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:34:20,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:21,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:22,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:23,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:24,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:25,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:25,772 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:34:26,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:27,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:28,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:29,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:30,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:30,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:34:30,312 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:34:31,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:31,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:34:32,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:33,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:34,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:35,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:36,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:36,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:34:37,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:34:37,166 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:34:37,167 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:34:37,168 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:34:37,465 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:34:37,575 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:37,755 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:34:37,756 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:34:38,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:39,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:40,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:41,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:41,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:34:42,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:43,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:44,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:45,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:45,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:34:45,312 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:34:46,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:47,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:47,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:34:48,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:49,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:50,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:51,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:52,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:52,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:34:53,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:54,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:55,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:56,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:57,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:58,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:34:58,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:34:59,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:00,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:35:00,312 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:35:00,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:01,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:02,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:03,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:04,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:04,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:35:05,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:06,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:07,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:07,469 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:35:08,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:09,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:10,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:10,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:35:11,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:11,993 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:35:11,995 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:35:11,996 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:35:11,997 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:35:12,790 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:35:13,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:13,791 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:35:14,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:15,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:35:15,312 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:35:15,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:15,472 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:35:16,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:17,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:18,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:19,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:20,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:20,867 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:35:21,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:22,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:23,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:24,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:25,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:26,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:26,867 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:35:27,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:28,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:29,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:30,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:35:30,312 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:35:30,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:31,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:32,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:32,867 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:35:33,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:34,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:35,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:36,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:37,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:37,480 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:35:38,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:38,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:35:39,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:40,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:41,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:42,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:43,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:43,549 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:35:43,551 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:35:43,551 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:35:43,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:35:43,553 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:35:43,831 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:35:44,357 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:45,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:35:45,313 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:35:45,357 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:45,832 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:35:46,357 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:47,357 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:48,357 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:48,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:35:49,357 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:50,357 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:51,357 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:52,357 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:53,357 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:53,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:35:54,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:55,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:56,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:57,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:58,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:35:58,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:35:59,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:00,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:36:00,313 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:36:00,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:01,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:02,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:03,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:04,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:04,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:36:05,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:06,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:07,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:07,482 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:36:08,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:09,363 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:10,363 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:10,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:36:11,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:12,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:13,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:13,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:36:13,617 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:36:13,617 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:36:13,619 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:36:13,856 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:36:14,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:15,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:36:15,313 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:36:15,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:15,858 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:36:16,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:16,451 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:36:17,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:18,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:19,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:20,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:21,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:21,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:36:22,365 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:23,365 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:24,365 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:25,365 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:26,365 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:26,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:36:27,365 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:28,365 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:29,365 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:30,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:36:30,313 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:36:30,365 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:31,365 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:31,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:36:32,366 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:33,366 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:34,366 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:35,366 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:36,366 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:36,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:36:37,366 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:37,478 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:36:38,366 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:39,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:40,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:41,373 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:41,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:36:42,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:43,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:44,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:44,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:36:44,669 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:36:44,670 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:36:44,675 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:36:44,887 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:36:45,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:36:45,313 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:36:45,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:45,888 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:36:46,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:47,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:47,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:36:48,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:49,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:50,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:51,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:52,372 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:53,372 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:53,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:36:54,372 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:55,372 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:56,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:57,392 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:58,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:36:58,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:36:59,392 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:00,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:37:00,313 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:37:00,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:01,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:02,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:03,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:04,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:04,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:37:05,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:06,399 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:07,396 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:07,480 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:37:08,404 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:09,397 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:10,397 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:10,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:37:11,397 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:12,030 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:37:12,031 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:37:12,032 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:37:12,034 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:37:12,912 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:37:13,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:13,913 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:37:14,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:15,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:37:15,313 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:37:15,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:16,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:16,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:37:17,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:18,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:19,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:20,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:21,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:21,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:37:22,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:23,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:24,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:25,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:26,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:26,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:37:27,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:28,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:29,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:30,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:37:30,313 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:37:30,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:31,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:32,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:32,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:37:33,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:34,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:35,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:36,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:37,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:37,484 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:37:38,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:38,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:37:39,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:40,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:41,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:42,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:43,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:43,884 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:37:44,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:45,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:37:45,317 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:37:45,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:45,946 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:37:45,946 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:37:45,947 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:37:45,949 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:37:46,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:46,949 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:37:47,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:47,950 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:37:48,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:49,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:49,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:37:50,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:51,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:52,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:53,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:54,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:55,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:55,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:37:56,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:57,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:58,366 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:37:59,366 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:00,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:38:00,315 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:38:00,366 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:01,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:01,406 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:38:02,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:03,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:04,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:05,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:06,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:06,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:38:07,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:07,488 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:38:08,368 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:09,373 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:10,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:11,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:11,883 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:38:12,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:13,377 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:14,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:14,481 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:38:14,482 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:38:14,483 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:38:14,484 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:38:14,972 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:38:15,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:38:15,315 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:38:15,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:15,973 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:38:16,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:17,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:17,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:38:18,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:19,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:20,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:21,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:22,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:22,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:38:23,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:24,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:25,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:26,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:27,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:27,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:38:28,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:29,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:30,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:38:30,315 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:38:30,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:31,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:32,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:33,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:33,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:38:34,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:35,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:36,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:37,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:37,492 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:38:38,390 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:39,386 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:39,497 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:38:40,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:41,386 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:42,386 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:43,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:43,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:38:43,703 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:38:43,703 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:38:43,705 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:38:44,003 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:38:44,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:44,745 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:38:45,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:38:45,315 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:38:45,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:46,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:47,005 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:38:47,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:48,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:49,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:49,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:38:50,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:51,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:52,388 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:53,388 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:54,388 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:54,883 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:38:55,388 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:56,388 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:57,390 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:58,388 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:38:59,388 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:00,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:39:00,316 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:39:00,389 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:00,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:39:01,389 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:02,389 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:03,389 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:04,389 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:05,389 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:05,883 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:39:06,389 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:07,389 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:07,496 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:39:08,389 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:09,390 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:10,394 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:11,397 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:11,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:39:12,399 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:13,398 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:14,402 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:15,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:39:15,316 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:39:15,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:15,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:39:15,508 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:39:15,508 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:39:15,509 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:39:16,037 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:39:16,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:16,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:39:17,038 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:39:17,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:18,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:19,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:20,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:21,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:21,886 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:39:22,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:23,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:24,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:25,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:26,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:27,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:27,886 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:39:28,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:29,402 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:30,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:39:30,316 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:39:30,402 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:31,402 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:32,402 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:32,886 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:39:33,402 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:34,402 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:35,402 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:36,402 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:37,402 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:37,501 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:39:38,402 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:38,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:39:39,403 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:40,403 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:41,411 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:42,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:43,411 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:43,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:39:44,412 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:45,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:39:45,324 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:39:45,412 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:46,412 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:46,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:39:46,907 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:39:46,907 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:39:46,909 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:39:47,066 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:39:47,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:48,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:48,949 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:39:49,068 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:39:49,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:50,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:51,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:52,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:53,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:54,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:54,887 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:39:55,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:56,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:57,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:58,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:39:59,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:00,319 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:40:00,319 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:40:00,404 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:40:00,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:01,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:02,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:03,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:04,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:05,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:05,888 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:40:06,415 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:07,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:07,504 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:40:08,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:09,415 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:10,415 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:11,415 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:11,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:40:12,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:13,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:14,422 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:15,319 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:40:15,319 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:40:15,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:15,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:40:15,593 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:40:15,593 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:40:15,595 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:40:16,090 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:40:16,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:16,634 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:40:17,091 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:40:17,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:18,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:19,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:20,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:21,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:21,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:40:22,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:23,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:24,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:25,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:26,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:26,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:40:27,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:28,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:29,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:30,319 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:40:30,319 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:40:30,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:31,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:32,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:32,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:40:33,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:34,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:35,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:36,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:37,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:37,508 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:40:38,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:38,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:40:39,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:40,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:41,428 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:42,429 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:43,429 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:43,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:40:44,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:45,319 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:40:45,319 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:40:45,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:46,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:40:46,195 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:40:46,196 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:40:46,197 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:40:46,605 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:47,117 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:40:47,605 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:48,605 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:49,119 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:40:49,605 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:49,895 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:40:50,605 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:51,605 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:52,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:53,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:54,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:55,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:55,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:40:56,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:57,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:58,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:40:59,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:00,319 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:41:00,320 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:41:00,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:01,476 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:41:01,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:02,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:03,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:04,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:05,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:06,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:06,895 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:41:07,512 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:41:07,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:08,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:09,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:10,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:11,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:11,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:41:12,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:13,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:14,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:15,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:41:15,325 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:41:15,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:16,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:17,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:41:17,172 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:41:17,173 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:41:17,173 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:41:17,174 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:41:17,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:18,146 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:41:18,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:19,147 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:41:19,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:20,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:21,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:22,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:22,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:41:23,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:24,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:25,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:26,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:27,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:28,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:28,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:41:29,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:30,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:41:30,322 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:41:30,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:31,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:32,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:33,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:33,900 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:41:34,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:35,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:36,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:37,516 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:41:37,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:38,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:39,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:41:39,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:40,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:41,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:42,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:43,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:44,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:44,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:41:45,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:41:45,327 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:41:45,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:46,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:47,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:48,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:49,120 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:41:49,121 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:41:49,122 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:41:49,124 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:41:49,176 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:41:49,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:50,161 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:41:50,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:51,177 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:41:51,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:52,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:53,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:54,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:55,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:55,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:41:56,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:57,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:58,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:41:59,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:00,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:42:00,326 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:42:00,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:01,409 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:42:01,690 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:02,690 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:03,690 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:04,690 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:05,691 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:06,691 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:06,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:42:07,519 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:42:07,691 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:08,691 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:09,691 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:10,691 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:11,691 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:11,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:42:12,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:13,700 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:14,699 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:15,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:42:15,330 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:42:15,696 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:16,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:17,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:17,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:42:18,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:42:18,174 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:42:18,174 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:42:18,176 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:42:18,200 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:42:18,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:19,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:20,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:21,203 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:42:21,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:22,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:22,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:42:23,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:24,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:25,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:26,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:27,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:28,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:28,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:42:29,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:30,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:42:30,330 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:42:30,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:31,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:32,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:33,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:33,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:42:34,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:35,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:36,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:37,523 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:42:37,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:38,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:39,525 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:42:39,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:40,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:41,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:42,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:43,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:44,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:44,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:42:45,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:42:45,333 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:42:45,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:46,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:47,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:48,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:48,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:42:48,788 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:42:48,789 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:42:48,790 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:42:49,228 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:42:49,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:50,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:50,828 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:42:51,230 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:42:51,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:52,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:53,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:54,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:55,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:55,908 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:42:56,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:57,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:58,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:42:59,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:00,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:43:00,371 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:43:00,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:01,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:43:01,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:02,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:03,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:04,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:05,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:06,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:06,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:43:07,526 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:43:07,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:08,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:09,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:10,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:11,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:11,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:43:12,720 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:13,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:14,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:15,372 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:43:15,374 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:43:15,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:16,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:17,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:43:17,621 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:43:17,622 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:43:17,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:43:17,623 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:43:18,006 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:18,257 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:43:18,767 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:19,258 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:43:19,767 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:20,768 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:21,768 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:22,768 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:22,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:43:23,768 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:24,768 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:25,768 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:26,768 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:27,768 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:28,768 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:28,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:43:29,768 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:30,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:43:30,514 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:43:30,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:31,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:32,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:33,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:33,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:43:34,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:35,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:36,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:37,530 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:43:37,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:38,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:39,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:43:39,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:40,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:41,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:42,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:43,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:44,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:44,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:43:45,419 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:43:45,423 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:43:45,774 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:46,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:47,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:48,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:48,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:43:48,825 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:43:48,826 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:43:48,827 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:43:49,289 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:43:49,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:50,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:50,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:43:51,290 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:43:51,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:52,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:53,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:54,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:55,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:55,917 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:43:56,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:57,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:58,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:43:59,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:00,419 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:44:00,420 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:44:00,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:01,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:44:01,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:02,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:03,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:04,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:05,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:06,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:06,917 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:44:07,534 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:44:07,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:08,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:09,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:10,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:11,778 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:11,922 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:44:12,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:13,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:14,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:15,429 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:44:15,429 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:44:15,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:16,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:16,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:44:17,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:17,886 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:44:17,887 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:44:17,888 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:44:17,889 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:44:18,318 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:44:18,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:19,318 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:44:19,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:20,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:21,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:21,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:44:22,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:23,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:24,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:25,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:26,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:27,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:27,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:44:28,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:29,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:30,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:44:30,424 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:44:30,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:31,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:32,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:32,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:44:33,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:34,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:35,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:36,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:37,538 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:44:37,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:38,540 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:44:38,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:39,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:40,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:41,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:42,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:43,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:43,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:44:44,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:45,427 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:44:45,429 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:44:45,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:46,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:47,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:47,840 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:44:47,841 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:44:47,841 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:44:47,842 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:44:48,342 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:44:48,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:49,343 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:44:49,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:49,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:44:50,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:51,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:52,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:53,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:54,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:54,926 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:44:55,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:56,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:57,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:58,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:44:59,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:00,426 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:45:00,426 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:45:00,593 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:45:00,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:01,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:02,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:03,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:04,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:05,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:05,927 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:45:06,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:07,543 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:45:07,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:08,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:09,833 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:10,833 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:11,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:45:11,833 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:12,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:13,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:14,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:15,429 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:45:15,433 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:45:15,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:16,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:45:16,840 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:17,842 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:18,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:18,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:45:18,940 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:45:18,940 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:45:18,942 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:45:19,376 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:45:19,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:20,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:21,377 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:45:21,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:21,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:45:22,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:23,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:24,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:25,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:26,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:27,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:27,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:45:28,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:29,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:30,429 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:45:30,430 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:45:30,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:31,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:32,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:32,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:45:33,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:34,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:35,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:36,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:37,546 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:45:37,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:38,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:45:38,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:39,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:40,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:41,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:42,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:43,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:43,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:45:44,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:45,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:45:45,430 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:45:45,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:46,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:47,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:48,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:48,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:45:49,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:50,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:51,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:52,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:45:52,040 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:45:52,041 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:45:52,042 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:45:52,408 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:45:52,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:53,409 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:45:53,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:54,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:45:54,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:55,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:56,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:57,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:58,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:59,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:45:59,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:46:00,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:46:00,430 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:46:00,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:01,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:02,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:03,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:04,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:05,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:05,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:46:06,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:07,550 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:46:07,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:08,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:09,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:10,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:11,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:46:11,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:12,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:13,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:14,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:15,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:46:15,430 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:46:15,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:16,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:46:16,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:17,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:18,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:19,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:20,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:21,867 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:21,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:46:22,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:23,867 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:24,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:25,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:46:25,745 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:46:25,745 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:46:25,746 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:46:26,155 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:26,452 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:46:26,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:27,453 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:46:27,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:27,933 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:46:28,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:29,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:30,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:46:30,430 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:46:30,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:31,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:32,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:33,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:33,933 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:46:34,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:35,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:36,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:37,554 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:46:37,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:38,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:39,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:46:39,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:40,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:41,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:42,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:43,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:44,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:44,934 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:46:45,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:46:45,430 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:46:45,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:46,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:47,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:48,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:49,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:49,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:46:50,898 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:51,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:52,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:53,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:54,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:46:54,489 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:46:54,489 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:46:54,490 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:46:54,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:55,482 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:46:55,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:46:55,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:56,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:57,484 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:46:57,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:58,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:46:59,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:00,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:47:00,430 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:47:00,549 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:47:00,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:01,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:02,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:03,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:04,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:05,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:05,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:47:06,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:07,558 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:47:07,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:08,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:09,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:10,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:11,561 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:47:11,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:12,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:13,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:14,908 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:15,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:47:15,431 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:47:15,908 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:16,908 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:16,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:47:17,908 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:18,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:19,917 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:20,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:21,913 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:22,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:22,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:47:23,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:24,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:25,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:47:25,523 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:47:25,523 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:47:25,525 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:47:25,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:26,516 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:47:26,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:27,517 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:47:27,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:27,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:47:28,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:29,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:30,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:47:30,431 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:47:30,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:31,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:32,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:33,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:33,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:47:34,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:35,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:36,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:37,562 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:47:37,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:38,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:39,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:47:39,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:40,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:41,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:42,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:43,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:44,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:44,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:47:45,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:47:45,431 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:47:45,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:46,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:47,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:48,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:49,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:47:49,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:50,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:51,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:52,945 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:53,945 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:54,945 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:54,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:47:55,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:47:55,826 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:47:55,826 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:47:55,827 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:47:56,241 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:56,543 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:47:57,238 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:57,544 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:47:58,238 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:47:59,238 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:00,238 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:00,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:48:00,431 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:48:00,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:48:01,238 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:02,238 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:03,238 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:04,238 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:05,239 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:05,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:48:06,239 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:07,239 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:07,567 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:48:08,239 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:09,239 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:10,239 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:11,239 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:11,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:48:12,239 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:13,240 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:14,240 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:15,240 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:15,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:48:15,433 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:48:16,240 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:16,573 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:48:17,240 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:18,240 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:19,240 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:20,251 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:21,246 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:21,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:48:22,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:23,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:24,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:25,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:26,114 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:48:26,116 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:48:26,116 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:48:26,118 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:48:26,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:26,572 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:48:27,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:27,573 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:48:27,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:48:28,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:29,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:30,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:48:30,431 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:48:30,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:31,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:32,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:33,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:33,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:48:34,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:35,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:36,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:37,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:37,570 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:48:38,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:39,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:39,573 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:48:40,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:41,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:42,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:43,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:44,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:44,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:48:45,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:48:45,431 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:48:45,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:46,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:47,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:48,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:49,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:50,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:50,953 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:48:51,531 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:52,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:53,531 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:54,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:55,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:56,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:56,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:48:56,671 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:48:56,672 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:48:56,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:48:56,673 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:48:57,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:57,605 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:48:57,606 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:48:58,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:48:59,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:00,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:49:00,432 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:49:00,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:01,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:02,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:49:02,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:03,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:04,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:05,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:06,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:07,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:07,574 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:49:07,581 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:49:08,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:09,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:10,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:11,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:12,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:13,085 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:49:13,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:14,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:15,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:49:15,432 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:49:15,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:16,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:17,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:18,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:19,085 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:49:19,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:20,541 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:21,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:22,542 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:23,543 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:24,085 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:49:24,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:25,117 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:49:25,119 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:49:25,119 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:49:25,121 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:49:25,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:25,633 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:49:26,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:27,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:27,634 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:49:28,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:29,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:30,085 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:49:30,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:49:30,432 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:49:30,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:31,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:32,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:33,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:34,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:35,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:36,085 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:49:36,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:37,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:37,578 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:49:38,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:39,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:40,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:41,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:41,581 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:49:42,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:43,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:44,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:45,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:49:45,432 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:49:45,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:46,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:46,595 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:49:47,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:48,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:49,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:50,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:51,547 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:52,085 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:49:52,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:53,549 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:54,549 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:55,550 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:56,550 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:56,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:49:56,851 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:49:56,851 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:49:56,853 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:49:57,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:49:57,550 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:57,664 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:49:57,665 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:49:58,550 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:49:59,550 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:00,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:50:00,432 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:50:00,550 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:01,550 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:02,550 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:03,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:50:03,551 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:04,551 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:05,551 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:06,551 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:07,551 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:07,582 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:50:08,551 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:08,584 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:50:09,551 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:10,551 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:11,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:12,551 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:13,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:14,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:50:14,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:15,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:50:15,432 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:50:15,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:16,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:17,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:18,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:19,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:20,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:50:20,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:21,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:22,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:23,559 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:24,561 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:25,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:50:25,560 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:26,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:27,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:28,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:50:28,394 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:50:28,394 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:50:28,396 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:50:28,691 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:50:28,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:29,692 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:50:29,804 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:30,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:50:30,432 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:50:30,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:50:30,804 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:31,804 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:32,804 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:33,804 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:34,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:35,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:36,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:50:36,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:37,587 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:50:37,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:38,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:39,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:40,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:41,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:50:41,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:42,921 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:43,917 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:44,917 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:45,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:50:45,433 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:50:45,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:46,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:47,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:50:47,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:48,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:49,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:50,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:51,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:52,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:50:52,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:53,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:54,926 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:55,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:56,927 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:57,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:50:57,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:58,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:50:59,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:00,117 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:51:00,118 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:51:00,119 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:51:00,120 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:51:00,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:51:00,530 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:51:00,735 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:51:00,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:01,736 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:51:01,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:02,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:03,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:51:03,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:04,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:05,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:06,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:07,590 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:51:07,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:08,592 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:51:08,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:09,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:10,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:11,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:12,925 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:13,925 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:14,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:51:14,925 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:15,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:51:15,528 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:51:15,925 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:16,925 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:17,925 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:18,926 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:19,926 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:20,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:51:20,926 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:21,926 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:22,926 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:23,926 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:24,926 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:25,932 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:26,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:51:26,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:27,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:28,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:29,932 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:30,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:51:30,530 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:51:30,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:31,186 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:51:31,188 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:51:31,189 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:51:31,190 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:51:31,766 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:51:31,767 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:51:31,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:32,228 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:51:32,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:33,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:34,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:35,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:36,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:37,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:51:37,593 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:51:37,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:38,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:39,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:40,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:41,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:42,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:43,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:51:43,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:44,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:45,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:51:45,530 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:51:45,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:46,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:47,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:48,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:51:48,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:49,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:50,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:51,933 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:52,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:53,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:54,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:51:54,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:55,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:56,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:57,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:58,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:51:59,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:51:59,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:00,531 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:52:00,532 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:52:00,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:01,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:52:01,355 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:52:01,355 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:52:01,356 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:52:01,796 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:52:01,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:02,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:03,798 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:52:03,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:04,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:05,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:52:05,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:06,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:07,597 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:52:07,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:08,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:09,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:10,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:52:10,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:11,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:12,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:13,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:14,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:15,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:52:15,532 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:52:15,624 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:52:15,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:16,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:17,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:18,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:19,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:20,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:21,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:52:21,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:22,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:23,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:24,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:25,948 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:26,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:52:26,946 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:27,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:28,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:29,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:30,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:52:30,535 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:52:30,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:31,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:52:31,132 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:52:31,132 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:52:31,134 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:52:31,828 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:52:31,828 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:52:31,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:32,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:52:32,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:33,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:34,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:35,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:36,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:37,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:52:37,601 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:52:37,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:38,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:39,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:40,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:41,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:42,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:52:42,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:43,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:44,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:45,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:52:45,535 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:52:45,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:46,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:47,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:52:47,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:48,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:49,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:50,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:51,953 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:52,953 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:53,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:52:53,958 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:54,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:55,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:56,965 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:57,965 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:58,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:52:58,965 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:52:59,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:52:59,310 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:52:59,310 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:52:59,311 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:52:59,860 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:52:59,860 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:52:59,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:00,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:53:00,535 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:53:00,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:01,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:02,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:03,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:04,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:53:04,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:05,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:06,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:07,605 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:53:07,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:08,967 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:09,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:53:09,967 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:10,967 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:11,967 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:12,967 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:13,967 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:14,967 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:15,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:53:15,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:53:15,535 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:53:15,967 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:16,967 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:17,968 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:18,968 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:19,968 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:20,968 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:21,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:53:21,968 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:22,968 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:23,968 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:24,968 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:25,968 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:26,968 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:27,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:53:27,969 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:28,969 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:29,975 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:30,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:53:30,535 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:53:30,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:31,981 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:32,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:53:32,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:33,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:53:33,034 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:53:33,034 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:53:33,035 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:53:33,895 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:53:33,896 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:53:33,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:34,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:35,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:36,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:37,610 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:53:37,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:53:37,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:38,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:39,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:40,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:41,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:42,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:43,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:53:43,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:44,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:45,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:53:45,535 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:53:45,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:46,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:47,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:48,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:53:48,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:49,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:50,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:51,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:52,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:53,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:54,403 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:53:54,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:55,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:56,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:57,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:58,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:53:59,981 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:00,403 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:54:00,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:54:00,535 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:54:00,993 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:01,994 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:02,996 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:03,993 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:04,993 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:05,411 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:54:05,990 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:06,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:54:06,717 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:54:06,718 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:54:06,720 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:54:06,927 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:54:07,124 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:07,613 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:54:07,929 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:54:08,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:09,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:10,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:10,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:54:11,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:12,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:13,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:14,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:15,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:15,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:54:15,536 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:54:15,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:54:16,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:17,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:18,124 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:19,124 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:20,124 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:21,124 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:21,409 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:54:22,124 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:23,124 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:24,124 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:25,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:26,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:26,409 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:54:27,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:28,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:29,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:30,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:30,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:54:30,536 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:54:31,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:31,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:54:32,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:33,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:34,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:35,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:36,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:37,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:37,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:54:37,623 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:54:38,133 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:39,133 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:39,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:54:39,726 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:54:39,727 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:54:39,728 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:54:39,962 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:54:40,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:41,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:41,963 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:54:42,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:43,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:43,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:54:44,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:45,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:45,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:54:45,536 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:54:46,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:47,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:48,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:49,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:49,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:54:50,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:51,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:52,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:53,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:54,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:54,415 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:54:55,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:56,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:57,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:58,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:54:59,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:00,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:00,415 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:55:00,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:55:00,536 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:55:01,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:02,140 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:03,140 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:04,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:05,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:05,419 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:55:06,147 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:07,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:07,623 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:55:08,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:09,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:09,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:55:09,825 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:55:09,825 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:55:09,826 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:55:09,987 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:55:10,241 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:10,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:55:11,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:11,989 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:55:12,247 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:13,247 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:14,247 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:15,247 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:15,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:55:15,536 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:55:16,247 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:16,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:55:17,247 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:18,247 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:19,247 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:20,247 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:21,247 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:22,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:22,416 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:55:23,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:24,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:25,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:26,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:27,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:27,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:55:28,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:29,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:30,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:30,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:55:30,536 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:55:31,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:32,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:33,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:33,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:55:34,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:35,258 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:36,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:37,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:37,636 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:55:38,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:38,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:55:39,258 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:40,258 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:40,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:55:40,719 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:55:40,719 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:55:40,721 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:55:41,024 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:55:41,259 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:42,025 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:55:42,259 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:43,259 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:44,259 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:44,420 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:55:45,259 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:45,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:55:45,536 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:55:46,259 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:47,259 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:48,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:49,259 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:50,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:50,421 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:55:51,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:52,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:53,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:54,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:55,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:55,421 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:55:56,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:57,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:58,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:55:59,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:00,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:00,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:56:00,536 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:56:00,624 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:56:01,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:02,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:03,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:04,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:05,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:06,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:06,421 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:56:07,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:07,631 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:56:08,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:09,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:10,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:11,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:11,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:56:12,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:13,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:14,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:56:14,059 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:56:14,059 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:56:14,062 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:56:14,063 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:56:14,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:15,476 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:15,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:56:15,536 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:56:16,064 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:56:16,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:17,422 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:56:17,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:18,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:19,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:20,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:21,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:22,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:23,422 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:56:23,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:24,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:25,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:26,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:27,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:28,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:56:28,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:29,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:30,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:30,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:56:30,536 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:56:31,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:32,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:33,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:34,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:56:34,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:35,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:36,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:37,481 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:37,639 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:56:38,481 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:39,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:39,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:56:40,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:41,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:42,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:43,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:56:43,360 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:56:43,360 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:56:43,361 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:56:43,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:44,089 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:56:44,090 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:56:44,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:45,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:56:45,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:45,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:56:45,537 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:56:46,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:47,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:48,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:49,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:50,427 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:56:50,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:51,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:52,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:53,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:54,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:55,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:56,427 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:56:56,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:57,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:58,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:56:59,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:00,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:00,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:57:00,537 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:57:01,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:01,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:57:02,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:03,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:04,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:05,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:06,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:07,427 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:57:07,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:07,639 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:57:08,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:09,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:10,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:11,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:12,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:57:12,521 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:13,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:14,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:15,404 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:57:15,406 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:57:15,406 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:57:15,408 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:57:15,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:57:15,811 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:57:15,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:16,123 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:57:16,123 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:57:16,809 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:17,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:57:17,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:18,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:19,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:20,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:21,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:22,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:23,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:57:23,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:24,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:25,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:26,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:27,811 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:28,811 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:29,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:57:29,811 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:30,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:57:30,555 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:57:30,811 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:31,811 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:32,811 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:33,811 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:34,811 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:35,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:57:35,811 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:36,812 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:37,643 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:57:37,812 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:38,812 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:39,812 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:40,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:57:40,812 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:41,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:42,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:43,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:44,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:45,559 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:57:45,559 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:57:45,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:46,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:57:46,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:47,547 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:57:47,549 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:57:47,549 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:57:47,551 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:57:47,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:48,154 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:57:48,154 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:57:48,959 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:49,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:50,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:51,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:52,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:57:52,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:53,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:54,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:55,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:56,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:57,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:58,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:57:58,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:57:59,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:00,559 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:58:00,559 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:58:00,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:01,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:02,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:03,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:58:03,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:04,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:05,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:06,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:07,647 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:58:07,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:08,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:58:08,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:09,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:10,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:11,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:12,969 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:13,974 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:14,451 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:58:14,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:15,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:58:15,564 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:58:15,974 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:16,977 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:17,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:18,221 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:58:18,223 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:58:18,224 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:58:18,225 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:58:18,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:19,183 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:58:19,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:20,184 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:58:20,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:58:20,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:21,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:22,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:23,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:24,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:25,443 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:58:25,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:26,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:27,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:28,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:29,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:30,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:58:30,565 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:58:30,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:58:30,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:31,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:32,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:33,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:34,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:35,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:36,444 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:58:36,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:37,652 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:58:37,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:38,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:39,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:40,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:41,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:58:41,991 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:42,991 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:44,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:45,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:45,567 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:58:45,570 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:58:46,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:46,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:58:46,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:47,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:48,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:58:48,020 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:58:48,020 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:58:48,022 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:58:48,214 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:58:48,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:49,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:50,216 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:58:50,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:51,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:52,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:58:52,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:53,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:54,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:55,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:56,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:57,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:58:57,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:58,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:58:59,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:00,567 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:59:00,568 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:59:00,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:02,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:02,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:59:03,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:04,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:05,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:06,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:07,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:07,655 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:59:07,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:59:08,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:09,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:10,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:11,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:12,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:13,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:13,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:59:14,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:15,030 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:15,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:59:15,574 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:59:16,034 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:17,034 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:18,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:18,452 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:59:19,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:19,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:59:19,730 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:59:19,731 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:59:19,732 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:59:20,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:20,246 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:59:20,247 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:59:21,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:22,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:23,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:23,452 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:59:24,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:25,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:26,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:27,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:28,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:29,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:29,453 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:59:30,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:30,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:59:30,571 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:59:31,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:32,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:33,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:34,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:35,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:35,453 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:59:36,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:37,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:37,659 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-26 23:59:38,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:39,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:40,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:40,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:59:41,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:42,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:43,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:44,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:45,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:45,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-26 23:59:45,571 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-26 23:59:45,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:59:46,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:47,142 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:48,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:49,142 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:50,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:51,147 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:51,460 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:59:52,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:52,429 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-26 23:59:52,431 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-26 23:59:52,431 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-26 23:59:52,432 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-26 23:59:53,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:53,275 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-26 23:59:54,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:54,276 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-26 23:59:55,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:56,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:56,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-26 23:59:57,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:58,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-26 23:59:59,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:00,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:00,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:00:00,571 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:00:01,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:02,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:02,576 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:00:03,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:04,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:05,147 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:06,147 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:07,147 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:07,663 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:00:07,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:00:08,147 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:09,147 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:10,147 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:11,147 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:12,147 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:13,147 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:13,576 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:00:14,147 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:15,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:15,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:00:15,571 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:00:16,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:17,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:18,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:18,585 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:00:19,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:20,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:21,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:22,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:22,247 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:00:22,248 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:00:22,248 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:00:22,249 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:00:22,304 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:00:23,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:23,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:00:24,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:24,306 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:00:25,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:26,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:27,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:28,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:29,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:29,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:00:30,158 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:30,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:00:30,571 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:00:31,158 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:32,158 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:33,158 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:34,158 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:35,158 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:35,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:00:36,158 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:37,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:37,667 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:00:38,158 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:39,158 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:40,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:40,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:00:41,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:42,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:43,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:44,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:45,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:45,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:00:45,571 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:00:45,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:00:46,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:47,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:48,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:49,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:50,168 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:51,161 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:51,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:00:52,161 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:53,166 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:54,161 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:55,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:55,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:00:55,349 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:00:55,349 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:00:55,350 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:00:56,161 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:56,339 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:00:56,340 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:00:57,161 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:57,389 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:00:58,161 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:00:59,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:00,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:00,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:01:00,572 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:01:01,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:02,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:02,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:01:03,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:04,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:05,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:06,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:07,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:07,671 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:01:07,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:01:08,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:09,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:10,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:11,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:12,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:13,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:13,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:01:14,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:15,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:15,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:01:15,572 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:01:16,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:17,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:18,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:19,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:19,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:01:20,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:21,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:22,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:23,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:24,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:24,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:01:25,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:26,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:27,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:01:27,157 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:01:27,158 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:01:27,159 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:01:27,377 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:01:27,562 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:28,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:28,378 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:01:29,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:30,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:30,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:01:30,572 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:01:30,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:01:31,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:32,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:33,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:34,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:35,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:36,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:36,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:01:37,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:37,676 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:01:38,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:39,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:40,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:41,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:41,678 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:01:42,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:43,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:44,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:45,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:45,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:01:45,572 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:01:46,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:47,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:47,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:01:48,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:49,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:50,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:51,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:52,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:52,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:01:53,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:54,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:55,308 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:56,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:01:56,293 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:01:56,293 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:01:56,294 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:01:56,402 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:01:56,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:57,696 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:58,404 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:01:58,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:01:58,696 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:01:59,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:00,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:02:00,572 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:02:00,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:01,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:02,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:03,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:04,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:02:04,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:05,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:06,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:07,679 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:02:07,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:08,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:09,681 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:02:09,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:10,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:11,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:12,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:13,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:14,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:15,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:02:15,572 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:02:15,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:15,731 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:02:16,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:17,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:18,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:19,699 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:20,699 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:21,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:02:21,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:22,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:23,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:24,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:25,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:26,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:27,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:02:27,455 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:02:27,456 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:02:27,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:02:27,458 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:02:27,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:28,434 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:02:28,434 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:02:28,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:29,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:30,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:02:30,572 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:02:30,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:31,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:32,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:02:32,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:33,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:34,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:35,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:36,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:37,683 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:02:37,690 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:02:37,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:38,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:39,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:40,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:41,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:42,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:02:42,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:43,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:44,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:45,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:02:45,572 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:02:45,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:46,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:47,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:02:47,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:48,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:49,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:50,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:51,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:52,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:53,857 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:02:53,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:55,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:56,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:57,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:58,006 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:58,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:02:59,007 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:02:59,989 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:02:59,990 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:02:59,991 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:02:59,992 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:03:00,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:00,462 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:03:00,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:03:00,573 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:03:01,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:01,463 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:03:02,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:03,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:04,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:04,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:03:05,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:06,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:07,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:07,687 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:03:08,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:09,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:10,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:10,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:03:11,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:12,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:13,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:14,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:15,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:15,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:03:15,573 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:03:16,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:16,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:03:17,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:18,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:19,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:20,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:21,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:21,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:03:22,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:23,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:24,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:25,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:26,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:26,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:03:27,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:28,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:29,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:30,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:30,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:03:30,297 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:03:30,298 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:03:30,300 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:03:30,493 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:03:30,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:03:30,712 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:03:31,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:32,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:32,794 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:03:33,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:33,495 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:03:34,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:35,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:36,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:37,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:37,690 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:03:37,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:03:38,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:39,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:40,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:41,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:42,466 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:42,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:03:43,465 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:44,465 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:45,465 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:45,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:03:45,712 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:03:46,465 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:47,465 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:48,465 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:48,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:03:49,466 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:50,466 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:51,466 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:52,466 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:53,466 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:54,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:54,866 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:03:55,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:56,472 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:57,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:58,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:59,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:03:59,925 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:03:59,926 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:03:59,926 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:03:59,927 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:03:59,928 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:04:00,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:00,521 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:04:00,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:04:00,713 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:04:01,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:01,522 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:04:02,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:03,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:04,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:05,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:05,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:04:06,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:07,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:07,695 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:04:08,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:09,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:10,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:11,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:11,866 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:04:12,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:13,523 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:14,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:15,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:15,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:04:15,713 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:04:16,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:17,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:17,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:04:18,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:19,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:20,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:21,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:22,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:22,866 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:04:23,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:24,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:25,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:26,525 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:27,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:27,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:04:28,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:29,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:30,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:30,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:04:30,717 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:04:31,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:04:31,155 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:04:31,155 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:04:31,156 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:04:31,550 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:04:31,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:32,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:33,551 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:04:33,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:33,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:04:34,567 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:35,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:36,567 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:37,567 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:37,698 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:04:38,567 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:39,567 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:39,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:04:40,567 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:41,567 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:42,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:43,928 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:44,928 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:44,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:04:45,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:04:45,714 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:04:45,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:46,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:47,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:48,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:49,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:50,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:50,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:04:51,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:52,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:53,933 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:54,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:55,935 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:04:55,935 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:56,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:57,935 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:58,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:04:58,965 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:04:58,965 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:04:58,966 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:04:58,967 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:04:59,579 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:04:59,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:00,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:05:00,714 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:05:00,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:01,586 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:05:01,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:05:01,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:02,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:03,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:04,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:05,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:06,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:05:06,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:07,702 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:05:07,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:08,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:09,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:10,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:11,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:12,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:05:12,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:13,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:14,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:15,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:05:15,714 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:05:15,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:16,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:17,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:05:17,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:18,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:19,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:20,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:21,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:22,934 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:05:22,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:23,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:24,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:25,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:26,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:27,934 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:05:27,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:28,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:05:28,133 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:05:28,133 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:05:28,134 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:05:28,613 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:05:28,948 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:29,614 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:05:29,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:30,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:05:30,716 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:05:30,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:31,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:32,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:33,934 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:05:33,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:34,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:35,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:36,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:37,706 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:05:37,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:38,934 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:05:38,948 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:39,948 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:40,948 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:41,948 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:42,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:43,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:43,968 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:05:44,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:45,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:05:45,715 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:05:45,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:46,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:47,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:48,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:49,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:49,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:05:50,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:51,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:52,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:53,959 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:54,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:05:54,970 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:55,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:56,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:57,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:58,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:05:58,878 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:05:58,879 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:05:58,880 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:05:59,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:05:59,641 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:06:00,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:00,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:06:00,714 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:06:00,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:06:01,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:01,643 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:06:02,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:03,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:04,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:05,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:05,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:06:06,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:07,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:07,710 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:06:08,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:09,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:10,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:11,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:11,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:06:12,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:13,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:14,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:15,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:15,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:06:15,714 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:06:16,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:17,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:17,866 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:06:18,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:19,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:20,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:21,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:22,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:22,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:06:23,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:24,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:25,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:26,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:27,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:27,967 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:06:28,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:29,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:30,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:30,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:06:30,721 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:06:31,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:31,699 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:06:31,700 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:06:31,701 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:06:31,702 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:06:32,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:32,669 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:06:33,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:33,670 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:06:33,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:06:34,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:35,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:36,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:37,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:37,714 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:06:38,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:39,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:39,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:06:40,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:41,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:42,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:43,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:44,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:45,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:45,718 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:06:45,718 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:06:45,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:06:46,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:47,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:48,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:49,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:50,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:50,965 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:06:51,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:52,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:53,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:54,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:55,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:56,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:56,971 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:06:57,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:58,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:06:59,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:00,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:00,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:07:00,727 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:07:01,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:02,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:02,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:07:02,602 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:07:02,603 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:07:02,603 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:07:02,605 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:07:02,700 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:07:03,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:04,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:05,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:05,702 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:07:06,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:07,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:07,718 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:07:07,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:07:08,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:09,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:10,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:11,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:12,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:13,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:07:13,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:14,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:15,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:15,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:07:15,722 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:07:16,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:17,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:18,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:07:18,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:19,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:20,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:21,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:22,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:23,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:24,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:07:24,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:25,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:26,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:27,365 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:28,357 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:29,024 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:07:29,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:30,361 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:30,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:07:30,728 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:07:31,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:31,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:07:31,675 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:07:31,675 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:07:31,676 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:07:31,726 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:07:32,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:33,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:33,727 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:07:34,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:35,014 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:07:35,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:36,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:37,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:37,722 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:07:38,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:39,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:40,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:41,014 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:07:41,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:42,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:43,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:44,446 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:45,446 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:45,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:07:45,728 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:07:46,446 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:46,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:07:47,446 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:48,446 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:49,446 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:50,446 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:51,446 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:52,015 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:07:52,446 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:53,446 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:54,447 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:55,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:56,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:57,021 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:07:57,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:58,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:07:59,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:00,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:00,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:08:00,729 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:08:00,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:08:00,924 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:08:00,925 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:08:00,926 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:08:01,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:01,754 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:08:02,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:02,927 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:08:03,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:03,756 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:08:04,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:05,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:06,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:07,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:07,726 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:08:08,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:08:08,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:09,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:10,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:11,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:12,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:13,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:08:13,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:14,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:15,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:15,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:08:15,728 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:08:16,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:17,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:18,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:08:18,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:19,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:20,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:21,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:22,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:23,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:24,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:08:24,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:25,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:26,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:27,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:28,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:29,026 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:08:29,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:30,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:30,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:08:30,729 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:08:30,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:08:30,857 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:08:30,857 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:08:30,859 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:08:31,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:31,785 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:08:32,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:33,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:33,786 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:08:34,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:35,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:08:35,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:36,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:37,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:37,730 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:08:38,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:39,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:40,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:41,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:08:41,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:42,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:43,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:44,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:45,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:45,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:08:45,729 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:08:46,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:46,821 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:08:47,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:48,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:49,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:50,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:51,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:52,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:08:52,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:53,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:54,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:55,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:56,464 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:57,034 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:08:57,460 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:58,461 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:08:59,464 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:00,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:00,735 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:09:00,738 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:09:01,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:01,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:09:01,753 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:09:01,754 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:09:01,755 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:09:01,814 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:09:02,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:02,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:09:03,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:03,816 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:09:04,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:05,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:06,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:07,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:07,734 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:09:08,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:09:08,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:09,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:10,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:11,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:12,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:13,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:09:13,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:14,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:15,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:15,735 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:09:15,736 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:09:16,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:17,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:18,026 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:09:18,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:19,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:20,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:21,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:22,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:23,471 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:24,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:09:24,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:25,472 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:26,472 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:27,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:28,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:29,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:09:29,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:29,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:09:29,658 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:09:29,659 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:09:29,660 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:09:29,850 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:09:30,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:30,736 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:09:30,736 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:09:31,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:31,852 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:09:32,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:33,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:34,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:35,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:09:35,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:36,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:37,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:37,738 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:09:38,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:39,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:40,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:41,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:09:41,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:42,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:43,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:44,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:45,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:45,736 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:09:45,736 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:09:46,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:46,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:09:47,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:48,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:49,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:50,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:51,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:52,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:09:52,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:53,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:54,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:55,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:56,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:57,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:09:57,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:58,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:09:59,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:00,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:10:00,058 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:10:00,059 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:10:00,060 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:10:00,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:00,736 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:10:00,736 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:10:00,883 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:10:01,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:01,884 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:10:02,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:02,802 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:10:03,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:04,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:05,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:06,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:07,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:07,742 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:10:08,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:10:08,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:09,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:10,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:11,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:12,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:13,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:10:13,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:14,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:15,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:15,736 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:10:15,736 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:10:16,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:17,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:18,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:19,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:10:19,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:20,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:21,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:22,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:23,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:24,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:10:24,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:25,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:26,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:27,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:28,496 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:29,045 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:10:29,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:29,604 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:10:29,605 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:10:29,606 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:10:29,608 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:10:29,910 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:10:30,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:30,736 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:10:30,737 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:10:31,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:31,912 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:10:32,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:33,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:34,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:35,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:10:35,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:36,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:37,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:37,746 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:10:38,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:39,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:40,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:10:40,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:41,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:42,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:43,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:44,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:45,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:45,736 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:10:45,737 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:10:45,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:10:46,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:47,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:48,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:49,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:50,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:51,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:10:51,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:52,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:53,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:54,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:55,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:56,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:57,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:10:57,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:58,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:10:59,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:00,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:00,745 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:11:00,758 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:11:01,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:02,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:11:02,218 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:11:02,219 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:11:02,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:11:02,220 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:11:02,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:02,943 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:11:03,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:03,944 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:11:04,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:05,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:06,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:07,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:07,750 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:11:07,756 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:11:08,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:09,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:10,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:11,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:12,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:13,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:11:13,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:14,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:15,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:15,741 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:11:15,742 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:11:16,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:17,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:18,042 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:11:18,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:19,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:20,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:21,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:22,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:23,042 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:11:23,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:24,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:25,767 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:26,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:27,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:28,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:29,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:11:29,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:30,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:11:30,747 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:11:30,791 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:31,772 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:32,772 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:33,773 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:34,048 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:11:34,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:11:34,449 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:11:34,450 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:11:34,451 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:11:34,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:34,978 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:11:35,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:35,979 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:11:36,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:37,754 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:11:37,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:38,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:39,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:40,044 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:11:40,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:41,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:42,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:43,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:44,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:45,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:11:45,744 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:11:45,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:11:45,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:46,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:47,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:48,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:49,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:50,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:51,045 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:11:51,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:52,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:53,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:54,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:55,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:56,866 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:57,045 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:11:57,866 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:58,866 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:11:59,866 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:00,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:12:00,750 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:12:00,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:01,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:02,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:12:02,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:03,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:04,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:05,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:06,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:12:06,138 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:12:06,139 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:12:06,140 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:12:06,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:07,013 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:12:07,759 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:12:07,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:08,014 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:12:08,048 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:12:08,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:09,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:10,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:11,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:12,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:13,048 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:12:13,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:14,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:15,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:12:15,744 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:12:15,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:16,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:17,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:18,048 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:12:18,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:19,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:20,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:21,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:22,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:23,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:24,049 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:12:24,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:25,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:26,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:27,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:28,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:29,050 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:12:29,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:30,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:12:30,750 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:12:30,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:31,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:32,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:33,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:34,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:12:34,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:12:34,640 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:12:34,641 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:12:34,643 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:12:35,041 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:12:35,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:36,043 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:12:36,049 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:37,049 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:37,759 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:12:38,049 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:39,049 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:39,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:12:40,049 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:41,049 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:42,049 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:43,049 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:44,049 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:45,049 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:45,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:12:45,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:12:45,747 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:12:46,050 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:47,050 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:48,050 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:49,050 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:50,050 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:50,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:12:51,050 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:52,050 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:53,050 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:54,050 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:55,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:56,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:56,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:12:57,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:58,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:12:59,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:00,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:00,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:13:00,747 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:13:01,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:01,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:13:02,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:02,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:13:02,591 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:13:02,591 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:13:02,592 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:13:03,058 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:03,069 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:13:04,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:04,070 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:13:05,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:06,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:07,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:13:07,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:07,762 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:13:08,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:09,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:10,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:11,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:12,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:13,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:13:13,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:14,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:15,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:15,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:13:15,851 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:13:16,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:17,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:18,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:19,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:13:19,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:20,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:21,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:22,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:23,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:24,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:13:24,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:25,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:26,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:27,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:28,065 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:29,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:29,065 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:13:30,065 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:30,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:13:30,769 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:13:31,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:31,958 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:13:31,960 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:13:31,961 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:13:31,962 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:13:32,097 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:13:32,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:33,369 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:34,099 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:13:34,369 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:35,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:13:35,369 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:36,369 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:37,369 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:37,766 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:13:38,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:39,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:40,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:41,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:13:41,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:42,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:43,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:44,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:45,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:45,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:13:45,766 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:13:46,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:46,935 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:13:47,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:48,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:49,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:50,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:51,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:52,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:13:52,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:53,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:54,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:55,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:56,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:57,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:13:57,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:58,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:13:59,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:00,115 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:14:00,117 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:14:00,117 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:14:00,118 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:14:00,127 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:14:00,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:00,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:14:00,766 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:14:01,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:02,129 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:14:02,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:02,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:14:03,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:04,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:05,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:06,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:07,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:07,770 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:14:08,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:14:08,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:09,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:10,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:11,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:12,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:13,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:14:13,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:14,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:15,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:15,767 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:14:15,767 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:14:16,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:17,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:18,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:14:18,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:19,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:20,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:21,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:22,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:23,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:24,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:14:24,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:25,541 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:26,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:27,542 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:28,540 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:29,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:14:29,538 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:30,538 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:30,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:14:30,672 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:14:30,672 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:14:30,673 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:14:31,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:14:31,079 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:14:31,160 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:14:31,538 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:32,161 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:14:32,538 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:33,538 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:34,538 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:35,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:14:35,538 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:36,538 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:37,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:37,774 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:14:38,538 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:39,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:40,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:41,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:14:41,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:42,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:43,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:44,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:45,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:45,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:14:45,819 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:14:46,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:46,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:14:47,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:48,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:49,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:50,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:51,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:52,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:14:52,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:53,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:54,551 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:55,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:56,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:57,076 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:14:57,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:58,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:14:59,363 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:14:59,364 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:14:59,365 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:14:59,366 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:14:59,778 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:00,194 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:15:00,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:00,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:15:00,819 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:15:01,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:02,196 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:15:02,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:02,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:15:03,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:04,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:05,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:06,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:07,778 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:07,779 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:15:08,074 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:15:08,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:09,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:10,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:11,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:12,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:13,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:14,074 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:15:14,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:15,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:15:15,819 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:15:15,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:16,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:17,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:18,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:19,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:20,074 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:15:20,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:21,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:22,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:23,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:24,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:25,075 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:15:25,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:26,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:27,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:28,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:29,835 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:15:29,835 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:15:29,836 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:15:29,837 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:15:30,232 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:15:30,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:30,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:15:30,819 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:15:30,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:15:31,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:32,233 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:15:32,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:33,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:34,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:35,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:36,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:15:36,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:37,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:37,782 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:15:38,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:39,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:40,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:41,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:42,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:15:42,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:43,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:44,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:45,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:45,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:15:45,819 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:15:46,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:47,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:47,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:15:48,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:49,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:50,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:51,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:52,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:53,085 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:15:53,258 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:54,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:55,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:56,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:57,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:58,087 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:15:58,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:15:58,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:15:58,724 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:15:58,724 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:15:58,726 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:15:59,261 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:15:59,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:00,262 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:16:00,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:00,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:16:00,819 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:16:01,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:02,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:03,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:04,087 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:16:04,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:05,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:06,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:07,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:07,786 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:16:08,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:09,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:10,087 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:16:10,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:11,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:12,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:13,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:14,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:15,088 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:16:15,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:15,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:16:15,820 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:16:16,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:17,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:18,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:19,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:20,088 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:16:20,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:21,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:22,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:23,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:24,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:25,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:16:25,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:26,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:27,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:28,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:28,695 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:16:28,696 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:16:28,697 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:16:28,698 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:16:29,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:29,293 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:16:30,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:30,294 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:16:30,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:16:30,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:16:30,819 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:16:31,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:32,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:33,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:34,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:35,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:36,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:16:36,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:37,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:37,790 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:16:38,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:39,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:40,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:41,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:16:41,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:42,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:43,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:44,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:45,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:45,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:16:45,820 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:16:46,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:46,895 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:16:47,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:48,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:49,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:50,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:51,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:52,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:16:52,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:53,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:54,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:55,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:56,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:57,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:16:57,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:58,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:16:58,157 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:16:58,158 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:16:58,160 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:16:58,325 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:16:58,574 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:16:59,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:00,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:00,326 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:17:00,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:17:00,820 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:17:01,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:02,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:02,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:17:03,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:04,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:05,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:06,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:07,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:07,794 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:17:08,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:17:08,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:09,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:10,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:11,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:12,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:13,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:17:13,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:14,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:15,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:15,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:17:15,820 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:17:16,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:17,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:18,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:19,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:17:19,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:20,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:21,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:22,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:23,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:24,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:17:24,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:25,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:26,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:27,317 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:28,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:28,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:17:28,834 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:17:28,834 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:17:28,836 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:17:29,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:29,353 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:17:29,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:17:30,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:30,354 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:17:30,820 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:17:30,821 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:17:31,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:32,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:33,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:34,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:35,092 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:17:35,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:36,317 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:37,317 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:37,799 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:17:38,317 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:39,317 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:40,317 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:41,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:17:41,317 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:42,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:43,317 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:44,317 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:45,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:45,820 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:17:45,820 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:17:46,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:46,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:17:47,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:48,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:49,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:50,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:51,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:52,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:17:52,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:53,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:54,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:55,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:56,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:57,103 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:17:57,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:58,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:59,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:17:59,420 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:17:59,421 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:17:59,422 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:17:59,424 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:18:00,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:00,377 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:18:00,377 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:18:00,820 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:18:00,821 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:18:01,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:02,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:02,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:18:03,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:04,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:05,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:06,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:07,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:07,802 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:18:08,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:18:08,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:09,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:10,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:11,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:12,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:13,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:18:13,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:14,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:15,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:15,820 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:18:15,821 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:18:16,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:17,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:18,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:19,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:18:19,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:20,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:21,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:22,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:23,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:24,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:25,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:18:25,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:26,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:27,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:28,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:29,328 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:18:29,329 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:18:29,330 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:18:29,332 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:18:29,369 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:29,404 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:18:30,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:30,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:18:30,405 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:18:30,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:18:30,828 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:18:31,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:32,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:33,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:34,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:35,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:36,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:18:36,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:37,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:37,806 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:18:38,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:39,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:40,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:41,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:18:41,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:42,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:43,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:44,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:45,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:45,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:18:45,827 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:18:46,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:46,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:18:47,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:48,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:49,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:50,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:51,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:52,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:18:52,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:53,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:54,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:55,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:56,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:57,106 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:18:57,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:58,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:18:59,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:00,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:00,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:19:00,827 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:19:01,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:01,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:19:01,778 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:19:01,778 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:19:01,780 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:19:02,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:02,433 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:19:02,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:19:03,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:04,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:04,434 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:19:05,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:06,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:07,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:07,810 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:19:08,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:19:08,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:09,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:10,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:11,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:12,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:13,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:19:13,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:14,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:15,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:15,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:19:15,828 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:19:16,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:17,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:18,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:19:18,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:19,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:20,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:21,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:22,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:23,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:19:23,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:24,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:25,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:26,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:27,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:28,107 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:19:28,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:29,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:30,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:30,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:19:30,835 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:19:31,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:32,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:32,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:19:32,716 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:19:32,717 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:19:32,718 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:19:33,131 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:19:33,462 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:19:33,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:34,463 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:19:34,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:35,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:36,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:37,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:37,814 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:19:38,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:39,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:19:39,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:40,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:41,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:42,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:43,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:44,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:19:44,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:45,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:45,959 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:19:45,960 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:19:46,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:47,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:48,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:49,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:19:49,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:50,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:51,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:52,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:53,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:54,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:19:54,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:55,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:56,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:57,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:58,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:19:59,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:00,131 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:20:00,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:00,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:20:00,876 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:20:01,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:02,524 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:03,525 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:04,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:20:04,306 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:20:04,307 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:20:04,309 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:20:04,488 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:20:04,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:05,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:20:05,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:06,490 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:20:06,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:07,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:07,818 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:20:08,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:09,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:10,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:11,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:20:11,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:12,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:13,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:14,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:15,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:15,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:20:15,872 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:20:16,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:17,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:20:17,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:18,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:19,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:20,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:21,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:22,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:20:22,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:23,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:24,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:25,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:26,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:27,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:20:27,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:28,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:29,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:30,729 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:30,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:20:30,876 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:20:31,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:32,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:32,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:20:32,852 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:20:32,852 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:20:32,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:20:32,854 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:20:33,517 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:20:33,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:34,518 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:20:34,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:35,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:36,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:37,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:37,822 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:20:38,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:20:38,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:39,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:40,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:41,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:42,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:43,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:20:43,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:44,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:45,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:20:45,938 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:20:45,977 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:46,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:47,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:48,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:20:48,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:49,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:50,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:51,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:52,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:53,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:54,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:20:54,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:55,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:56,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:57,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:58,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:20:59,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:20:59,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:00,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:00,935 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:21:00,935 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:21:01,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:02,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:03,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:21:03,416 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:21:03,416 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:21:03,418 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:21:03,551 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:21:03,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:04,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:21:04,552 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:21:04,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:05,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:06,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:07,826 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:21:07,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:08,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:09,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:10,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:21:10,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:11,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:12,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:13,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:14,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:15,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:15,935 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:21:15,936 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:21:16,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:21:16,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:17,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:18,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:19,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:20,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:21,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:21:21,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:22,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:23,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:24,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:25,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:26,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:27,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:21:27,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:28,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:29,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:30,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:30,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:21:30,947 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:21:31,857 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:32,857 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:33,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:21:33,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:34,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:35,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:36,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:21:36,862 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:21:36,862 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:21:36,863 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:21:36,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:37,583 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:21:37,830 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:21:37,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:38,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:21:38,584 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:21:38,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:39,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:40,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:41,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:42,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:43,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:21:43,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:44,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:45,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:45,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:21:45,939 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:21:46,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:47,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:48,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:49,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:21:49,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:50,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:51,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:52,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:53,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:54,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:21:54,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:55,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:56,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:57,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:58,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:21:59,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:21:59,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:00,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:00,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:22:00,958 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:22:01,867 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:02,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:03,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:04,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:22:04,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:05,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:06,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:22:06,825 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:22:06,825 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:22:06,826 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:22:07,238 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:07,612 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:22:07,834 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:22:07,976 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:08,613 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:22:08,976 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:09,976 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:10,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:22:10,976 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:11,976 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:12,976 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:13,976 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:14,976 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:15,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:22:15,943 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:22:15,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:16,108 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:22:16,977 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:17,977 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:18,977 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:19,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:20,977 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:21,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:22:21,977 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:22,977 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:23,977 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:24,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:25,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:26,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:22:26,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:27,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:28,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:29,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:30,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:22:30,954 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:22:30,988 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:31,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:32,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:22:32,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:33,990 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:34,991 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:35,988 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:36,809 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:22:36,811 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:22:36,812 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:22:36,813 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:22:37,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:37,646 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:22:37,838 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:22:37,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:22:38,223 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:38,647 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:22:39,223 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:40,223 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:41,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:42,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:43,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:43,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:22:44,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:45,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:45,946 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:22:45,947 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:22:46,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:47,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:48,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:48,472 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:22:49,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:50,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:51,225 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:52,225 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:53,225 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:54,225 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:54,373 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:22:55,225 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:56,225 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:57,228 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:58,226 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:59,230 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:22:59,382 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:23:00,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:00,946 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:23:00,947 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:23:01,228 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:02,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:02,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:23:02,851 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:23:02,851 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:23:02,853 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:23:03,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:03,673 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:23:04,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:04,674 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:23:05,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:05,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:23:06,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:07,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:07,843 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:23:08,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:09,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:10,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:10,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:23:11,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:12,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:13,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:14,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:15,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:15,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:23:15,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:23:15,947 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:23:16,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:17,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:18,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:19,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:20,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:21,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:21,377 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:23:22,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:23,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:24,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:25,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:26,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:26,377 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:23:27,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:28,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:29,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:30,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:30,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:23:30,953 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:23:31,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:32,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:23:32,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:32,419 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:23:32,420 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:23:32,420 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:23:32,422 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:23:32,707 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:23:32,708 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:23:33,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:34,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:35,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:36,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:37,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:37,380 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:23:37,846 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:23:38,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:39,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:40,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:41,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:42,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:43,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:43,379 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:23:44,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:45,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:45,953 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:23:45,953 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:23:46,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:47,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:48,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:48,380 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:23:49,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:50,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:51,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:52,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:53,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:54,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:54,380 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:23:55,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:56,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:57,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:58,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:59,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:23:59,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:24:00,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:00,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:24:00,958 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:24:01,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:02,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:03,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:04,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:04,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:24:04,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:24:04,449 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:24:04,450 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:24:04,451 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:24:04,738 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:24:04,739 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:24:05,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:06,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:07,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:07,850 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:24:08,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:09,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:10,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:10,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:24:11,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:12,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:13,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:14,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:15,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:15,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:24:15,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:24:15,955 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:24:16,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:17,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:18,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:19,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:20,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:21,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:21,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:24:22,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:23,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:24,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:25,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:26,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:26,386 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:24:27,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:28,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:29,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:30,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:30,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:24:30,955 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:24:31,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:32,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:24:32,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:33,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:34,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:35,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:36,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:37,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:37,404 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:24:37,855 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:24:38,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:39,092 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:24:39,093 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:24:39,093 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:24:39,094 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:24:39,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:39,771 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:24:40,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:40,772 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:24:41,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:42,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:43,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:24:43,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:44,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:45,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:45,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:24:45,955 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:24:46,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:47,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:48,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:24:48,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:49,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:50,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:51,543 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:52,542 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:53,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:24:53,542 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:54,542 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:55,542 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:56,542 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:57,542 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:58,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:24:58,542 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:24:59,542 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:00,542 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:00,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:25:00,955 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:25:01,543 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:02,543 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:03,543 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:04,394 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:25:04,549 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:05,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:06,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:07,549 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:07,864 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:25:08,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:09,392 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:25:09,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:09,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:25:09,611 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:25:09,611 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:25:09,613 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:25:09,803 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:25:10,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:10,804 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:25:11,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:12,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:13,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:14,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:25:14,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:15,547 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:15,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:25:15,955 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:25:16,547 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:17,547 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:18,547 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:19,547 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:20,392 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:25:20,547 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:21,547 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:22,547 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:23,547 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:24,547 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:25,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:25:25,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:26,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:27,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:28,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:29,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:30,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:30,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:25:30,956 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:25:31,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:25:31,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:32,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:33,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:34,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:35,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:36,396 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:25:36,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:37,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:37,865 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:25:38,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:39,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:40,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:40,741 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:25:40,743 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:25:40,743 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:25:40,744 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:25:40,832 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:25:41,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:41,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:25:42,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:42,834 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:25:43,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:44,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:45,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:45,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:25:45,956 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:25:46,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:47,074 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:25:47,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:48,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:49,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:50,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:51,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:52,397 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:25:52,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:53,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:54,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:55,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:56,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:57,397 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:25:57,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:58,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:25:59,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:00,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:00,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:26:00,956 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:26:01,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:02,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:03,065 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:26:03,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:04,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:05,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:06,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:07,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:07,875 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:26:08,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:26:08,556 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:09,556 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:10,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:10,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:26:10,648 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:26:10,649 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:26:10,650 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:26:10,863 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:26:11,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:12,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:12,865 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:26:13,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:26:13,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:14,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:15,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:15,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:26:15,956 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:26:16,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:17,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:18,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:19,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:26:19,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:20,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:21,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:22,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:23,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:24,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:26:24,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:25,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:26,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:27,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:28,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:29,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:26:29,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:30,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:30,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:26:30,956 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:26:31,559 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:32,559 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:33,559 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:34,404 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:26:34,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:35,565 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:36,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:37,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:37,869 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:26:38,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:39,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:40,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:26:40,388 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:26:40,388 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:26:40,388 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:26:40,391 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:26:40,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:40,895 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:26:40,895 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:26:41,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:42,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:43,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:44,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:45,404 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:26:45,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:45,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:26:45,956 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:26:46,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:47,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:48,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:49,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:50,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:51,404 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:26:51,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:52,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:53,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:54,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:55,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:56,404 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:26:56,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:57,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:58,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:26:59,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:00,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:00,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:27:00,956 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:27:01,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:02,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:27:02,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:03,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:04,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:05,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:06,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:07,408 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:27:07,874 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:27:07,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:08,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:09,319 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:27:09,321 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:27:09,321 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:27:09,323 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:27:09,920 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:27:09,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:10,921 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:27:10,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:11,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:12,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:13,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:27:13,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:14,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:15,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:15,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:27:15,956 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:27:16,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:17,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:18,410 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:27:18,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:19,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:20,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:22,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:23,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:23,411 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:27:24,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:25,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:26,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:27,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:28,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:29,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:29,411 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:27:30,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:30,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:27:30,956 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:27:31,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:32,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:33,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:34,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:34,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:27:35,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:36,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:37,147 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:37,882 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:27:38,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:39,104 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:27:39,105 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:27:39,106 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:27:39,107 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:27:39,523 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:39,954 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:27:40,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:27:40,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:40,955 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:27:41,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:42,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:43,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:44,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:45,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:45,419 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:27:45,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:27:45,957 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:27:46,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:47,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:48,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:49,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:50,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:51,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:51,419 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:27:52,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:53,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:54,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:55,394 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:56,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:56,420 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:27:57,394 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:58,394 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:27:59,394 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:00,394 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:00,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:28:00,957 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:28:01,394 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:02,026 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:28:02,394 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:03,394 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:04,394 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:05,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:06,397 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:07,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:07,429 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:28:07,885 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:28:08,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:09,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:10,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:11,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:28:11,271 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:28:11,271 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:28:11,274 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:28:11,985 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:28:12,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:12,496 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:12,986 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:28:13,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:28:13,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:14,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:15,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:15,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:28:15,957 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:28:16,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:17,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:18,422 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:28:18,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:19,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:20,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:21,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:22,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:23,422 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:28:23,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:24,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:25,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:26,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:27,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:28,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:28:28,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:29,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:30,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:30,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:28:30,957 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:28:31,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:32,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:33,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:34,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:28:34,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:35,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:36,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:37,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:37,882 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:28:38,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:39,429 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:28:39,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:40,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:41,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:42,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:43,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:44,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:44,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:28:44,559 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:28:44,560 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:28:44,560 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:28:44,565 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:28:45,027 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:28:45,028 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:28:45,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:45,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:28:45,957 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:28:46,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:47,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:48,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:49,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:50,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:28:50,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:51,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:52,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:53,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:54,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:55,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:56,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:28:56,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:57,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:58,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:28:59,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:00,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:00,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:29:00,957 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:29:01,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:02,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:29:02,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:03,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:04,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:05,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:06,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:07,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:29:07,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:07,886 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:29:08,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:09,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:10,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:11,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:12,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:29:12,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:13,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:14,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:15,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:15,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:29:15,961 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:29:16,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:17,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:29:17,616 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:29:17,616 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:29:17,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:29:17,622 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:29:18,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:18,064 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:29:18,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:19,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:20,066 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:29:20,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:21,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:22,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:23,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:29:23,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:24,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:25,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:26,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:27,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:28,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:29,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:29:30,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:30,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:30,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:29:30,962 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:29:31,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:32,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:33,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:34,426 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:29:34,764 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:35,764 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:36,764 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:37,764 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:37,890 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:29:38,764 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:39,764 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:40,426 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:29:40,764 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:41,772 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:42,767 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:43,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:44,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:45,429 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:29:45,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:45,971 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:29:45,971 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:29:46,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:47,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:29:47,656 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:29:47,656 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:29:47,662 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:29:48,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:48,096 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:29:48,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:49,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:50,098 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:29:50,429 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:29:50,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:51,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:52,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:53,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:54,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:55,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:56,429 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:29:56,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:57,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:58,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:29:59,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:00,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:00,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:30:00,964 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:30:01,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:02,076 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:30:02,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:03,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:04,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:05,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:06,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:07,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:30:07,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:07,894 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:30:08,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:09,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:10,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:11,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:12,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:30:12,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:13,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:14,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:15,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:15,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:30:15,967 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:30:16,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:16,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:30:16,878 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:30:16,879 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:30:16,880 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:30:17,125 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:30:17,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:17,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:30:18,126 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:30:18,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:19,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:20,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:21,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:22,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:23,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:30:23,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:24,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:25,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:26,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:27,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:28,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:29,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:30:29,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:30,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:30,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:30:30,967 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:30:31,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:32,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:33,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:34,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:30:34,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:35,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:36,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:37,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:37,898 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:30:38,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:39,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:40,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:30:40,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:41,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:42,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:43,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:44,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:45,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:30:45,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:45,976 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:30:45,982 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:30:46,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:47,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:48,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:49,412 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:30:49,413 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:30:49,414 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:30:49,416 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:30:49,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:50,153 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:30:50,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:30:50,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:51,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:52,155 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:30:52,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:53,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:54,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:55,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:56,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:30:56,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:57,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:58,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:30:59,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:00,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:00,969 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:31:00,970 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:31:01,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:02,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:31:02,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:03,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:04,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:05,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:06,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:07,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:31:07,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:07,902 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:31:08,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:09,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:10,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:11,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:12,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:13,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:31:16,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:16,173 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:31:18,635 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:31:22,821 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:23,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:31:23,333 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:31:23,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:23,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:31:24,794 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:28,980 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:31:31,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:32,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:31:32,861 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:31:33,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:33,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:31:37,906 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:31:39,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:31:41,629 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:42,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:45,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:31:49,166 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:49,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:31:49,734 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:31:50,605 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:50,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:31:55,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:31:57,565 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:31:59,178 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:01,096 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:32:06,097 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:32:07,910 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:32:10,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:10,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:32:10,534 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:32:10,574 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:11,420 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:32:11,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:11,581 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:32:12,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:13,421 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:32:13,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:14,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:15,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:16,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:17,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:32:17,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:18,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:19,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:20,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:20,729 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:32:20,730 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:32:21,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:22,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:22,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:32:23,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:24,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:25,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:26,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:27,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:28,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:32:28,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:29,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:30,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:31,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:32,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:33,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:32:33,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:34,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:35,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:35,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:32:35,825 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:32:36,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:37,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:37,914 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:32:38,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:38,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:32:39,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:40,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:41,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:42,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:43,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:44,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:32:44,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:45,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:46,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:47,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:48,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:49,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:50,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:32:50,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:50,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:32:50,816 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:32:51,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:52,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:53,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:54,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:55,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:56,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:32:56,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:57,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:58,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:32:59,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:00,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:01,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:33:06,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:33:07,922 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:33:11,925 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:33:16,926 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:33:21,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:33:21,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:21,639 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:33:22,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:22,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:33:23,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:24,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:25,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:26,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:27,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:28,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:33:28,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:29,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:30,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:31,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:32,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:33,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:34,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:33:34,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:35,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:36,632 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:33:36,633 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:33:36,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:37,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:37,922 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:33:38,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:39,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:39,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:33:40,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:41,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:42,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:43,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:44,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:45,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:33:45,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:46,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:47,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:48,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:49,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:50,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:51,632 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:33:51,633 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:33:51,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:51,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:33:52,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:53,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:54,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:55,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:56,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:57,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:33:57,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:58,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:33:59,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:00,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:01,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:02,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:34:02,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:03,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:04,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:05,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:06,633 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:34:06,633 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:34:06,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:07,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:07,699 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:34:07,927 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:34:08,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:09,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:10,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:11,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:12,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:13,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:34:13,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:14,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:15,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:16,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:17,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:18,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:34:18,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:19,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:20,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:21,633 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:34:21,633 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:34:21,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:22,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:23,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:23,771 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:34:24,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:25,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:26,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:27,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:28,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:29,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:34:29,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:30,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:31,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:32,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:33,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:34,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:35,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:34:35,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:36,633 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:34:36,634 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:34:36,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:37,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:37,930 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:34:38,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:39,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:40,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:40,932 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:34:41,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:42,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:43,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:44,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:45,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:46,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:34:46,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:47,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:48,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:49,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:50,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:51,633 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:34:51,634 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:34:51,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:51,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:34:52,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:53,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:54,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:55,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:56,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:57,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:34:57,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:34:58,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:02,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:35:07,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:35:07,934 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:35:12,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:35:17,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:35:22,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:35:27,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:35:32,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:35:37,937 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:35:38,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:35:40,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:40,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:35:40,304 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:35:41,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:43,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:43,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:43,798 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:35:44,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:35:44,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:44,799 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:35:45,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:46,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:47,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:47,802 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:35:48,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:49,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:49,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:35:50,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:51,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:52,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:53,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:54,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:55,299 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:35:55,300 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:35:55,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:55,416 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:35:56,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:57,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:58,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:35:59,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:00,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:01,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:36:01,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:02,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:03,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:04,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:05,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:06,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:07,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:36:07,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:07,941 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:36:08,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:09,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:10,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:36:10,305 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:36:10,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:11,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:12,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:12,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:36:12,802 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:36:12,804 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:36:12,804 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:36:12,806 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:36:12,830 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:36:13,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:13,832 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:36:14,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:15,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:16,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:17,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:18,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:36:18,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:19,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:20,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:21,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:22,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:23,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:24,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:36:24,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:25,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:36:25,305 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:36:25,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:26,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:27,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:28,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:29,379 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:36:29,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:30,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:31,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:32,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:33,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:34,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:35,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:36:35,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:36,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:37,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:37,946 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:36:38,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:39,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:40,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:36:40,317 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:36:40,416 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:36:40,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:41,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:42,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:43,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:44,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:36:44,203 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:36:44,204 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:36:44,205 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:36:44,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:44,874 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:36:45,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:45,875 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:36:46,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:36:46,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:47,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:48,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:49,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:50,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:51,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:36:51,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:52,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:53,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:54,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:55,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:36:55,308 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:36:55,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:56,463 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:36:56,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:57,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:58,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:36:59,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:00,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:01,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:02,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:37:02,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:03,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:04,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:05,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:06,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:07,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:07,949 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:37:07,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:37:08,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:09,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:10,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:37:10,316 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:37:10,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:11,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:12,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:13,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:37:13,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:14,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:15,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:15,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:37:15,912 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:37:15,913 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:37:15,914 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:37:16,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:16,905 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:37:17,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:17,905 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:37:18,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:18,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:37:19,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:20,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:21,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:22,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:23,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:24,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:37:24,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:25,308 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:37:25,309 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:37:25,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:26,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:27,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:28,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:29,443 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:37:29,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:30,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:31,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:32,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:33,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:34,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:37:34,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:35,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:36,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:37,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:37,953 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:37:38,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:39,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:39,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:37:40,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:37:40,309 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:37:40,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:41,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:42,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:43,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:44,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:45,419 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:37:45,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:46,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:47,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:48,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:49,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:50,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:37:50,173 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:37:50,174 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:37:50,176 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:37:50,576 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:37:50,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:50,937 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:37:51,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:51,939 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:37:52,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:53,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:54,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:55,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:37:55,309 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:37:55,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:56,575 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:37:56,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:57,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:58,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:37:59,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:00,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:01,575 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:38:01,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:02,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:03,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:04,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:05,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:06,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:07,576 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:38:07,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:07,957 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:38:08,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:09,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:10,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:38:10,309 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:38:10,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:11,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:12,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:13,576 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:38:13,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:14,736 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:15,729 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:16,729 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:17,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:18,587 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:38:18,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:19,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:20,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:38:20,441 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:38:20,442 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:38:20,444 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:38:20,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:20,971 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:38:21,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:21,972 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:38:22,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:23,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:23,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:38:24,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:25,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:38:25,310 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:38:25,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:26,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:27,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:28,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:29,588 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:38:29,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:30,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:31,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:32,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:33,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:34,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:38:35,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:35,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:36,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:37,961 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:38:37,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:38,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:39,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:38:39,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:40,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:38:40,310 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:38:40,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:41,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:42,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:43,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:44,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:45,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:38:45,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:46,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:47,987 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:48,987 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:49,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:50,594 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:38:50,988 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:51,984 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:52,984 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:53,984 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:54,014 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:38:54,016 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:38:54,017 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:38:54,018 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:38:54,984 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:55,010 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:38:55,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:38:55,310 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:38:55,594 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:38:55,984 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:56,011 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:38:56,984 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:57,984 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:58,984 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:38:59,984 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:00,594 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:39:00,985 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:01,985 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:02,985 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:03,985 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:04,985 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:05,594 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:39:05,985 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:07,186 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:07,965 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:39:08,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:09,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:10,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:10,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:39:10,310 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:39:10,594 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:39:11,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:12,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:13,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:14,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:15,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:15,595 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:39:16,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:17,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:18,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:19,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:20,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:20,604 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:39:21,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:22,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:23,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:24,185 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:25,186 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:25,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:39:25,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:39:25,438 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:39:25,460 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:39:25,460 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:39:25,461 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:39:25,884 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:39:26,039 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:39:26,040 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:39:26,185 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:27,185 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:28,185 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:29,186 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:30,186 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:31,186 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:31,883 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:39:32,186 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:33,186 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:34,186 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:35,186 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:36,186 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:37,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:37,883 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:39:37,970 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:39:38,186 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:39,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:40,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:40,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:39:40,310 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:39:41,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:42,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:43,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:43,883 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:39:44,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:45,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:46,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:47,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:48,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:48,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:39:49,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:50,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:51,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:52,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:52,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:39:52,538 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:39:52,538 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:39:52,540 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:39:53,067 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:39:53,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:54,068 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:39:54,211 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:54,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:39:55,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:55,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:39:55,310 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:39:56,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:57,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:58,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:59,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:39:59,888 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:40:00,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:01,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:02,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:03,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:04,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:05,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:05,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:40:06,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:07,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:07,973 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:40:08,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:09,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:10,206 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:10,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:40:10,478 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:40:10,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:40:11,206 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:12,206 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:13,206 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:14,206 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:15,206 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:16,206 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:16,895 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:40:17,208 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:18,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:19,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:20,214 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:21,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:21,896 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:40:22,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:22,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:40:22,568 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:40:22,569 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:40:22,570 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:40:23,097 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:40:23,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:24,098 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:40:24,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:25,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:25,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:40:25,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:40:26,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:27,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:27,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:40:28,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:29,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:30,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:31,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:32,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:32,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:40:33,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:34,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:35,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:36,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:37,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:37,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:40:37,977 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:40:38,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:39,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:40,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:40,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:40:40,310 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:40:41,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:42,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:42,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:40:43,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:44,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:45,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:46,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:47,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:48,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:48,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:40:49,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:50,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:51,230 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:52,222 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:53,222 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:53,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:40:54,223 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:55,223 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:55,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:40:55,316 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:40:56,223 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:56,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:40:56,852 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:40:56,852 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:40:56,854 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:40:57,127 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:40:57,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:58,128 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:40:58,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:59,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:40:59,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:41:00,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:01,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:02,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:03,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:04,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:04,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:41:05,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:06,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:07,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:07,981 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:41:08,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:09,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:09,984 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:41:10,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:10,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:41:10,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:41:11,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:12,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:13,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:14,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:15,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:15,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:41:16,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:17,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:18,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:19,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:20,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:20,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:41:21,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:22,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:22,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:41:22,504 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:41:22,505 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:41:22,507 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:41:23,154 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:41:23,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:24,156 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:41:24,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:25,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:25,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:41:25,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:41:25,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:41:26,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:27,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:28,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:29,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:30,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:30,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:41:31,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:32,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:33,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:34,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:35,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:36,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:36,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:41:37,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:37,985 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:41:38,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:39,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:40,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:40,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:41:40,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:41:41,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:42,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:42,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:41:43,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:44,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:45,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:46,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:47,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:47,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:41:48,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:49,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:50,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:51,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:52,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:53,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:53,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:41:53,661 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:41:53,662 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:41:53,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:41:53,662 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:41:54,182 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:41:54,183 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:41:54,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:55,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:55,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:41:55,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:41:56,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:57,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:58,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:41:58,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:41:59,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:00,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:01,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:02,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:03,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:03,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:42:04,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:05,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:06,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:07,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:07,990 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:42:08,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:08,992 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:42:09,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:10,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:10,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:42:10,311 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:42:11,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:12,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:13,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:14,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:14,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:42:15,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:16,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:17,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:18,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:19,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:19,917 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:42:20,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:21,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:22,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:23,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:23,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:42:23,312 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:42:23,312 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:42:23,314 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:42:24,212 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:42:24,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:25,213 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:42:25,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:25,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:42:25,312 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:42:25,415 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:42:26,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:27,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:28,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:29,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:30,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:30,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:42:31,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:32,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:33,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:34,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:35,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:35,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:42:36,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:37,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:37,994 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:42:38,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:39,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:40,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:40,311 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:42:40,312 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:42:40,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:42:41,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:42,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:43,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:44,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:45,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:45,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:42:46,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:47,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:48,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:49,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:50,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:50,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:42:51,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:52,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:53,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:54,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:55,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:55,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:42:55,316 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:42:56,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:56,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:42:56,611 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:42:56,612 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:42:56,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:42:56,613 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:42:57,247 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:42:57,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:58,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:42:59,249 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:42:59,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:00,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:01,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:01,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:43:02,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:03,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:04,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:05,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:06,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:07,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:07,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:43:07,998 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:43:08,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:09,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:10,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:10,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:43:10,315 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:43:11,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:12,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:12,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:43:13,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:14,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:15,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:16,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:17,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:17,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:43:18,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:19,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:20,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:21,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:22,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:22,921 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:43:23,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:24,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:25,308 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:25,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:43:25,325 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:43:26,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:27,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:27,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:43:27,352 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:43:27,352 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:43:27,353 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:43:28,277 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:43:28,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:28,392 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:43:29,278 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:43:29,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:30,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:31,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:32,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:33,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:33,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:43:34,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:35,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:36,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:37,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:38,002 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:43:38,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:39,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:43:39,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:40,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:40,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:43:40,322 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:43:41,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:42,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:43,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:44,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:44,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:43:45,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:46,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:47,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:48,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:49,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:49,927 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:43:50,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:51,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:52,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:53,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:54,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:54,967 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:43:54,968 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:43:54,968 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:43:54,969 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:43:54,970 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:43:55,302 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:43:55,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:43:55,386 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:43:55,392 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:56,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:57,304 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:43:57,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:58,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:43:59,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:00,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:00,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:44:01,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:02,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:03,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:04,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:05,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:06,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:06,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:44:07,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:08,005 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:44:08,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:09,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:10,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:44:10,384 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:44:10,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:11,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:12,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:12,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:44:13,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:14,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:15,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:16,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:17,386 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:18,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:18,922 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:44:19,398 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:20,398 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:21,395 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:22,399 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:23,399 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:23,927 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:44:24,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:44:24,164 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:44:24,164 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:44:24,166 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:44:24,337 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:44:24,581 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:25,338 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:44:25,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:44:25,385 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:44:25,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:26,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:27,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:28,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:29,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:29,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:44:30,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:31,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:32,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:33,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:34,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:35,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:35,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:44:36,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:37,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:38,009 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:44:38,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:39,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:40,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:44:40,385 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:44:40,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:40,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:44:41,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:42,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:43,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:44,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:45,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:45,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:44:46,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:47,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:48,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:49,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:50,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:51,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:51,927 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:44:52,586 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:53,594 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:54,594 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:55,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:44:55,387 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:44:55,593 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:56,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:44:56,254 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:44:56,254 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:44:56,256 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:44:56,370 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:44:56,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:57,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:44:57,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:58,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:44:59,373 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:44:59,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:00,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:01,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:02,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:02,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:45:03,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:04,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:05,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:06,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:07,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:08,013 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:45:08,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:45:08,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:09,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:10,386 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:45:10,387 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:45:10,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:11,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:12,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:13,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:13,925 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:45:14,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:15,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:16,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:17,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:18,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:19,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:19,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:45:20,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:21,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:22,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:23,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:24,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:25,334 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:45:25,336 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:45:25,336 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:45:25,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:45:25,338 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:45:25,403 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:45:25,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:45:25,751 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:45:25,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:26,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:27,405 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:45:27,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:28,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:29,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:30,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:30,927 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:45:31,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:32,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:33,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:34,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:35,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:36,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:36,927 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:45:37,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:38,017 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:45:39,108 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:40,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:40,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:45:40,481 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:45:41,012 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:41,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:45:42,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:43,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:44,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:45,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:46,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:47,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:47,928 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:45:48,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:49,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:50,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:51,023 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:52,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:52,948 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:45:53,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:54,026 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:55,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:55,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:45:55,486 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:45:56,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:56,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:45:56,732 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:45:56,732 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:45:56,734 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:45:57,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:57,440 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:45:58,129 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:58,772 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:45:59,129 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:45:59,441 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:46:00,129 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:01,129 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:02,129 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:03,129 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:03,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:46:04,129 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:05,129 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:06,129 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:07,129 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:08,021 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:46:08,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:09,023 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:46:09,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:10,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:10,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:46:10,486 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:46:11,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:12,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:13,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:14,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:14,932 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:46:15,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:16,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:17,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:18,131 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:19,131 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:20,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:20,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:46:21,131 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:22,131 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:23,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:24,131 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:25,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:25,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:46:25,492 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:46:26,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:46:26,054 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:46:26,055 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:46:26,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:46:26,056 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:46:26,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:26,468 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:46:27,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:27,469 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:46:28,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:29,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:30,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:31,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:31,935 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:46:32,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:33,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:34,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:35,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:36,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:37,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:37,935 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:46:38,025 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:46:38,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:39,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:40,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:40,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:46:40,489 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:46:41,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:42,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:42,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:46:43,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:44,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:45,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:46,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:47,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:48,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:48,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:46:49,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:50,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:51,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:52,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:53,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:54,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:54,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:46:55,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:55,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:46:55,492 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:46:56,207 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:57,208 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:58,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:59,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:46:59,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:47:00,214 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:00,381 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:47:00,382 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:47:00,382 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:47:00,384 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:47:00,506 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:47:01,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:02,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:03,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:03,508 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:47:04,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:05,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:05,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:47:06,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:07,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:08,029 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:47:08,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:09,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:10,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:10,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:47:10,492 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:47:11,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:11,936 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:47:12,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:13,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:14,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:15,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:16,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:16,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:47:17,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:18,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:19,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:20,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:21,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:22,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:22,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:47:23,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:24,222 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:25,223 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:25,497 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:47:25,498 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:47:26,223 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:27,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:28,223 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:28,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:47:28,884 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:47:28,885 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:47:28,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:47:28,886 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:47:29,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:29,532 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:47:30,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:31,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:31,534 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:47:32,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:33,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:33,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:47:34,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:35,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:36,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:37,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:38,033 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:47:38,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:39,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:47:39,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:40,447 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:40,497 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:47:40,498 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:47:41,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:42,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:43,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:44,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:44,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:47:45,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:46,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:47,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:48,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:49,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:49,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:47:50,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:51,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:52,357 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:53,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:54,357 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:54,949 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:47:55,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:55,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:47:55,503 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:47:56,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:57,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:57,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:47:57,826 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:47:57,826 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:47:57,827 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:47:58,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:58,560 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:47:59,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:47:59,562 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:48:00,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:00,866 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:48:01,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:02,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:03,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:04,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:05,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:05,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:48:06,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:07,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:08,037 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:48:08,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:09,416 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:10,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:10,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:48:10,503 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:48:10,940 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:48:11,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:12,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:13,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:14,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:15,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:15,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:48:16,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:17,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:18,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:19,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:20,422 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:20,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:48:21,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:22,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:23,421 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:24,422 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:25,422 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:25,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:48:25,505 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:48:25,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:48:26,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:48:26,254 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:48:26,255 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:48:26,256 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:48:26,585 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:48:26,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:27,585 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:48:27,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:28,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:29,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:30,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:30,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:48:31,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:32,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:33,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:34,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:35,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:36,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:36,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:48:37,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:38,041 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:48:38,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:39,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:40,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:48:40,505 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:48:40,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:41,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:42,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:42,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:48:43,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:44,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:45,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:46,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:47,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:47,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:48:48,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:49,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:50,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:51,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:52,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:52,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:48:53,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:54,680 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:55,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:48:55,510 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:48:55,683 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:56,683 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:57,684 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:57,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:48:58,597 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:48:58,599 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:48:58,599 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:48:58,601 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:48:58,615 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:48:59,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:48:59,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:00,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:01,617 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:49:01,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:02,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:03,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:03,949 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:49:04,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:05,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:06,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:07,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:08,045 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:49:08,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:09,047 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:49:09,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:10,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:49:10,507 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:49:10,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:11,745 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:12,745 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:13,745 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:14,083 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:49:14,745 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:15,745 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:16,745 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:17,745 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:18,745 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:19,745 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:19,994 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:49:20,745 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:21,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:22,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:23,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:24,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:25,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:49:25,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:49:25,507 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:49:25,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:26,752 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:27,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:28,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:29,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:49:29,203 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:49:29,203 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:49:29,204 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:49:29,643 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:49:29,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:30,242 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:49:30,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:31,645 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:49:31,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:32,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:33,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:34,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:35,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:35,996 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:49:36,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:37,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:38,049 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:49:38,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:39,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:40,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:49:40,507 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:49:40,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:41,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:41,996 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:49:42,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:43,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:44,755 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:45,755 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:46,755 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:46,997 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:49:47,755 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:48,755 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:49,755 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:50,755 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:51,755 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:52,755 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:52,997 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:49:53,755 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:54,755 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:55,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:49:55,509 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:49:55,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:56,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:57,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:58,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:49:58,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:49:59,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:00,561 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:50:00,562 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:50:00,562 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:50:00,563 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:50:00,676 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:50:00,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:01,677 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:50:01,974 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:02,975 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:03,975 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:03,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:50:04,975 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:05,975 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:06,975 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:07,975 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:08,053 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:50:08,975 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:09,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:50:09,975 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:10,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:50:10,509 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:50:10,975 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:12,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:13,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:14,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:15,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:50:15,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:16,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:17,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:18,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:19,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:20,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:50:20,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:21,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:22,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:23,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:24,443 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:25,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:50:25,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:25,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:50:25,513 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:50:26,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:27,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:28,438 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:29,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:30,012 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:50:30,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:50:30,066 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:50:30,066 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:50:30,068 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:50:30,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:30,710 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:50:31,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:31,711 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:50:32,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:33,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:34,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:35,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:36,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:50:36,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:37,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:38,057 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:50:38,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:39,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:40,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:40,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:50:40,513 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:50:41,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:42,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:50:42,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:43,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:44,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:45,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:46,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:47,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:50:47,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:48,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:49,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:50,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:51,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:52,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:53,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:50:53,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:54,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:55,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:55,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:50:55,518 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:50:56,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:57,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:58,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:50:58,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:59,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:50:59,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:50:59,556 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:50:59,556 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:50:59,558 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:50:59,743 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:51:00,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:01,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:01,744 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:51:02,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:03,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:04,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:51:04,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:05,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:06,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:07,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:08,061 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:51:08,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:09,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:51:09,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:10,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:10,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:51:10,518 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:51:11,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:12,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:13,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:14,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:15,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:51:15,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:16,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:17,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:18,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:19,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:20,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:51:20,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:21,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:22,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:23,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:24,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:25,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:25,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:51:25,519 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:51:25,624 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:51:26,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:27,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:28,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:29,496 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:30,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:31,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:51:31,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:31,994 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:51:31,995 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:51:31,996 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:51:31,997 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:51:32,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:32,777 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:51:33,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:33,778 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:51:34,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:35,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:36,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:51:36,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:37,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:38,065 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:51:38,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:39,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:40,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:40,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:51:40,519 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:51:41,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:42,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:51:42,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:43,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:44,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:45,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:46,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:47,012 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:51:47,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:48,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:49,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:50,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:51,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:52,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:53,012 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:51:53,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:54,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:55,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:55,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:51:55,529 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:51:56,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:57,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:58,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:51:58,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:51:59,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:00,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:00,887 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:52:00,888 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:52:00,888 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:52:00,890 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:52:01,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:01,809 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:52:02,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:03,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:52:03,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:03,811 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:52:04,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:05,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:06,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:07,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:08,069 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:52:08,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:09,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:52:09,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:10,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:10,521 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:52:10,522 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:52:11,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:12,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:13,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:14,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:52:14,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:15,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:16,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:17,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:18,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:19,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:52:19,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:20,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:21,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:22,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:23,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:24,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:25,308 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:52:25,523 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:25,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:52:25,531 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:52:26,521 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:27,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:28,525 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:29,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:30,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:52:30,524 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:31,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:52:31,062 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:52:31,062 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:52:31,063 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:52:31,521 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:31,842 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:52:32,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:33,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:33,844 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:52:34,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:35,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:52:35,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:36,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:37,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:38,073 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:52:38,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:39,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:40,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:40,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:52:40,528 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:52:40,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:52:41,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:42,523 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:43,523 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:44,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:45,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:46,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:52:46,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:47,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:48,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:49,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:50,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:51,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:52,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:52:52,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:53,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:54,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:55,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:52:55,528 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:52:55,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:56,574 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:57,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:52:57,574 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:58,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:52:59,574 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:00,576 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:01,576 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:01,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:53:01,902 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:53:01,902 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:53:01,903 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:53:02,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:02,875 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:53:02,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:53:03,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:03,876 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:53:04,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:05,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:06,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:07,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:08,078 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:53:08,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:53:08,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:09,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:10,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:53:10,528 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:53:10,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:11,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:12,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:13,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:53:13,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:14,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:15,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:16,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:17,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:18,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:19,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:53:19,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:20,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:21,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:22,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:23,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:24,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:25,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:53:25,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:53:25,537 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:53:25,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:26,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:27,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:28,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:29,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:30,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:53:30,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:31,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:53:31,001 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:53:31,001 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:53:31,002 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:53:31,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:31,915 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:53:32,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:33,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:33,916 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:53:34,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:35,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:36,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:53:36,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:37,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:38,081 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:53:38,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:39,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:40,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:53:40,534 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:53:40,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:41,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:42,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:53:42,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:43,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:44,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:45,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:46,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:47,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:53:47,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:48,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:49,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:50,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:51,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:52,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:53,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:53:53,731 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:54,732 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:55,540 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:53:55,540 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:53:55,735 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:56,732 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:57,732 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:58,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:53:58,433 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:53:58,433 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:53:58,435 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:53:58,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:58,941 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:53:59,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:53:59,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:53:59,942 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:54:00,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:01,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:02,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:03,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:04,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:54:04,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:05,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:06,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:07,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:08,085 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:54:08,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:09,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:10,087 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:54:10,539 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:54:10,540 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:54:10,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:12,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:13,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:14,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:15,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:16,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:16,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:54:17,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:18,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:19,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:20,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:21,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:21,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:54:22,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:23,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:24,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:25,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:25,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:54:25,552 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:54:26,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:26,581 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:54:27,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:28,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:29,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:29,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:54:29,460 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:54:29,460 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:54:29,462 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:54:29,977 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:54:30,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:31,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:31,582 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:54:31,978 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:54:32,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:33,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:34,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:35,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:36,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:37,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:37,582 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:54:38,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:38,088 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:54:39,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:40,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:40,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:54:40,546 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:54:41,242 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:42,112 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:42,582 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:54:43,112 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:44,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:45,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:46,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:47,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:48,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:48,582 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:54:49,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:50,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:51,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:52,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:53,124 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:54,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:54,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:54:55,119 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:55,551 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:54:55,551 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:54:56,119 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:57,119 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:58,120 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:58,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:54:58,660 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:54:58,661 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:54:58,662 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:54:59,005 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:54:59,120 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:54:59,700 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:55:00,007 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:55:00,120 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:01,120 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:02,120 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:03,120 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:04,120 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:05,120 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:05,585 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:55:06,120 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:07,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:08,092 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:55:08,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:09,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:10,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:10,550 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:55:10,550 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:55:10,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:55:11,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:12,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:13,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:14,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:15,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:16,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:16,585 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:55:17,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:18,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:19,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:20,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:21,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:21,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:55:22,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:23,191 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:24,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:25,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:25,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:55:25,557 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:55:26,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:26,427 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:55:26,428 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:55:26,428 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:55:26,429 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:55:26,812 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:55:27,035 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:55:27,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:28,036 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:55:28,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:29,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:30,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:31,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:32,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:32,811 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:55:33,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:34,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:35,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:36,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:37,191 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:38,097 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:55:38,103 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:55:38,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:39,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:40,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:40,550 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:55:40,550 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:55:41,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:42,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:43,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:43,811 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:55:44,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:45,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:46,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:47,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:48,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:48,814 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:55:49,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:50,222 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:51,226 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:52,223 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:53,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:53,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:55:54,226 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:55,179 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:55:55,180 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:55:55,181 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:55:55,182 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:55:55,594 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:55,595 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:55:55,596 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:55:56,068 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:55:56,327 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:57,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:58,069 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:55:58,327 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:59,327 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:55:59,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:56:00,327 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:01,327 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:02,327 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:03,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:04,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:05,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:05,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:56:06,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:07,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:08,100 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:56:08,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:09,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:10,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:10,592 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:56:10,593 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:56:11,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:11,752 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:56:12,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:13,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:14,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:15,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:16,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:16,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:56:17,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:18,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:19,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:20,336 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:21,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:21,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:56:22,345 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:23,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:24,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:25,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:25,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:56:25,581 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:56:25,581 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:56:25,583 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:56:25,994 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:56:25,994 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:56:26,092 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:56:26,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:26,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:56:27,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:28,094 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:56:28,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:29,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:30,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:31,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:31,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:56:32,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:33,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:34,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:35,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:36,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:36,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:56:37,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:38,104 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:56:38,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:39,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:40,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:40,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:56:40,728 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:56:41,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:42,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:42,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:56:43,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:44,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:45,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:46,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:47,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:48,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:48,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:56:49,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:50,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:51,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:52,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:53,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:53,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:56:54,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:55,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:55,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:56:55,729 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:56:56,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:56,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:56:56,436 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:56:56,436 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:56:56,437 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:56:57,134 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:56:57,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:58,135 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:56:58,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:59,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:56:59,840 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:57:00,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:01,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:02,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:03,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:04,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:05,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:05,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:57:06,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:07,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:08,108 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:57:08,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:09,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:10,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:10,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:57:10,729 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:57:10,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:57:11,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:12,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:13,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:14,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:15,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:16,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:16,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:57:17,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:18,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:19,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:20,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:21,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:22,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:22,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:57:23,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:24,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:25,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:25,735 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:57:25,741 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:57:26,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:27,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:27,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:57:28,445 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:29,443 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:29,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:57:29,834 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:57:29,834 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:57:29,836 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:57:30,169 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:57:30,443 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:31,444 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:32,171 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:57:32,444 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:32,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:57:33,444 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:34,444 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:35,444 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:36,444 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:37,444 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:38,112 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:57:38,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:57:38,444 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:39,444 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:40,444 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:40,735 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:57:40,736 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:57:41,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:42,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:43,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:43,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:57:44,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:45,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:46,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:47,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:48,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:49,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:49,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:57:50,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:51,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:52,452 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:53,452 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:54,461 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:54,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:57:55,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:55,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:57:55,744 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:57:56,460 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:57,460 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:57,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:57:57,849 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:57:57,850 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:57:57,851 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:57:58,205 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:57:58,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:59,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:57:59,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:58:00,206 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:58:00,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:01,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:02,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:03,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:04,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:05,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:05,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:58:06,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:07,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:08,116 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:58:08,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:09,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:10,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:10,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:58:10,744 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:58:10,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:58:11,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:12,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:13,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:14,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:15,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:16,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:16,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:58:17,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:18,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:19,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:20,460 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:21,460 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:22,461 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:22,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:58:23,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:24,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:25,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:25,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:58:25,751 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:58:26,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:26,629 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:58:26,631 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:58:26,631 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:58:26,632 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:58:27,245 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:58:27,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:28,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:58:28,246 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:58:28,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:29,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:30,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:31,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:32,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:33,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:34,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:58:34,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:35,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:36,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:37,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:38,121 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:58:38,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:39,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:58:39,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:40,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:40,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:58:40,747 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:58:41,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:42,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:43,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:44,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:45,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:58:45,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:46,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:47,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:48,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:49,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:50,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:51,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:58:51,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:52,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:53,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:54,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:55,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:55,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:58:55,755 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:58:56,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:56,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:58:57,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:58,478 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:58:59,415 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:58:59,416 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:58:59,416 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:58:59,418 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:58:59,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:00,277 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:59:00,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:01,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:02,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:59:02,279 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:59:02,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:03,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:04,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:05,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:06,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:07,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:08,013 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:59:08,123 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:59:08,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:09,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:10,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:59:10,751 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:59:10,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:11,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:12,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:13,015 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:59:13,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:14,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:15,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:16,828 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:17,828 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:18,828 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:19,014 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:59:19,828 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:20,828 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:21,828 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:22,828 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:23,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:24,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:59:24,835 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:25,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:59:25,754 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:59:25,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:26,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:27,835 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:28,835 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:29,021 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:59:29,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:59:29,220 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:59:29,221 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:59:29,224 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:59:29,308 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:59:29,836 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:30,309 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 00:59:30,836 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:31,836 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:32,836 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:33,836 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:34,836 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:35,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:59:35,836 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:36,836 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:37,836 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:38,128 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 00:59:38,836 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:39,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:40,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:59:40,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:59:40,751 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:59:40,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:41,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:42,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:43,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:44,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:45,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:46,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:59:46,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:47,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:48,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:49,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:50,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:51,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:52,024 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:59:52,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:53,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:54,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:55,757 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 00:59:55,757 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 00:59:55,842 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:56,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:57,030 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 00:59:57,588 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 00:59:57,588 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 00:59:57,588 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 00:59:57,590 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 00:59:58,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:58,354 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 00:59:58,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 00:59:59,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:00,356 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:00:00,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:01,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:02,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:03,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:00:03,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:04,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:05,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:06,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:08,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:08,132 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:00:08,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:00:09,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:10,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:10,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:00:10,755 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:00:11,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:12,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:13,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:14,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:14,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:00:15,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:16,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:17,000 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:18,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:19,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:20,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:20,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:00:21,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:22,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:23,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:24,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:25,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:25,042 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:00:25,764 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:00:25,764 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:00:26,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:27,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:28,014 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:29,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:29,593 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:00:29,593 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:00:29,594 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:00:29,595 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:00:30,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:30,395 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:00:30,634 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:00:31,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:32,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:32,397 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:00:33,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:34,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:35,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:36,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:36,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:00:37,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:38,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:38,136 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:00:39,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:40,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:40,757 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:00:40,758 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:00:41,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:41,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:00:42,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:43,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:44,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:45,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:46,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:47,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:47,031 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:00:48,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:49,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:50,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:51,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:52,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:52,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:00:53,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:54,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:55,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:55,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:00:55,758 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:00:56,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:57,012 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:58,012 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:00:58,031 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:00:59,012 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:00,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:01,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:02,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:03,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:03,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:01:04,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:05,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:05,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:01:05,485 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:01:05,485 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:01:05,487 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:01:06,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:06,428 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:01:06,429 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:01:07,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:08,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:08,140 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:01:08,147 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:01:09,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:10,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:10,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:01:10,758 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:01:11,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:12,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:13,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:14,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:14,034 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:01:15,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:16,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:17,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:18,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:19,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:20,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:20,034 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:01:21,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:22,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:23,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:24,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:25,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:25,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:01:25,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:01:25,758 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:01:26,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:27,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:28,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:29,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:30,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:31,024 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:31,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:01:32,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:33,023 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:34,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:35,024 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:35,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:01:35,955 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:01:35,956 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:01:35,957 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:01:36,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:36,462 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:01:36,995 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:01:37,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:38,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:38,144 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:01:38,464 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:01:39,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:40,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:40,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:01:40,759 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:01:41,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:42,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:01:42,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:43,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:44,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:45,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:46,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:47,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:01:47,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:48,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:49,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:50,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:51,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:52,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:01:52,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:53,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:54,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:55,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:55,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:01:55,759 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:01:56,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:57,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:58,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:01:58,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:01:59,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:00,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:01,103 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:02,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:03,042 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:02:03,105 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:04,116 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:05,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:06,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:07,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:07,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:02:07,172 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:02:07,172 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:02:07,174 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:02:07,495 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:02:08,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:08,148 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:02:08,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:02:08,496 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:02:09,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:10,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:10,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:02:10,759 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:02:11,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:12,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:13,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:14,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:02:14,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:15,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:16,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:17,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:18,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:19,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:02:19,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:20,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:21,112 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:22,112 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:23,112 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:24,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:02:24,112 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:25,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:25,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:02:25,759 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:02:26,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:27,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:28,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:29,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:30,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:02:30,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:31,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:32,116 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:33,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:34,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:35,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:02:35,124 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:36,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:37,124 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:37,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:02:37,924 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:02:37,924 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:02:37,925 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:02:38,152 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:02:38,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:38,532 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:02:39,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:40,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:02:40,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:40,533 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:02:40,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:02:40,759 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:02:41,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:42,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:43,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:44,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:45,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:46,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:02:46,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:47,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:48,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:49,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:50,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:51,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:52,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:02:52,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:53,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:54,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:55,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:55,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:02:55,759 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:02:56,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:57,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:58,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:02:58,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:02:59,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:00,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:01,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:02,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:03,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:03:03,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:04,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:05,345 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:06,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:07,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:08,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:03:08,168 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:03:08,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:09,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:09,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:03:09,530 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:03:09,530 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:03:09,532 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:03:09,562 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:03:10,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:10,563 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:03:10,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:03:10,759 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:03:11,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:12,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:13,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:03:13,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:14,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:15,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:16,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:17,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:18,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:19,044 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:03:19,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:20,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:21,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:22,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:23,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:24,044 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:03:24,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:25,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:25,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:03:25,759 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:03:26,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:27,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:28,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:29,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:30,044 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:03:30,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:31,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:32,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:33,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:34,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:35,044 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:03:35,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:36,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:37,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:38,170 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:03:38,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:39,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:40,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:03:40,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:40,760 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:03:40,766 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:03:41,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:42,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:03:42,270 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:03:42,271 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:03:42,276 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:03:42,594 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:03:43,178 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:43,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:44,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:44,596 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:03:45,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:03:45,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:46,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:47,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:48,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:49,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:50,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:51,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:03:51,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:52,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:53,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:54,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:55,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:55,760 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:03:55,761 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:03:56,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:56,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:03:57,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:58,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:03:59,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:00,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:01,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:02,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:04:02,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:03,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:04,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:05,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:06,419 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:07,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:04:07,419 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:08,167 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:04:08,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:09,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:10,428 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:10,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:04:10,769 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:04:11,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:12,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:04:12,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:13,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:04:13,415 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:04:13,416 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:04:13,419 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:04:13,631 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:04:13,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:14,632 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:04:14,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:15,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:16,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:17,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:18,058 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:04:18,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:19,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:20,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:21,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:22,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:23,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:24,058 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:04:24,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:25,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:04:25,763 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:04:25,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:26,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:27,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:28,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:29,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:04:29,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:30,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:31,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:32,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:33,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:34,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:04:34,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:35,828 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:36,828 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:37,828 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:38,170 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:04:38,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:39,177 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:04:39,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:40,772 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:04:40,772 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:04:40,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:41,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:42,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:43,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:44,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:04:44,840 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:04:44,840 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:04:44,841 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:04:44,843 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:04:45,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:45,669 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:04:46,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:46,670 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:04:47,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:48,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:49,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:50,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:04:50,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:51,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:52,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:53,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:54,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:55,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:04:55,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:55,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:04:55,767 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:04:56,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:57,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:58,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:04:59,249 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:00,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:01,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:05:01,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:02,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:03,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:04,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:05,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:06,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:07,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:05:07,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:08,174 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:05:08,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:09,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:10,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:10,773 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:05:10,776 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:05:11,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:12,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:05:12,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:13,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:14,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:15,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:15,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:05:15,355 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:05:15,356 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:05:15,360 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:05:15,696 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:05:16,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:16,697 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:05:17,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:18,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:05:18,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:19,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:20,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:21,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:22,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:23,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:24,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:05:24,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:25,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:25,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:05:25,770 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:05:26,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:27,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:28,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:29,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:30,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:05:30,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:31,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:32,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:33,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:34,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:35,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:05:35,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:36,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:37,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:38,178 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:05:38,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:39,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:40,180 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:05:40,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:40,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:05:40,770 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:05:41,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:42,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:43,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:44,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:45,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:46,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:05:46,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:47,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:48,017 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:05:48,019 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:05:48,019 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:05:48,025 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:05:48,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:48,735 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:05:48,735 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:05:49,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:50,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:51,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:52,058 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:05:52,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:53,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:54,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:55,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:55,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:05:55,770 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:05:56,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:57,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:05:57,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:58,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:05:59,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:00,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:01,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:02,073 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:06:02,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:03,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:04,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:05,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:06,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:07,073 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:06:07,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:08,182 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:06:08,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:09,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:10,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:10,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:06:10,770 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:06:11,648 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:12,082 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:06:12,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:13,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:14,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:15,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:16,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:17,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:06:17,258 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:06:17,258 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:06:17,259 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:06:17,260 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:06:17,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:17,774 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:06:18,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:18,775 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:06:19,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:20,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:21,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:22,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:23,077 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:06:23,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:24,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:25,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:25,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:06:25,771 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:06:26,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:27,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:28,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:29,076 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:06:29,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:30,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:31,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:32,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:33,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:34,077 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:06:34,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:35,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:36,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:37,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:38,187 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:06:38,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:39,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:06:39,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:40,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:40,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:06:40,771 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:06:41,752 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:42,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:43,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:44,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:45,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:06:45,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:46,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:47,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:06:47,293 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:06:47,294 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:06:47,295 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:06:47,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:47,801 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:06:48,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:48,802 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:06:49,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:50,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:06:50,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:51,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:52,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:53,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:54,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:55,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:55,770 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:06:55,771 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:06:55,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:06:56,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:57,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:58,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:06:59,800 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:00,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:01,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:07:01,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:02,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:03,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:04,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:05,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:06,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:07,077 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:07:07,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:08,190 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:07:08,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:09,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:10,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:07:10,784 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:07:10,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:11,801 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:12,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:07:12,801 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:13,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:14,808 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:15,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:16,248 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:07:16,249 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:07:16,250 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:07:16,251 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:07:16,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:16,826 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:07:16,827 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:07:17,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:07:17,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:18,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:19,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:20,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:21,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:22,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:23,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:07:23,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:24,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:25,773 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:07:25,774 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:07:25,814 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:26,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:27,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:28,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:29,082 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:07:29,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:30,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:31,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:32,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:33,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:34,082 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:07:34,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:35,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:36,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:37,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:38,194 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:07:38,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:39,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:07:39,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:40,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:07:40,779 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:07:40,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:41,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:42,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:43,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:44,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:45,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:07:45,396 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:07:45,397 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:07:45,398 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:07:45,399 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:07:45,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:45,857 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:07:46,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:46,859 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:07:47,857 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:48,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:49,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:50,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:51,087 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:07:51,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:52,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:53,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:54,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:55,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:07:55,777 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:07:55,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:56,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:07:57,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:58,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:07:59,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:00,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:01,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:02,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:02,087 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:08:03,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:04,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:05,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:06,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:07,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:08,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:08,087 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:08:08,199 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:08:09,076 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:10,073 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:10,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:08:10,783 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:08:11,073 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:12,077 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:13,073 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:13,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:08:14,073 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:14,757 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:08:14,758 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:08:14,759 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:08:14,760 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:08:14,889 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:08:15,173 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:16,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:16,892 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:08:17,175 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:18,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:08:18,175 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:19,175 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:20,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:21,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:22,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:23,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:24,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:08:24,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:25,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:25,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:08:25,777 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:08:26,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:27,176 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:28,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:29,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:08:29,251 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:30,251 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:31,251 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:32,251 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:33,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:34,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:35,091 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:08:35,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:36,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:37,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:38,203 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:08:38,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:39,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:40,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:08:40,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:40,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:08:40,787 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:08:41,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:42,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:43,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:44,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:45,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:46,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:08:46,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:46,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:08:46,749 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:08:46,749 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:08:46,750 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:08:46,934 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:08:46,935 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:08:47,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:48,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:49,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:50,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:51,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:08:51,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:52,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:53,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:54,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:55,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:55,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:08:55,777 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:08:56,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:56,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:08:57,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:58,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:08:59,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:00,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:01,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:02,161 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:09:02,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:03,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:04,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:05,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:06,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:07,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:09:07,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:08,207 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:09:08,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:09,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:10,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:10,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:09:10,802 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:09:11,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:12,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:09:12,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:13,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:14,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:15,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:16,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:16,567 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:09:16,568 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:09:16,568 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:09:16,570 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:09:16,969 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:09:16,970 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:09:17,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:18,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:09:18,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:19,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:20,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:21,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:22,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:23,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:24,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:09:24,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:25,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:25,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:09:25,783 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:09:26,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:27,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:28,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:29,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:30,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:09:30,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:31,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:32,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:33,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:34,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:35,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:09:35,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:36,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:37,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:38,211 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:09:38,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:39,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:40,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:09:40,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:40,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:09:40,783 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:09:41,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:42,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:43,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:44,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:45,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:46,178 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:09:46,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:47,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:47,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:09:47,750 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:09:47,751 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:09:47,752 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:09:48,001 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:09:48,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:49,003 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:09:49,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:50,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:51,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:51,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:09:52,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:53,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:54,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:55,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:55,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:09:55,783 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:09:56,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:56,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:09:57,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:58,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:09:59,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:00,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:01,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:02,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:10:02,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:03,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:04,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:05,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:06,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:07,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:10:07,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:08,215 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:10:08,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:09,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:10,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:10,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:10:10,784 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:10:11,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:12,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:13,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:10:13,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:14,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:15,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:16,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:17,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:18,177 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:10:18,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:19,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:20,299 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:21,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:21,328 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:10:21,329 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:10:21,330 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:10:21,331 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:10:22,037 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:10:22,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:23,038 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:10:23,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:24,174 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:10:24,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:25,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:25,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:10:25,784 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:10:26,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:27,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:28,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:29,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:30,174 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:10:30,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:31,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:32,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:33,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:34,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:35,174 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:10:35,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:36,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:37,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:38,219 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:10:38,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:39,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:40,221 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:10:40,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:40,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:10:40,784 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:10:41,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:42,307 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:43,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:44,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:45,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:46,193 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:10:46,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:47,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:47,794 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:10:47,795 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:10:47,795 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:10:47,796 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:10:48,074 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:10:48,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:49,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:50,076 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:10:50,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:51,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:51,835 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:10:52,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:53,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:54,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:55,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:55,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:10:55,784 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:10:56,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:56,921 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:10:57,313 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:58,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:10:59,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:00,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:01,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:02,177 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:11:02,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:03,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:04,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:05,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:06,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:07,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:08,177 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:11:08,223 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:11:08,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:09,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:10,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:10,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:11:10,784 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:11:11,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:12,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:13,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:11:13,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:14,319 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:15,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:16,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:17,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:18,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:18,362 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:11:18,363 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:11:18,364 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:11:18,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:11:18,366 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:11:19,107 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:11:19,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:20,108 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:11:20,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:21,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:22,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:23,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:24,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:11:24,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:25,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:25,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:11:25,784 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:11:26,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:27,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:28,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:29,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:30,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:11:30,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:31,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:32,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:33,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:34,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:35,181 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:11:35,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:36,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:37,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:38,227 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:11:38,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:39,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:40,230 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:11:40,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:40,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:11:40,784 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:11:41,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:42,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:43,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:44,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:45,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:46,208 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:11:46,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:47,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:47,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:11:47,844 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:11:47,844 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:11:47,846 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:11:48,138 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:11:48,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:49,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:50,140 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:11:50,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:51,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:51,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:11:52,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:53,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:54,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:55,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:55,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:11:55,784 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:11:56,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:56,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:11:57,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:58,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:11:59,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:00,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:01,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:02,185 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:12:02,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:03,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:04,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:05,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:06,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:07,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:08,185 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:12:08,232 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:12:08,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:09,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:10,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:10,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:12:10,784 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:12:11,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:12,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:13,199 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:12:13,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:14,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:15,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:16,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:17,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:18,086 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:12:18,087 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:12:18,087 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:12:18,088 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:12:18,169 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:12:18,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:19,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:12:19,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:20,170 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:12:20,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:21,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:22,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:23,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:24,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:12:24,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:25,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:25,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:12:25,784 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:12:26,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:27,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:28,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:29,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:30,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:12:30,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:31,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:32,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:33,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:34,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:35,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:12:35,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:36,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:37,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:38,235 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:12:38,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:39,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:40,238 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:12:40,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:40,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:12:40,785 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:12:41,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:42,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:43,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:44,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:45,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:46,193 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:12:46,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:47,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:48,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:49,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:50,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:51,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:12:51,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:51,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:12:51,766 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:12:51,767 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:12:51,768 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:12:52,202 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:12:52,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:53,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:54,204 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:12:54,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:55,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:55,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:12:55,784 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:12:56,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:56,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:12:57,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:58,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:12:59,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:00,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:01,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:02,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:13:02,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:03,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:04,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:05,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:06,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:07,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:13:07,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:08,239 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:13:08,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:09,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:10,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:10,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:13:10,785 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:13:11,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:12,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:13,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:13:13,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:14,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:15,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:16,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:17,525 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:18,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:13:18,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:19,524 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:20,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:21,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:22,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:22,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:13:22,558 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:13:22,558 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:13:22,560 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:13:23,234 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:13:23,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:23,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:13:24,235 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:13:24,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:25,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:25,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:13:25,785 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:13:26,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:27,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:28,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:29,191 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:13:29,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:30,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:31,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:32,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:33,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:34,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:35,191 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:13:35,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:36,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:37,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:38,243 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:13:38,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:39,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:40,246 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:13:40,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:40,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:13:40,785 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:13:41,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:42,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:43,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:44,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:45,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:46,191 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:13:46,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:47,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:48,531 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:49,531 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:50,531 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:51,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:13:51,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:52,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:53,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:13:53,468 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:13:53,468 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:13:53,470 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:13:53,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:54,272 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:13:54,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:55,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:13:55,785 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:13:55,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:56,273 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:13:56,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:13:56,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:57,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:58,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:13:59,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:00,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:01,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:02,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:14:02,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:03,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:04,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:05,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:06,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:07,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:14:07,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:08,247 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:14:08,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:09,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:10,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:14:10,785 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:14:10,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:11,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:12,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:13,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:14:13,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:14,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:15,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:16,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:17,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:18,199 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:14:18,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:19,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:20,883 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:21,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:22,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:22,952 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:14:22,953 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:14:22,954 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:14:22,955 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:14:23,313 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:14:23,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:23,993 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:14:24,313 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:14:24,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:25,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:14:25,786 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:14:25,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:26,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:27,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:28,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:29,198 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:14:29,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:30,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:31,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:32,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:33,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:34,199 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:14:34,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:36,096 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:37,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:38,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:38,252 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:14:39,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:39,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:14:40,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:40,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:14:40,786 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:14:41,001 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:42,002 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:43,002 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:44,002 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:45,002 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:45,199 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:14:46,002 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:47,002 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:48,002 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:49,008 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:50,006 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:51,006 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:51,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:14:52,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:53,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:54,008 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:54,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:14:54,840 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:14:54,840 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:14:54,842 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:14:55,259 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:55,343 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:14:55,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:14:55,786 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:14:56,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:56,344 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:14:56,883 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:14:57,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:58,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:14:59,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:00,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:01,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:02,200 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:15:02,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:03,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:04,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:05,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:06,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:07,200 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:15:07,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:08,254 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:15:08,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:09,464 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:10,463 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:10,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:15:10,786 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:15:11,463 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:12,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:15:12,463 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:13,464 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:14,464 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:15,464 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:16,464 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:17,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:15:17,464 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:18,464 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:19,464 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:20,464 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:21,471 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:22,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:15:22,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:23,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:24,471 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:25,471 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:25,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:15:25,789 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:15:26,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:27,206 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:15:27,446 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:15:27,447 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:15:27,448 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:15:27,449 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:15:27,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:28,374 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:15:28,597 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:29,597 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:30,375 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:15:30,597 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:31,597 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:32,597 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:33,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:15:33,597 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:34,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:35,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:36,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:37,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:38,258 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:15:38,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:15:38,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:39,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:40,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:40,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:15:40,789 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:15:41,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:42,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:43,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:44,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:15:44,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:45,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:46,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:47,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:48,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:49,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:15:49,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:50,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:51,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:52,605 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:53,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:54,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:15:54,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:55,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:55,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:15:55,796 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:15:56,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:57,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:58,066 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:15:58,067 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:15:58,067 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:15:58,068 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:15:58,407 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:15:58,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:15:59,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:00,107 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:16:00,409 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:16:00,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:01,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:02,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:03,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:04,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:05,206 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:16:05,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:06,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:07,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:08,262 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:16:08,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:09,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:10,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:16:10,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:10,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:16:10,790 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:16:11,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:12,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:13,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:14,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:15,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:16,207 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:16:16,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:17,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:18,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:19,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:20,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:21,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:22,207 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:16:22,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:23,624 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:24,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:25,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:25,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:16:25,793 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:16:26,629 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:27,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:16:27,629 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:28,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:29,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:16:29,609 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:16:29,609 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:16:29,611 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:16:30,015 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:30,440 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:16:30,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:31,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:32,441 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:16:32,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:33,208 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:16:33,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:34,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:35,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:36,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:37,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:38,266 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:16:38,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:16:38,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:39,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:40,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:40,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:16:40,793 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:16:41,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:43,147 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:43,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:44,209 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:16:44,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:45,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:46,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:47,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:48,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:49,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:50,209 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:16:50,778 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:51,778 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:52,778 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:53,778 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:54,778 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:55,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:16:55,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:55,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:16:55,793 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:16:56,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:57,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:58,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:16:59,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:00,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:01,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:17:01,101 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:17:01,101 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:17:01,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:17:01,103 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:17:01,473 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:17:01,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:02,474 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:17:02,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:03,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:04,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:05,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:06,210 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:17:06,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:07,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:08,270 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:17:08,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:09,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:10,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:10,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:17:10,794 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:17:11,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:11,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:17:12,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:13,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:14,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:15,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:16,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:17,210 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:17:17,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:18,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:19,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:20,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:21,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:22,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:23,210 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:17:23,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:24,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:25,795 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:17:25,799 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:17:25,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:26,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:27,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:28,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:17:28,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:29,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:30,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:17:30,278 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:17:30,278 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:17:30,279 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:17:30,504 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:17:30,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:31,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:32,506 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:17:32,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:33,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:17:33,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:34,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:35,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:36,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:37,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:38,275 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:17:38,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:17:38,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:39,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:40,795 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:17:40,796 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:17:40,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:41,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:42,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:43,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:44,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:17:44,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:45,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:46,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:47,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:48,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:49,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:17:49,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:50,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:51,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:52,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:53,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:54,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:55,221 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:17:55,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:17:55,799 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:17:55,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:56,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:57,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:58,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:17:59,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:00,228 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:18:00,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:18:00,743 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:18:00,743 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:18:00,745 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:18:01,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:01,531 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:18:01,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:02,532 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:18:02,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:03,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:04,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:05,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:06,221 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:18:06,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:07,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:08,278 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:18:08,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:09,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:10,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:18:10,799 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:18:10,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:11,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:11,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:18:12,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:13,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:14,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:15,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:16,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:17,221 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:18:17,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:18,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:19,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:20,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:21,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:22,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:23,228 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:18:23,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:24,898 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:25,800 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:18:25,801 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:18:25,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:26,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:27,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:28,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:29,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:18:29,218 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:18:29,218 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:18:29,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:18:29,221 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:18:29,563 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:18:29,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:30,565 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:18:30,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:31,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:32,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:33,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:34,226 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:18:34,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:35,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:36,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:37,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:38,282 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:18:38,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:39,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:18:39,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:40,800 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:18:40,801 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:18:40,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:41,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:42,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:43,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:44,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:45,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:18:45,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:46,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:47,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:48,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:49,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:50,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:51,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:18:51,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:52,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:53,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:54,908 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:55,803 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:18:55,805 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:18:55,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:56,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:18:56,908 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:57,908 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:58,908 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:18:59,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:18:59,807 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:18:59,807 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:18:59,809 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:19:00,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:00,592 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:19:00,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:01,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:02,230 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:19:02,594 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:19:02,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:03,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:04,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:05,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:06,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:07,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:08,230 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:19:08,286 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:19:08,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:09,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:10,801 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:19:10,802 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:19:10,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:11,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:12,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:13,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:14,230 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:19:14,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:15,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:16,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:17,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:18,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:19,231 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:19:19,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:20,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:21,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:22,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:23,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:24,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:25,241 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:19:25,808 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:19:25,812 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:19:25,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:26,959 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:27,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:28,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:29,968 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:30,240 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:19:30,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:31,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:19:31,011 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:19:31,012 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:19:31,013 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:19:31,628 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:19:31,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:32,629 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:19:32,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:33,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:34,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:35,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:36,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:19:36,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:37,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:38,290 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:19:38,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:39,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:40,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:19:40,806 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:19:40,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:41,935 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:19:41,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:43,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:44,026 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:45,026 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:46,026 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:47,026 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:47,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:19:48,026 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:49,026 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:50,026 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:51,026 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:52,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:53,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:53,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:19:54,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:55,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:55,809 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:19:55,809 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:19:56,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:57,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:58,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:19:58,245 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:19:59,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:00,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:00,542 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:20:00,543 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:20:00,544 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:20:00,545 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:20:00,657 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:20:01,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:02,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:02,658 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:20:03,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:04,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:04,240 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:20:05,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:06,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:07,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:08,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:08,295 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:20:09,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:09,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:20:10,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:10,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:20:10,808 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:20:11,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:12,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:13,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:14,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:15,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:15,241 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:20:16,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:17,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:18,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:19,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:20,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:20,241 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:20:21,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:22,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:23,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:24,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:25,049 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:25,245 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:20:25,812 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:20:25,816 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:20:26,047 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:27,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:28,044 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:29,042 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:30,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:30,239 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:20:30,241 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:20:30,241 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:20:30,242 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:20:30,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:20:30,680 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:20:31,042 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:32,042 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:32,681 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:20:33,042 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:34,042 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:35,042 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:36,042 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:36,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:20:37,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:38,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:38,298 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:20:39,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:40,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:40,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:20:40,811 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:20:41,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:41,927 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:20:42,165 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:43,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:44,164 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:45,165 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:46,165 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:47,165 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:47,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:20:48,165 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:49,165 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:50,165 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:51,165 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:52,165 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:52,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:20:53,165 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:54,167 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:55,168 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:55,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:20:55,817 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:20:56,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:57,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:57,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:20:58,112 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:20:58,114 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:20:58,114 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:20:58,116 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:20:58,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:20:58,703 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:20:59,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:00,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:00,704 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:21:01,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:02,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:03,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:21:03,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:04,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:05,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:06,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:07,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:08,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:08,302 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:21:08,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:21:09,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:10,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:10,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:21:10,817 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:21:11,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:12,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:13,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:14,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:21:14,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:15,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:16,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:17,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:18,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:19,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:20,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:21:20,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:21,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:22,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:23,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:24,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:25,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:21:25,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:25,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:21:25,834 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:21:26,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:27,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:28,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:21:28,195 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:21:28,195 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:21:28,196 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:21:28,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:28,727 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:21:29,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:30,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:30,729 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:21:31,235 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:21:31,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:32,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:33,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:34,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:35,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:36,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:21:36,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:37,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:38,307 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:21:38,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:39,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:40,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:40,819 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:21:40,820 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:21:41,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:41,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:21:42,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:43,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:44,696 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:45,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:46,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:47,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:21:47,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:48,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:49,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:50,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:51,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:52,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:53,387 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:21:53,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:54,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:55,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:55,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:21:55,827 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:21:56,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:57,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:58,391 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:21:58,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:21:59,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:00,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:22:00,614 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:22:00,614 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:22:00,616 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:22:00,759 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:22:01,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:02,031 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:02,760 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:22:03,031 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:04,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:04,390 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:22:05,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:06,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:07,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:08,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:08,310 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:22:09,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:10,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:10,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:22:10,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:22:10,825 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:22:11,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:12,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:13,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:14,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:15,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:15,391 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:22:16,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:17,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:18,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:19,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:20,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:20,391 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:22:21,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:22,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:23,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:24,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:25,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:25,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:22:25,825 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:22:25,917 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:22:26,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:27,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:28,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:29,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:30,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:31,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:31,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:22:32,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:32,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:22:32,073 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:22:32,073 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:22:32,074 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:22:32,786 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:22:33,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:34,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:34,788 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:22:35,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:36,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:36,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:22:37,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:38,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:38,315 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:22:39,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:40,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:40,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:22:40,825 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:22:41,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:41,991 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:22:42,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:43,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:44,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:45,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:46,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:47,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:47,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:22:48,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:49,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:50,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:51,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:52,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:53,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:53,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:22:54,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:55,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:55,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:22:55,825 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:22:56,050 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:57,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:58,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:22:58,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:22:59,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:00,047 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:01,049 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:01,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:23:01,920 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:23:01,920 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:23:01,922 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:23:02,334 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:02,814 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:23:02,815 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:23:03,066 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:04,066 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:04,556 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:23:05,066 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:06,066 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:07,066 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:08,066 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:08,318 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:23:09,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:10,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:10,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:23:10,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:23:10,825 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:23:11,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:12,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:13,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:14,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:15,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:15,556 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:23:16,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:17,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:18,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:19,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:20,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:20,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:23:21,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:22,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:23,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:24,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:25,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:25,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:23:25,826 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:23:25,973 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:23:26,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:27,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:28,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:29,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:30,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:31,075 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:31,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:23:32,075 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:33,077 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:34,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:35,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:35,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:23:35,336 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:23:35,336 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:23:35,337 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:23:35,847 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:23:36,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:36,848 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:23:37,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:37,377 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:23:38,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:38,322 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:23:39,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:40,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:40,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:23:40,826 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:23:41,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:42,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:42,562 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:23:43,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:44,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:45,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:46,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:47,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:47,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:23:48,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:49,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:50,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:51,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:52,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:52,691 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:23:53,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:54,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:55,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:55,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:23:55,826 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:23:56,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:57,081 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:58,082 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:23:58,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:23:59,091 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:00,087 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:01,088 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:02,088 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:03,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:03,699 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:24:04,092 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:04,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:24:04,509 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:24:04,510 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:24:04,511 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:24:04,880 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:24:05,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:06,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:06,882 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:24:07,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:08,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:08,327 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:24:09,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:09,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:24:10,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:10,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:24:10,826 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:24:11,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:12,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:13,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:14,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:14,695 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:24:15,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:16,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:17,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:18,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:19,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:19,696 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:24:20,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:21,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:22,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:23,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:24,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:25,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:25,695 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:24:25,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:24:25,826 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:24:26,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:27,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:28,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:29,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:30,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:30,700 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:24:31,221 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:32,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:33,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:34,223 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:35,221 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:35,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:24:35,534 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:24:35,534 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:24:35,535 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:24:35,911 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:24:36,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:36,574 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:24:36,912 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:24:37,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:38,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:38,329 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:24:39,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:40,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:40,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:24:40,826 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:24:41,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:41,993 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:24:42,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:43,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:44,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:45,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:46,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:47,221 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:47,699 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:24:48,312 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:49,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:50,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:51,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:52,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:52,700 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:24:53,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:54,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:55,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:55,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:24:55,826 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:24:56,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:57,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:58,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:24:58,699 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:24:59,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:00,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:01,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:02,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:03,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:03,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:25:04,323 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:05,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:05,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:25:05,668 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:25:05,668 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:25:05,669 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:25:05,949 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:25:06,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:06,951 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:25:07,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:08,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:08,333 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:25:09,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:09,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:25:10,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:10,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:25:10,826 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:25:11,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:12,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:13,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:14,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:14,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:25:15,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:16,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:17,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:18,476 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:19,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:20,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:20,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:25:21,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:22,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:23,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:24,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:25,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:25,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:25:25,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:25:25,826 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:25:26,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:27,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:28,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:29,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:30,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:31,396 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:31,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:25:32,391 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:33,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:34,392 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:35,392 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:36,396 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:37,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:25:37,267 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:25:37,268 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:25:37,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:25:37,269 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:25:37,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:37,983 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:25:38,338 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:25:38,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:38,984 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:25:39,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:40,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:40,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:25:40,827 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:25:41,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:42,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:42,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:25:43,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:44,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:45,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:46,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:47,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:48,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:48,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:25:49,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:50,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:51,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:52,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:53,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:53,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:25:54,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:55,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:55,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:25:55,827 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:25:56,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:57,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:58,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:25:58,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:25:59,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:00,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:01,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:02,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:03,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:03,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:26:04,687 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:05,680 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:06,685 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:07,691 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:08,342 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:26:08,684 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:09,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:26:09,685 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:10,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:26:10,331 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:26:10,331 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:26:10,332 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:26:10,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:10,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:26:10,827 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:26:11,021 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:26:11,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:12,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:13,023 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:26:13,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:14,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:26:14,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:15,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:16,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:17,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:18,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:19,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:20,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:26:20,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:21,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:22,753 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:23,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:24,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:25,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:26:25,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:25,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:26:25,827 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:26:26,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:27,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:28,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:29,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:30,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:31,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:26:31,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:32,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:33,760 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:34,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:35,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:36,718 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:26:36,761 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:37,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:38,343 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:26:38,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:39,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:26:39,654 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:26:39,654 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:26:39,656 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:26:40,051 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:26:40,086 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:40,808 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:40,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:26:40,827 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:26:41,052 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:26:41,808 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:41,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:26:42,808 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:43,808 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:44,808 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:45,808 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:46,808 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:47,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:26:47,808 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:48,808 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:49,808 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:50,809 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:51,809 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:52,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:26:52,811 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:53,809 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:54,809 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:55,809 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:55,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:26:55,827 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:26:56,809 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:57,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:26:57,809 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:58,809 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:26:59,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:00,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:01,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:02,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:27:02,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:03,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:04,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:05,820 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:06,814 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:07,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:27:07,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:08,348 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:27:08,820 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:09,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:10,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:10,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:27:10,827 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:27:10,867 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:27:10,904 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:27:10,904 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:27:10,910 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:27:11,080 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:27:11,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:12,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:13,082 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:27:13,718 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:27:13,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:14,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:15,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:16,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:17,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:18,718 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:27:18,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:19,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:20,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:21,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:22,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:23,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:24,719 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:27:24,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:25,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:25,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:27:25,827 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:27:26,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:27,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:28,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:29,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:30,719 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:27:30,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:31,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:32,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:33,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:34,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:35,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:36,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:27:36,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:37,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:38,353 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:27:38,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:39,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:40,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:40,831 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:27:40,831 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:27:41,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:41,921 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:27:42,525 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:27:42,526 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:27:42,526 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:27:42,532 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:27:43,112 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:27:43,442 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:44,438 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:45,114 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:27:45,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:46,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:47,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:47,444 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:27:48,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:49,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:50,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:51,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:52,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:53,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:27:53,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:54,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:55,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:55,891 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:27:55,892 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:27:56,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:57,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:58,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:27:58,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:27:59,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:00,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:01,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:02,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:03,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:04,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:28:04,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:05,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:06,447 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:07,452 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:08,368 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:28:08,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:09,445 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:28:09,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:10,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:10,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:28:10,895 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:28:11,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:12,429 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:28:12,430 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:28:12,430 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:28:12,432 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:28:13,138 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:28:13,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:13,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:14,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:28:14,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:15,140 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:28:15,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:16,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:17,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:18,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:19,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:20,525 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:28:20,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:21,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:22,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:23,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:24,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:25,525 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:28:25,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:25,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:28:25,895 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:28:26,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:27,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:28,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:29,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:30,525 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:28:30,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:31,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:32,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:33,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:34,592 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:35,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:36,525 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:28:36,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:37,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:38,363 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:28:38,591 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:39,592 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:40,594 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:40,898 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:28:40,899 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:28:41,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:42,017 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:28:42,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:43,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:44,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:45,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:46,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:28:46,130 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:28:46,130 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:28:46,133 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:28:46,169 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:28:46,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:47,170 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:28:47,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:28:47,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:48,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:49,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:50,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:51,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:52,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:28:52,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:53,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:54,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:55,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:55,898 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:28:55,899 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:28:56,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:57,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:57,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:28:58,603 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:28:59,603 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:00,603 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:01,603 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:02,603 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:03,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:29:03,603 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:04,603 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:05,603 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:06,603 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:07,603 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:08,367 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:29:08,604 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:09,530 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:29:09,604 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:10,604 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:10,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:29:10,899 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:29:11,604 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:12,604 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:13,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:14,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:29:14,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:15,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:16,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:17,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:18,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:19,065 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:29:19,066 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:29:19,067 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:29:19,072 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:29:19,200 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:29:19,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:20,106 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:29:20,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:21,202 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:29:21,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:22,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:23,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:24,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:25,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:29:25,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:25,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:29:25,899 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:29:26,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:27,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:28,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:29,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:30,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:31,531 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:29:31,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:32,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:33,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:34,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:35,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:36,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:29:36,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:37,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:38,371 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:29:38,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:39,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:40,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:40,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:29:40,899 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:29:41,744 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:42,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:29:42,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:43,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:44,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:45,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:46,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:47,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:29:47,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:48,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:48,761 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:29:48,763 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:29:48,763 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:29:48,765 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:29:49,229 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:29:49,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:50,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:51,231 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:29:51,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:52,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:52,804 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:29:53,752 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:54,752 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:55,752 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:55,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:29:55,899 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:29:56,752 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:57,752 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:58,065 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:29:58,752 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:29:59,752 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:00,752 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:01,752 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:02,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:03,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:30:03,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:04,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:05,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:06,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:07,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:08,375 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:30:08,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:09,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:30:09,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:10,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:10,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:30:10,900 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:30:11,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:12,866 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:13,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:14,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:30:14,867 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:15,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:16,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:17,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:18,683 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:30:18,685 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:30:18,685 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:30:18,687 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:30:19,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:19,259 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:30:19,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:30:20,097 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:21,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:21,260 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:30:22,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:23,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:24,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:25,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:25,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:30:25,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:30:25,900 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:30:26,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:27,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:28,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:29,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:30,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:31,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:31,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:30:32,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:33,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:34,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:35,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:36,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:36,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:30:37,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:38,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:38,379 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:30:39,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:40,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:40,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:30:40,900 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:30:41,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:42,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:30:42,214 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:43,214 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:44,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:45,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:46,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:47,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:47,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:30:48,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:49,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:49,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:30:49,339 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:30:49,340 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:30:49,342 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:30:50,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:50,286 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:30:51,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:51,287 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:30:52,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:53,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:53,381 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:30:54,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:55,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:55,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:30:55,900 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:30:56,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:57,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:58,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:30:58,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:30:59,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:00,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:01,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:02,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:03,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:03,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:31:04,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:05,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:06,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:07,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:08,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:08,384 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:31:09,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:09,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:31:10,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:10,900 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:31:10,900 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:31:11,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:12,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:13,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:14,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:15,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:15,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:31:16,234 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:17,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:18,226 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:19,232 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:20,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:21,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:21,236 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:31:21,237 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:31:21,238 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:31:21,238 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:31:21,240 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:31:21,318 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:31:22,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:23,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:23,320 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:31:24,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:25,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:25,900 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:31:25,900 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:31:26,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:27,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:31:27,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:28,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:29,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:30,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:31,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:32,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:32,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:31:33,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:34,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:35,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:36,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:37,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:37,538 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:31:38,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:38,387 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:31:39,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:40,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:40,900 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:31:40,900 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:31:41,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:42,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:43,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:31:43,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:44,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:45,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:46,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:47,259 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:48,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:48,540 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:31:49,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:50,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:50,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:31:50,725 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:31:50,726 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:31:50,727 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:31:51,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:51,355 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:31:52,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:53,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:53,356 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:31:53,541 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:31:54,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:55,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:55,900 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:31:55,901 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:31:56,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:57,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:58,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:59,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:31:59,541 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:32:00,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:01,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:02,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:03,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:04,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:04,541 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:32:05,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:06,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:07,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:08,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:08,391 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:32:09,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:10,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:10,542 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:32:10,900 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:32:10,900 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:32:11,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:12,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:13,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:14,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:15,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:15,557 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:32:16,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:17,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:18,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:18,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:32:18,334 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:32:18,335 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:32:18,337 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:32:18,384 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:32:19,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:19,385 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:32:20,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:20,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:32:21,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:22,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:23,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:24,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:25,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:25,900 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:32:25,901 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:32:25,992 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:32:26,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:27,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:28,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:29,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:30,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:31,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:31,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:32:32,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:33,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:34,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:35,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:36,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:36,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:32:37,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:38,395 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:32:38,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:39,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:40,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:40,900 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:32:40,901 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:32:41,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:42,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:32:42,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:43,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:44,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:45,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:46,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:47,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:47,756 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:32:48,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:49,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:49,973 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:32:49,974 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:32:49,975 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:32:49,976 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:32:50,422 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:32:50,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:51,423 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:32:51,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:52,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:53,015 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:32:53,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:54,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:55,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:55,900 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:32:55,901 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:32:56,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:57,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:58,074 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:32:58,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:32:59,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:00,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:01,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:02,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:03,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:03,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:33:04,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:05,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:06,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:07,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:08,398 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:33:08,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:09,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:09,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:33:10,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:10,900 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:33:10,901 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:33:11,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:12,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:13,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:14,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:14,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:33:15,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:16,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:17,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:18,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:19,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:19,757 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:33:20,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:21,362 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:33:21,365 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:33:21,366 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:33:21,367 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:33:21,451 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:33:21,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:22,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:23,453 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:33:23,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:24,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:25,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:25,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:33:25,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:33:25,901 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:33:26,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:27,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:28,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:29,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:30,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:31,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:31,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:33:32,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:33,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:34,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:35,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:36,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:36,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:33:37,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:38,402 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:33:38,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:39,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:40,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:40,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:33:40,901 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:33:41,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:42,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:33:42,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:43,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:44,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:45,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:46,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:47,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:47,761 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:33:48,461 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:33:48,462 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:33:48,462 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:33:48,463 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:33:48,475 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:33:48,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:49,476 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:33:49,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:50,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:51,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:52,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:52,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:33:53,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:54,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:55,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:55,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:33:55,901 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:33:56,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:57,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:57,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:33:58,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:33:59,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:00,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:01,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:02,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:03,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:03,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:34:04,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:05,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:06,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:07,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:08,406 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:34:08,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:09,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:09,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:34:10,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:10,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:34:10,901 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:34:11,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:12,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:13,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:14,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:14,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:34:15,624 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:16,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:17,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:18,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:19,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:20,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:20,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:34:20,849 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:34:20,849 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:34:20,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:34:20,852 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:34:21,505 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:34:21,506 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:34:21,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:22,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:23,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:24,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:25,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:25,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:34:25,902 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:34:26,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:34:26,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:27,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:28,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:29,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:30,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:31,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:31,908 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:34:32,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:33,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:34,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:35,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:36,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:36,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:34:37,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:38,410 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:34:38,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:39,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:40,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:40,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:34:40,901 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:34:41,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:41,984 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:34:42,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:43,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:44,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:45,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:46,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:47,628 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:47,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:34:48,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:49,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:50,629 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:51,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:52,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:52,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:34:52,848 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:34:52,848 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:34:52,850 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:34:53,533 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:34:53,534 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:34:53,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:53,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:34:54,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:55,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:55,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:34:55,902 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:34:56,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:57,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:58,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:34:58,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:34:59,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:00,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:01,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:02,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:03,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:03,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:35:04,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:05,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:06,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:07,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:08,414 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:35:08,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:08,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:35:09,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:10,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:10,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:35:10,902 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:35:11,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:12,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:13,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:13,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:35:14,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:15,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:16,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:17,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:18,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:19,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:19,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:35:20,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:21,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:22,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:23,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:24,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:24,923 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:35:25,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:35:25,375 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:35:25,376 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:35:25,377 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:35:25,569 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:35:25,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:25,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:35:25,902 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:35:26,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:27,572 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:35:27,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:28,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:29,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:30,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:30,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:35:31,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:32,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:33,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:34,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:35,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:36,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:36,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:35:37,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:38,418 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:35:38,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:39,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:40,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:40,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:35:40,902 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:35:41,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:42,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:35:42,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:43,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:44,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:45,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:46,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:47,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:47,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:35:48,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:49,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:50,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:51,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:52,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:53,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:53,925 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:35:54,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:55,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:55,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:35:55,906 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:35:56,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:57,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:58,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:58,927 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:35:59,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:35:59,898 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:35:59,900 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:35:59,901 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:35:59,902 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:36:00,601 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:36:00,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:01,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:02,602 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:36:02,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:03,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:03,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:36:04,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:05,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:06,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:07,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:08,422 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:36:08,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:09,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:09,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:36:10,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:10,904 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:36:10,904 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:36:11,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:12,997 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:13,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:14,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:14,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:36:15,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:16,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:17,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:18,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:19,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:20,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:20,925 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:36:21,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:22,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:23,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:24,906 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:25,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:36:25,919 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:36:25,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:26,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:36:26,913 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:27,913 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:28,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:29,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:30,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:31,380 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:36:31,381 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:36:31,381 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:36:31,381 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:36:31,382 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:36:31,633 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:36:31,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:32,634 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:36:32,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:33,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:34,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:35,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:36,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:36,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:36:37,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:38,425 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:36:38,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:39,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:40,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:36:40,910 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:36:40,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:41,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:42,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:36:42,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:43,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:44,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:45,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:46,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:47,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:47,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:36:48,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:49,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:50,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:51,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:52,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:52,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:36:53,917 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:54,917 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:55,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:36:55,910 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:36:55,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:56,917 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:57,927 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:58,083 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:36:58,922 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:36:59,924 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:00,922 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:01,922 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:02,922 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:03,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:37:03,715 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:37:03,716 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:37:03,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:37:03,718 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:37:04,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:04,664 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:37:05,131 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:06,131 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:06,666 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:37:07,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:08,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:08,429 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:37:09,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:09,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:37:10,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:10,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:37:10,910 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:37:11,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:12,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:13,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:14,133 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:15,133 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:37:15,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:16,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:17,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:18,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:19,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:20,133 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:37:20,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:21,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:22,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:23,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:24,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:25,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:25,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:37:25,910 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:37:26,022 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:37:26,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:27,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:28,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:29,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:30,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:31,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:37:31,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:32,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:32,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:37:32,535 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:37:32,536 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:37:32,537 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:37:32,693 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:37:33,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:34,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:34,695 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:37:35,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:36,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:37,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:37:37,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:38,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:38,433 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:37:39,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:40,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:40,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:37:40,910 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:37:41,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:42,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:42,991 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:37:43,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:44,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:45,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:46,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:47,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:48,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:37:48,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:49,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:50,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:51,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:52,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:53,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:37:53,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:54,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:55,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:55,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:37:55,910 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:37:56,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:57,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:58,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:37:59,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:37:59,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:00,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:01,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:02,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:03,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:04,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:38:04,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:05,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:05,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:38:05,348 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:38:05,348 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:38:05,350 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:38:05,727 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:38:06,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:06,728 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:38:07,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:08,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:08,437 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:38:09,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:10,141 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:38:10,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:10,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:38:10,911 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:38:11,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:12,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:13,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:14,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:15,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:16,141 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:38:16,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:17,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:18,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:19,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:20,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:21,142 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:38:21,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:22,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:23,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:24,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:25,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:25,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:38:25,911 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:38:26,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:26,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:38:27,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:28,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:29,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:30,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:31,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:32,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:38:32,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:33,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:34,295 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:35,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:36,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:38:36,214 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:38:36,214 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:38:36,216 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:38:36,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:36,755 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:38:37,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:38:37,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:38,441 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:38:38,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:38,757 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:38:39,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:40,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:40,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:38:40,911 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:38:41,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:42,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:43,046 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:38:43,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:44,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:45,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:46,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:47,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:48,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:38:48,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:49,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:50,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:51,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:52,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:53,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:38:53,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:54,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:55,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:55,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:38:55,911 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:38:56,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:57,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:58,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:38:59,086 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:38:59,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:00,629 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:01,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:02,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:03,623 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:04,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:39:04,624 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:05,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:06,299 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:39:06,300 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:39:06,301 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:39:06,302 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:39:06,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:06,785 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:39:07,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:08,445 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:39:08,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:08,786 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:39:09,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:39:09,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:10,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:10,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:39:10,911 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:39:11,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:12,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:13,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:14,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:39:14,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:15,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:16,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:17,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:18,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:19,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:39:19,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:20,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:21,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:22,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:23,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:24,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:39:24,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:25,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:25,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:39:25,911 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:39:26,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:27,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:28,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:29,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:30,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:39:30,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:31,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:32,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:33,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:34,720 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:35,222 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:39:35,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:36,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:37,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:39:37,570 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:39:37,571 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:39:37,572 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:39:37,817 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:39:37,981 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:38,449 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:39:38,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:38,818 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:39:39,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:40,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:40,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:39:40,911 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:39:41,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:39:41,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:42,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:43,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:44,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:45,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:46,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:39:46,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:47,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:48,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:49,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:50,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:51,217 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:39:51,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:52,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:53,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:54,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:55,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:55,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:39:55,911 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:39:56,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:57,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:39:57,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:58,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:39:59,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:00,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:01,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:02,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:40:02,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:03,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:04,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:05,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:06,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:40:06,316 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:40:06,317 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:40:06,318 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:40:06,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:06,846 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:40:07,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:40:07,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:08,453 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:40:08,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:08,848 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:40:09,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:10,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:10,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:40:10,911 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:40:11,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:12,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:13,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:40:13,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:14,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:15,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:16,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:17,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:18,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:40:18,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:19,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:20,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:21,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:22,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:23,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:40:23,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:24,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:25,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:25,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:40:25,912 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:40:26,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:27,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:28,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:29,088 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:40:29,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:30,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:31,739 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:32,736 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:33,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:34,225 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:40:34,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:35,739 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:36,740 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:37,022 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:40:37,024 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:40:37,025 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:40:37,026 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:40:37,740 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:37,871 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:40:38,457 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:40:38,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:38,872 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:40:39,225 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:40:39,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:40,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:40,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:40:40,912 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:40:41,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:42,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:43,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:44,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:45,225 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:40:45,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:46,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:47,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:48,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:49,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:50,226 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:40:50,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:51,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:52,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:53,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:54,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:55,226 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:40:55,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:55,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:40:55,912 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:40:56,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:57,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:58,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:40:59,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:00,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:01,226 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:41:01,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:02,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:03,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:04,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:05,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:06,232 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:41:06,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:07,756 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:41:07,757 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:41:07,758 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:41:07,759 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:41:07,901 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:41:08,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:08,461 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:41:08,902 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:41:09,168 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:10,168 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:10,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:41:10,912 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:41:11,168 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:12,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:41:12,168 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:13,168 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:14,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:15,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:16,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:17,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:17,226 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:41:18,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:19,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:20,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:21,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:22,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:23,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:23,226 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:41:24,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:25,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:25,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:41:25,912 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:41:26,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:27,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:28,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:29,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:41:29,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:30,171 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:31,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:32,175 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:33,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:34,175 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:34,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:41:35,174 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:36,177 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:37,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:37,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:41:37,880 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:41:37,880 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:41:37,882 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:41:37,934 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:41:38,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:38,466 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:41:38,936 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:41:39,230 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:41:39,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:40,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:40,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:41:40,912 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:41:41,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:42,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:43,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:44,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:45,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:41:45,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:46,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:47,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:48,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:49,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:50,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:51,230 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:41:51,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:52,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:53,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:54,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:55,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:55,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:41:55,912 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:41:56,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:57,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:41:57,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:58,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:41:59,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:00,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:01,300 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:02,235 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:42:02,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:03,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:04,308 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:05,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:06,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:07,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:42:07,158 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:42:07,158 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:42:07,160 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:42:07,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:07,963 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:42:08,199 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:42:08,469 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:42:08,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:08,964 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:42:09,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:10,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:10,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:42:10,912 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:42:11,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:12,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:13,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:14,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:42:14,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:15,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:16,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:17,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:18,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:19,234 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:42:19,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:20,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:21,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:22,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:23,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:24,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:25,234 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:42:25,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:25,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:42:25,912 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:42:26,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:27,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:28,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:29,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:30,581 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:31,236 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:42:31,576 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:32,576 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:33,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:34,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:35,580 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:36,239 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:42:36,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:42:36,520 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:42:36,520 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:42:36,522 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:42:36,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:36,989 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:42:37,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:38,473 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:42:38,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:38,990 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:42:39,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:40,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:40,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:42:40,913 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:42:41,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:41,985 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:42:42,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:43,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:44,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:45,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:46,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:47,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:42:47,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:48,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:49,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:50,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:51,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:52,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:53,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:42:53,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:54,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:55,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:55,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:42:55,913 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:42:56,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:57,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:58,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:42:59,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:42:59,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:00,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:01,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:02,679 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:03,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:04,240 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:43:04,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:05,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:06,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:07,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:08,475 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:43:08,551 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:43:08,553 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:43:08,553 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:43:08,555 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:43:08,958 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:09,020 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:43:09,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:43:09,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:10,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:10,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:43:10,913 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:43:11,021 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:43:11,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:12,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:13,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:14,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:15,242 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:43:15,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:16,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:17,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:18,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:19,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:20,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:43:20,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:21,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:22,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:23,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:24,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:25,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:25,912 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:43:25,913 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:43:26,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:43:26,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:27,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:28,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:29,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:30,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:31,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:43:31,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:32,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:33,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:34,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:35,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:36,251 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:43:36,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:37,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:38,484 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:43:38,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:39,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:40,438 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:43:40,440 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:43:40,441 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:43:40,442 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:43:40,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:40,913 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:43:40,913 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:43:41,055 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:43:41,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:42,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:43:42,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:43,057 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:43:43,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:44,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:45,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:46,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:47,245 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:43:47,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:48,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:49,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:50,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:51,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:52,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:53,245 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:43:53,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:54,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:55,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:55,913 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:43:55,913 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:43:56,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:57,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:58,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:43:59,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:43:59,857 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:00,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:01,857 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:02,857 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:03,857 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:04,245 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:44:04,857 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:05,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:06,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:07,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:08,488 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:44:08,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:09,251 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:44:09,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:10,863 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:10,913 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:44:10,914 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:44:11,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:44:11,079 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:44:11,079 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:44:11,081 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:44:11,086 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:44:12,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:13,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:13,088 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:44:14,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:14,251 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:44:15,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:16,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:17,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:18,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:19,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:20,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:20,251 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:44:21,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:22,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:23,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:24,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:25,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:25,251 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:44:25,913 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:44:25,913 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:44:26,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:27,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:28,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:29,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:30,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:31,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:31,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:44:32,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:33,029 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:34,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:35,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:36,034 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:36,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:44:37,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:38,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:38,489 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:44:39,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:39,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:44:39,621 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:44:39,621 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:44:39,623 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:44:40,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:40,118 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:44:40,913 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:44:40,914 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:44:41,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:41,119 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:44:42,022 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:44:42,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:43,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:44,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:45,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:46,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:47,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:47,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:44:48,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:49,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:50,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:51,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:52,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:52,253 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:44:53,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:54,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:55,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:55,913 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:44:55,914 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:44:56,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:57,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:58,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:44:58,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:44:59,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:00,057 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:01,058 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:02,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:03,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:03,264 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:45:04,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:05,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:06,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:06,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:45:06,203 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:45:06,203 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:45:06,204 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:45:07,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:07,145 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:45:07,146 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:45:08,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:08,492 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:45:08,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:45:09,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:10,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:10,913 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:45:10,914 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:45:11,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:12,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:13,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:14,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:14,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:45:15,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:16,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:17,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:18,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:19,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:19,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:45:20,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:21,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:22,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:23,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:24,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:25,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:25,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:45:25,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:45:25,914 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:45:26,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:27,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:28,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:29,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:30,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:30,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:45:31,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:32,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:33,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:34,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:35,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:35,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:45:36,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:36,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:45:36,345 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:45:36,345 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:45:36,346 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:45:37,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:37,173 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:45:37,173 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:45:38,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:38,496 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:45:39,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:40,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:40,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:45:40,914 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:45:41,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:45:41,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:42,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:43,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:44,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:45,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:46,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:46,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:45:47,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:48,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:49,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:50,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:51,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:51,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:45:52,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:53,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:54,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:55,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:55,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:45:55,914 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:45:56,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:57,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:45:57,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:58,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:45:59,084 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:00,074 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:01,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:02,083 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:02,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:46:03,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:04,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:04,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:46:04,838 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:46:04,839 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:46:04,840 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:46:05,212 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:46:05,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:06,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:07,214 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:46:07,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:07,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:46:08,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:08,501 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:46:09,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:10,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:10,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:46:10,914 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:46:11,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:12,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:13,077 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:46:13,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:14,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:15,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:16,485 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:17,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:18,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:46:18,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:19,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:20,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:21,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:22,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:23,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:24,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:46:24,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:25,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:25,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:46:25,914 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:46:26,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:27,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:28,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:29,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:46:29,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:30,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:31,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:32,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:33,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:33,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:46:33,786 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:46:33,786 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:46:33,788 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:46:34,247 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:46:34,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:46:34,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:35,248 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:46:35,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:36,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:37,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:38,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:38,505 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:46:39,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:40,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:46:40,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:40,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:46:40,915 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:46:41,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:42,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:43,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:44,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:45,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:46:45,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:46,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:47,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:48,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:49,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:50,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:46:50,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:51,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:52,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:53,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:54,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:55,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:55,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:46:55,915 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:46:55,993 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:46:56,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:57,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:58,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:46:59,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:00,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:01,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:47:01,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:02,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:02,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:47:02,859 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:47:02,859 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:47:02,861 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:47:03,275 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:47:03,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:04,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:05,276 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:47:05,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:06,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:47:06,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:07,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:08,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:08,509 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:47:09,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:10,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:10,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:47:10,915 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:47:11,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:12,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:47:12,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:13,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:14,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:15,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:16,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:17,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:47:17,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:18,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:19,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:20,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:21,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:22,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:47:22,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:23,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:24,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:25,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:25,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:47:25,915 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:47:26,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:27,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:28,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:47:28,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:29,501 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:30,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:31,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:32,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:33,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:47:33,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:34,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:34,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:47:34,613 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:47:34,614 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:47:34,616 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:47:35,305 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:47:35,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:36,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:37,307 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:47:37,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:38,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:38,513 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:47:38,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:47:39,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:40,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:40,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:47:40,915 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:47:41,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:42,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:43,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:43,993 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:47:44,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:45,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:46,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:47,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:48,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:49,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:47:49,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:50,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:51,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:52,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:53,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:54,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:47:54,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:55,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:55,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:47:55,915 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:47:56,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:57,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:58,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:47:59,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:47:59,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:00,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:01,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:02,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:03,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:04,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:05,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:48:05,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:05,965 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:48:05,966 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:48:05,966 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:48:05,967 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:48:06,332 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:48:06,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:07,333 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:48:07,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:08,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:08,516 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:48:09,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:10,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:10,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:48:10,915 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:48:11,082 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:48:11,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:12,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:13,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:14,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:15,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:16,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:48:16,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:17,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:18,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:19,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:20,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:21,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:48:21,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:22,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:23,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:24,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:25,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:25,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:48:25,915 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:48:26,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:27,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:48:27,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:28,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:29,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:30,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:31,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:32,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:48:32,524 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:33,525 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:34,525 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:35,523 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:36,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:37,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:48:37,203 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:48:37,203 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:48:37,205 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:48:37,361 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:48:37,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:38,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:48:38,520 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:48:38,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:39,362 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:48:39,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:40,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:40,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:48:40,916 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:48:41,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:42,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:43,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:44,082 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:48:44,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:45,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:46,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:47,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:48,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:49,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:48:49,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:50,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:51,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:52,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:53,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:54,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:48:54,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:55,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:55,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:48:55,916 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:48:56,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:57,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:58,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:48:59,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:00,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:49:00,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:01,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:02,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:03,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:04,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:05,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:49:05,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:06,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:07,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:08,527 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:49:08,623 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:09,623 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:09,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:49:09,648 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:49:09,649 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:49:09,651 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:49:10,392 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:49:10,623 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:10,689 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:49:10,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:49:10,921 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:49:11,393 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:49:11,623 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:12,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:13,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:14,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:15,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:16,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:49:16,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:17,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:18,641 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:19,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:20,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:21,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:22,289 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:49:22,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:23,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:24,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:25,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:25,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:49:25,920 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:49:26,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:27,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:28,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:49:28,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:29,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:30,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:31,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:32,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:33,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:49:33,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:34,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:35,652 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:36,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:37,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:38,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:49:38,534 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:49:38,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:39,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:40,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:49:40,288 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:49:40,289 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:49:40,290 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:49:40,423 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:49:40,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:40,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:49:40,921 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:49:41,424 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:49:41,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:42,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:43,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:44,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:49:44,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:45,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:46,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:47,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:48,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:49,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:49:49,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:50,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:51,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:52,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:53,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:54,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:49:54,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:55,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:55,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:49:55,920 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:49:56,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:57,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:58,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:49:59,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:00,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:50:00,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:01,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:02,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:03,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:04,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:05,293 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:50:05,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:06,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:07,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:08,539 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:50:08,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:09,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:10,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:50:10,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:50:10,437 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:50:10,437 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:50:10,439 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:50:10,452 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:50:10,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:10,921 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:50:10,922 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:50:11,453 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:50:11,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:12,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:13,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:14,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:15,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:50:15,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:16,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:17,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:18,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:19,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:20,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:21,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:50:21,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:22,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:23,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:24,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:25,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:25,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:50:25,921 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:50:26,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:26,994 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:50:27,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:28,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:29,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:30,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:31,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:32,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:50:32,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:33,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:34,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:35,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:36,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:37,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:50:37,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:38,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:50:38,522 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:50:38,522 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:50:38,523 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:50:38,538 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:50:38,935 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:39,486 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:50:39,486 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:50:39,932 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:40,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:50:40,921 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:50:40,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:41,932 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:42,932 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:43,058 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:50:43,932 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:44,932 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:45,932 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:46,933 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:47,933 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:48,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:50:48,933 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:49,933 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:50,933 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:51,933 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:52,934 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:53,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:50:53,933 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:54,933 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:55,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:50:55,921 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:50:55,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:56,934 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:57,934 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:58,934 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:50:59,006 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:50:59,934 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:00,934 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:01,934 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:02,934 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:03,934 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:04,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:51:04,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:05,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:06,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:07,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:08,543 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:51:08,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:09,305 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:51:09,938 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:10,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:51:10,632 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:51:10,632 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:51:10,635 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:51:11,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:11,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:51:11,055 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:51:11,512 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:51:11,512 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:51:12,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:13,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:14,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:15,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:15,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:51:16,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:17,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:18,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:19,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:20,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:21,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:21,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:51:22,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:23,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:24,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:25,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:26,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:51:26,052 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:51:26,092 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:27,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:27,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:51:28,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:29,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:30,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:31,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:32,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:32,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:51:33,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:34,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:35,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:36,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:37,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:37,317 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:51:38,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:38,545 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:51:39,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:39,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:51:39,788 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:51:39,789 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:51:39,791 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:51:40,198 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:40,538 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:51:41,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:51:41,053 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:51:41,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:41,539 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:51:42,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:43,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:43,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:51:44,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:45,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:46,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:47,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:48,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:48,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:51:49,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:50,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:51,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:52,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:53,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:53,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:51:54,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:55,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:56,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:51:56,053 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:51:56,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:57,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:58,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:51:59,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:51:59,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:00,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:01,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:02,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:03,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:04,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:04,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:52:05,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:06,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:07,172 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:52:07,174 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:52:07,174 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:52:07,176 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:52:07,571 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:52:07,592 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:08,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:08,549 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:52:09,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:09,574 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:52:10,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:52:10,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:11,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:52:11,052 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:52:11,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:12,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:13,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:14,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:15,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:16,309 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:52:16,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:17,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:18,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:19,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:20,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:21,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:22,310 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:52:22,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:23,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:24,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:25,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:26,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:52:26,053 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:52:26,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:27,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:28,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:52:28,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:29,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:30,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:31,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:32,328 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:33,314 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:52:33,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:34,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:35,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:36,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:37,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:38,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:52:38,064 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:52:38,065 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:52:38,068 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:52:38,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:38,554 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:52:38,556 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:52:38,605 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:52:39,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:39,606 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:52:40,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:41,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:52:41,053 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:52:41,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:42,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:43,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:44,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:52:44,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:45,534 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:46,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:47,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:48,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:49,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:52:49,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:50,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:51,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:52,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:53,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:54,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:52:54,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:55,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:56,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:52:56,053 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:52:56,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:57,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:58,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:52:59,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:00,316 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:53:00,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:01,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:02,547 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:03,543 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:04,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:05,320 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:53:05,543 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:06,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:07,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:07,967 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:53:07,968 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:53:07,968 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:53:07,970 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:53:08,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:08,557 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:53:08,631 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:53:09,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:09,632 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:53:10,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:11,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:53:11,053 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:53:11,167 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:53:11,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:12,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:13,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:14,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:15,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:16,318 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:53:16,544 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:17,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:18,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:19,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:20,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:21,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:22,319 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:53:22,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:23,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:24,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:25,545 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:26,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:53:26,053 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:53:26,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:27,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:28,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:53:28,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:29,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:30,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:31,546 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:32,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:33,325 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:53:33,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:34,549 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:35,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:36,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:37,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:37,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:53:37,839 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:53:37,839 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:53:37,841 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:53:38,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:38,561 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:53:38,567 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:53:38,664 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:53:39,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:39,666 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:53:40,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:41,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:53:41,053 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:53:41,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:42,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:43,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:44,315 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:53:44,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:45,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:46,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:47,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:48,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:49,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:53:49,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:50,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:51,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:52,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:53,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:54,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:55,322 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:53:55,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:56,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:53:56,053 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:53:56,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:57,558 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:58,559 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:53:59,559 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:00,336 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:54:00,559 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:01,559 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:02,559 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:03,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:54:03,457 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:54:03,458 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:54:03,460 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:54:03,687 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:54:03,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:04,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:05,689 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:54:05,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:06,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:54:06,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:07,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:08,565 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:54:08,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:09,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:10,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:11,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:54:11,053 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:54:11,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:12,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:54:12,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:13,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:14,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:15,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:16,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:17,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:54:17,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:18,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:19,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:20,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:21,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:22,326 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:54:22,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:23,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:24,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:25,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:26,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:54:26,053 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:54:26,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:27,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:28,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:54:28,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:29,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:30,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:31,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:32,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:33,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:54:33,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:34,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:35,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:36,452 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:54:36,453 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:54:36,453 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:54:36,454 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:54:36,717 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:54:36,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:37,718 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:54:37,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:38,493 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:54:38,569 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:54:38,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:39,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:40,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:41,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:54:41,053 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:54:41,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:42,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:43,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:44,167 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:54:44,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:45,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:46,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:47,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:48,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:49,330 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:54:49,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:50,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:51,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:52,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:53,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:54,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:55,329 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:54:55,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:56,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:54:56,054 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:54:56,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:57,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:58,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:54:59,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:00,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:55:00,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:01,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:02,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:03,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:04,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:05,344 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:55:05,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:06,177 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:55:06,179 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:55:06,179 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:55:06,181 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:55:06,756 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:55:06,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:07,757 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:55:07,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:08,574 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:55:08,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:09,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:10,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:11,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:55:11,054 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:55:11,212 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:55:11,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:12,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:13,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:14,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:15,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:16,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:55:16,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:17,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:18,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:19,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:20,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:21,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:55:21,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:22,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:23,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:24,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:25,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:26,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:55:26,054 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:55:26,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:27,168 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:55:27,888 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:28,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:29,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:30,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:31,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:32,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:55:32,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:33,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:55:33,857 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:55:33,858 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:55:33,860 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:55:34,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:34,786 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:55:35,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:35,786 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:55:36,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:37,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:38,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:38,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:55:38,577 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:55:39,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:40,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:41,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:55:41,054 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:55:41,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:42,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:43,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:44,175 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:55:44,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:45,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:46,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:47,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:48,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:49,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:49,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:55:50,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:51,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:52,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:53,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:54,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:55,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:55,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:55:56,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:55:56,054 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:55:56,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:57,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:58,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:55:59,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:00,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:00,356 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:56:01,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:02,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:03,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:04,088 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:56:04,089 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:56:04,089 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:56:04,091 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:56:04,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:04,817 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:56:05,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:05,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:56:05,818 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:56:06,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:07,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:08,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:08,581 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:56:09,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:10,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:11,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:56:11,054 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:56:11,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:56:11,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:12,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:13,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:14,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:15,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:16,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:56:16,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:17,556 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:18,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:19,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:20,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:21,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:56:21,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:22,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:23,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:24,553 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:25,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:26,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:56:26,054 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:56:26,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:27,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:56:27,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:28,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:29,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:30,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:31,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:32,535 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:56:32,556 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:33,559 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:34,560 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:35,199 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:56:35,200 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:56:35,200 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:56:35,201 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:56:35,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:35,852 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:56:35,853 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:56:36,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:37,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:38,240 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:56:38,585 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:56:38,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:39,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:40,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:41,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:56:41,054 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:56:41,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:42,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:43,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:44,210 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:56:44,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:45,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:46,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:47,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:48,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:49,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:56:49,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:50,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:51,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:52,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:53,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:54,510 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:56:54,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:55,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:56,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:56:56,054 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:56:56,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:57,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:58,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:56:59,521 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:56:59,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:00,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:01,616 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:02,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:03,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:04,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:04,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:57:04,882 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:57:04,883 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:57:04,883 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:57:04,884 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:57:04,885 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:57:05,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:05,887 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:57:06,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:07,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:08,588 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:57:08,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:09,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:10,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:57:10,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:11,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:57:11,055 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:57:11,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:12,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:13,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:14,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:15,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:16,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:57:16,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:17,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:18,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:19,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:20,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:21,516 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:57:21,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:22,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:23,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:24,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:25,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:26,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:57:26,055 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:57:26,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:27,213 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:57:27,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:28,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:29,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:30,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:31,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:32,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:57:32,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:33,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:34,629 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:35,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:35,678 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:57:35,680 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:57:35,680 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:57:35,682 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:57:35,913 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:57:36,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:37,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:37,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:57:37,915 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:57:38,593 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:57:38,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:39,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:40,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:41,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:57:41,055 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:57:41,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:42,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:43,216 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:57:43,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:44,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:45,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:46,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:47,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:48,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:57:48,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:49,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:50,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:51,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:52,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:53,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:54,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:57:54,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:55,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:56,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:57:56,055 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:57:56,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:57,632 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:58,632 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:57:59,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:57:59,632 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:00,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:01,635 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:02,635 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:03,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:04,521 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:58:04,640 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:05,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:05,996 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:58:05,997 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:58:05,997 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:58:05,998 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:58:06,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:06,953 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:58:07,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:07,954 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:58:08,595 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:58:08,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:09,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:10,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:58:10,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:11,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:58:11,055 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:58:11,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:12,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:13,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:14,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:15,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:16,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:58:16,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:17,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:18,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:19,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:20,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:21,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:22,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:58:22,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:23,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:24,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:25,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:26,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:58:26,055 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:58:26,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:27,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:28,192 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:58:28,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:29,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:30,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:31,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:32,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:33,525 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:58:33,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:34,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:35,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:36,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:58:36,343 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:58:36,343 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:58:36,345 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:58:36,755 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:36,980 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:58:37,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:37,981 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:58:38,599 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:58:38,605 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:58:38,757 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:39,757 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:40,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:41,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:58:41,056 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:58:41,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:42,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:43,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:44,187 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:58:44,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:45,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:46,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:47,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:48,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:49,523 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:58:49,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:50,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:51,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:52,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:53,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:54,524 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:58:54,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:55,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:56,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:58:56,056 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:58:56,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:57,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:58,760 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:58:59,760 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:00,524 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:59:00,760 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:01,760 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:02,760 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:03,760 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:04,764 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:05,547 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:59:05,764 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:06,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:07,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:08,601 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:59:08,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:08,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:59:08,979 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:59:08,980 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:59:08,982 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:59:09,007 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:59:09,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:10,008 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:59:10,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:11,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:59:11,056 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:59:11,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:59:11,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:12,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:13,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:14,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:15,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:16,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:59:16,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:17,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:18,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:19,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:20,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:21,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:22,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:59:22,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:23,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:24,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:25,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:26,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:59:26,056 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:59:26,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:27,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:28,177 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:59:28,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:29,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:30,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:31,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:32,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:33,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:59:33,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:34,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:35,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:36,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:37,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:38,540 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:59:38,606 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 01:59:38,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:39,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 01:59:39,231 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 01:59:39,231 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 01:59:39,232 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 01:59:39,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:40,036 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 01:59:40,036 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 01:59:40,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:41,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:59:41,056 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:59:41,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:42,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:43,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:44,175 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:59:44,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:45,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:46,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:47,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:48,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:49,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:59:49,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:50,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:51,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:52,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:53,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:54,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 01:59:54,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:55,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:56,055 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 01:59:56,056 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 01:59:56,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:57,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:58,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 01:59:59,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:00,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:00:00,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:01,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:02,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:03,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:04,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:05,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:00:05,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:06,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:07,882 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:08,613 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:00:08,886 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:09,883 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:10,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:00:10,372 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:00:10,373 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:00:10,374 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:00:10,886 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:11,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:00:11,056 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:00:11,069 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:00:11,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:00:11,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:12,070 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:00:12,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:13,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:14,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:15,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:16,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:00:16,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:17,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:18,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:19,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:20,886 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:21,886 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:22,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:00:22,886 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:23,886 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:24,886 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:25,886 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:26,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:00:26,056 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:00:26,887 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:27,886 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:28,215 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:00:28,886 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:29,887 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:30,887 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:31,887 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:32,887 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:33,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:00:33,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:34,887 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:35,892 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:36,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:37,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:38,613 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:00:38,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:39,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:00:39,896 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:40,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:40,958 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:00:40,960 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:00:40,960 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:00:40,962 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:00:41,101 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:00:41,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:00:41,738 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:00:41,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:42,102 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:00:42,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:43,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:44,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:00:44,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:45,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:46,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:47,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:48,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:49,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:50,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:00:50,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:51,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:52,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:53,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:54,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:55,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:00:55,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:56,104 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:00:56,105 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:00:56,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:57,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:58,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:00:59,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:00,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:01,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:01:01,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:02,895 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:03,895 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:04,895 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:05,895 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:06,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:01:06,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:07,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:08,618 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:01:08,905 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:09,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:10,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:11,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:01:11,110 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:01:11,902 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:11,958 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:01:11,960 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:01:11,960 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:01:11,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:01:11,962 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:01:12,133 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:01:13,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:14,088 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:14,134 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:01:15,088 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:16,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:17,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:17,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:01:18,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:19,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:20,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:21,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:22,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:23,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:23,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:01:24,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:25,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:26,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:26,106 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:01:26,107 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:01:27,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:28,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:29,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:29,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:01:30,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:31,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:32,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:33,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:34,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:34,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:01:35,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:36,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:37,091 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:38,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:38,630 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:01:39,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:39,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:01:40,096 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:41,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:41,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:01:41,112 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:01:42,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:42,427 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:01:42,428 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:01:42,429 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:01:42,430 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:01:43,165 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:01:43,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:44,167 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:01:44,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:45,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:45,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:01:46,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:47,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:48,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:49,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:50,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:51,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:51,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:01:52,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:53,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:54,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:55,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:56,112 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:01:56,113 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:01:56,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:57,193 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:01:57,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:58,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:01:59,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:00,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:01,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:02,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:02,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:02:03,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:04,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:05,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:06,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:07,504 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:07,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:02:08,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:08,634 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:02:09,512 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:10,505 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:11,116 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:02:11,120 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:02:11,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:12,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:13,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:02:13,057 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:02:13,058 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:02:13,058 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:02:13,059 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:02:13,199 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:02:13,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:14,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:15,201 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:02:15,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:16,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:17,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:18,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:18,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:02:19,506 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:20,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:21,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:22,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:23,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:24,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:24,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:02:25,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:26,116 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:02:26,117 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:02:26,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:27,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:28,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:29,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:29,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:02:30,507 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:31,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:32,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:33,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:34,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:35,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:35,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:02:36,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:37,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:38,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:38,632 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:02:39,508 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:40,509 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:40,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:02:41,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:02:41,127 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:02:41,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:42,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:43,521 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:44,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:45,518 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:45,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:02:46,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:46,602 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:02:46,604 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:02:46,604 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:02:46,607 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:02:47,235 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:02:47,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:48,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:49,237 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:02:49,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:50,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:50,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:02:51,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:52,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:53,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:54,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:55,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:56,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:02:56,122 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:02:56,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:02:56,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:57,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:58,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:02:59,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:00,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:01,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:03:01,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:02,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:03,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:04,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:05,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:06,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:03:06,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:07,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:08,637 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:03:08,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:09,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:10,683 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:11,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:03:11,129 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:03:11,686 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:12,223 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:03:12,680 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:13,678 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:14,678 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:15,678 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:16,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:03:16,669 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:03:16,669 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:03:16,674 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:03:17,075 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:17,270 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:03:17,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:03:18,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:19,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:19,272 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:03:20,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:21,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:22,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:23,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:23,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:03:24,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:25,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:26,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:26,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:03:26,126 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:03:27,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:28,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:29,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:29,251 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:03:30,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:31,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:32,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:33,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:34,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:34,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:03:35,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:36,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:37,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:38,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:38,640 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:03:39,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:40,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:40,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:03:41,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:41,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:03:41,126 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:03:42,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:43,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:44,076 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:45,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:45,687 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:03:46,083 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:47,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:48,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:49,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:49,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:03:49,292 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:03:49,293 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:03:49,296 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:03:49,314 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:03:50,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:50,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:03:51,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:51,316 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:03:52,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:53,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:54,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:55,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:56,078 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:56,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:03:56,126 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:03:56,233 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:03:57,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:58,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:03:59,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:00,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:01,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:01,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:04:02,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:03,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:04,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:05,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:06,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:06,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:04:07,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:08,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:08,644 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:04:09,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:10,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:11,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:11,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:04:11,126 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:04:12,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:12,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:04:13,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:14,086 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:15,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:16,087 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:17,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:17,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:04:18,088 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:19,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:19,317 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:04:19,318 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:04:19,318 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:04:19,320 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:04:19,355 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:04:20,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:21,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:21,358 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:04:22,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:22,729 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:04:23,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:24,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:25,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:26,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:26,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:04:26,126 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:04:27,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:28,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:28,222 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:04:29,089 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:30,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:31,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:32,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:33,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:33,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:04:34,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:35,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:36,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:37,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:38,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:38,648 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:04:39,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:39,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:04:40,091 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:41,091 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:41,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:04:41,126 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:04:42,091 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:43,091 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:44,091 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:44,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:04:45,091 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:46,091 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:47,091 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:48,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:49,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:49,731 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:04:50,104 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:51,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:52,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:52,487 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:04:52,489 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:04:52,489 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:04:52,494 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:04:53,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:53,384 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:04:54,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:54,731 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:04:55,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:55,385 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:04:56,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:56,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:04:56,127 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:04:57,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:58,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:04:59,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:00,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:00,731 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:05:01,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:02,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:03,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:04,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:05,103 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:05,731 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:05:06,103 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:07,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:08,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:08,653 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:05:09,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:10,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:10,732 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:05:11,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:05:11,127 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:05:11,167 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:12,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:13,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:14,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:15,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:15,732 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:05:16,148 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:17,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:18,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:19,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:20,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:21,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:21,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:05:22,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:23,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:24,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:25,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:25,917 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:05:25,919 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:05:25,920 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:05:25,925 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:05:26,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:26,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:05:26,341 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:05:26,412 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:05:27,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:27,413 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:05:27,511 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:05:28,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:29,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:30,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:31,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:32,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:32,736 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:05:33,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:34,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:35,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:36,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:37,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:37,736 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:05:38,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:38,657 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:05:39,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:40,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:41,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:05:41,338 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:05:41,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:42,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:43,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:43,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:05:44,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:45,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:46,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:47,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:48,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:48,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:05:49,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:50,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:51,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:52,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:53,344 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:53,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:05:54,344 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:55,193 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:05:55,195 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:05:55,195 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:05:55,197 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:05:55,441 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:05:55,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:56,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:05:56,338 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:05:56,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:57,443 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:05:57,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:58,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:05:59,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:05:59,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:00,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:01,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:02,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:03,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:04,607 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:04,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:06:05,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:06,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:07,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:08,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:08,661 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:06:09,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:09,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:06:10,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:11,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:06:11,338 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:06:11,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:12,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:13,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:14,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:14,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:06:15,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:16,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:17,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:18,609 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:19,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:19,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:06:20,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:21,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:22,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:23,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:24,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:24,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:06:24,888 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:06:24,889 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:06:24,890 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:06:24,892 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:06:25,468 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:06:25,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:26,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:06:26,338 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:06:26,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:27,469 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:06:27,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:28,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:29,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:29,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:06:30,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:31,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:32,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:33,617 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:34,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:35,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:35,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:06:36,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:37,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:38,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:38,665 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:06:39,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:40,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:40,746 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:06:41,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:06:41,338 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:06:41,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:42,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:43,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:44,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:45,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:45,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:06:46,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:47,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:48,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:49,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:50,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:50,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:06:51,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:52,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:53,668 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:54,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:55,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:55,719 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:06:55,720 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:06:55,720 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:06:55,721 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:06:56,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:06:56,339 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:06:56,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:06:56,510 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:06:56,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:57,511 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:06:57,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:58,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:06:59,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:00,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:01,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:01,761 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:07:02,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:03,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:04,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:05,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:06,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:07,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:07,761 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:07:08,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:08,670 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:07:09,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:10,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:11,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:07:11,338 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:07:11,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:12,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:13,451 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:07:13,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:14,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:15,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:16,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:17,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:18,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:18,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:07:19,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:20,678 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:21,679 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:22,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:23,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:23,768 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:07:24,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:25,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:26,288 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:07:26,289 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:07:26,290 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:07:26,292 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:07:26,541 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:07:26,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:26,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:07:26,729 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:07:27,542 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:07:27,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:28,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:28,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:07:29,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:30,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:31,724 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:32,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:33,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:34,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:34,764 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:07:35,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:36,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:37,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:38,673 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:07:38,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:39,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:39,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:07:40,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:41,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:07:41,436 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:07:41,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:42,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:43,725 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:44,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:45,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:45,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:07:46,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:47,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:48,726 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:49,732 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:50,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:50,772 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:07:51,737 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:52,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:53,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:54,733 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:55,531 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:07:55,532 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:07:55,532 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:07:55,534 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:07:55,567 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:07:55,945 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:56,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:07:56,437 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:07:56,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:07:56,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:57,569 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:07:57,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:58,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:07:59,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:00,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:01,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:08:01,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:02,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:03,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:04,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:05,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:06,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:07,767 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:08:07,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:08,677 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:08:08,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:09,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:10,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:11,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:08:11,437 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:08:11,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:12,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:13,560 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:08:13,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:14,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:15,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:16,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:17,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:18,767 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:08:18,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:19,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:20,949 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:21,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:22,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:23,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:08:23,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:24,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:08:24,763 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:08:24,763 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:08:24,764 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:08:25,185 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:25,599 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:08:25,600 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:08:26,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:26,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:08:26,436 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:08:27,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:28,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:29,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:29,584 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:08:30,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:31,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:32,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:33,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:34,182 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:34,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:08:35,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:36,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:37,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:38,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:38,682 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:08:39,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:39,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:08:40,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:41,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:41,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:08:41,437 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:08:42,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:43,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:44,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:44,911 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:08:45,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:46,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:47,183 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:48,184 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:49,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:50,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:50,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:08:51,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:52,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:53,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:54,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:54,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:08:54,439 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:08:54,439 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:08:54,441 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:08:54,628 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:08:55,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:55,629 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:08:55,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:08:56,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:56,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:08:56,437 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:08:57,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:58,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:08:59,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:00,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:01,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:01,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:09:02,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:03,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:04,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:05,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:06,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:07,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:07,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:09:08,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:08,685 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:09:09,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:10,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:11,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:11,436 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:09:11,437 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:09:12,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:13,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:13,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:09:14,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:15,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:16,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:17,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:18,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:18,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:09:19,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:20,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:21,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:22,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:23,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:23,930 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:09:24,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:24,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:09:24,845 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:09:24,845 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:09:24,847 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:09:25,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:25,664 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:09:25,664 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:09:26,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:26,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:09:26,437 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:09:27,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:28,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:29,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:29,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:09:30,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:31,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:32,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:33,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:34,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:34,991 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:09:35,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:36,280 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:37,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:38,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:38,689 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:09:39,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:39,991 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:09:40,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:41,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:41,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:09:41,437 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:09:42,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:43,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:44,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:44,991 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:09:45,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:46,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:47,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:48,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:49,282 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:49,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:09:50,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:51,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:52,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:53,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:54,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:54,999 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:09:55,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:55,397 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:09:55,399 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:09:55,399 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:09:55,401 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:09:55,695 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:09:56,284 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:56,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:09:56,438 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:09:57,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:57,697 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:09:58,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:09:59,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:00,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:00,997 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:10:01,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:02,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:03,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:04,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:05,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:06,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:06,997 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:10:07,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:08,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:08,692 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:10:09,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:10,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:11,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:11,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:10:11,438 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:10:12,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:12,533 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:10:13,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:14,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:15,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:16,372 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:17,372 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:17,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:10:18,372 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:19,372 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:20,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:21,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:22,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:23,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:10:23,380 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:24,377 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:25,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:10:25,277 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:10:25,278 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:10:25,279 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:10:25,691 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:25,726 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:10:26,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:26,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:10:26,438 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:10:27,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:27,728 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:10:28,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:28,604 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:10:29,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:30,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:31,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:32,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:33,423 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:34,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:10:34,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:35,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:36,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:37,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:38,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:38,696 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:10:39,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:39,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:10:40,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:41,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:10:41,438 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:10:41,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:42,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:43,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:44,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:45,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:10:45,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:46,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:47,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:48,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:49,573 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:50,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:51,006 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:10:51,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:52,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:53,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:54,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:55,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:10:55,431 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:10:55,431 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:10:55,432 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:10:55,758 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:10:55,841 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:56,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:10:56,438 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:10:56,523 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:10:56,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:57,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:57,760 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:10:58,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:10:59,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:00,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:01,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:02,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:11:02,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:03,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:04,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:05,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:06,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:07,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:08,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:11:08,700 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:11:08,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:09,778 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:10,778 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:11,438 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:11:11,438 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:11:11,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:12,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:13,532 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:11:13,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:14,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:15,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:16,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:17,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:18,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:19,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:11:19,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:20,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:21,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:22,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:23,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:24,012 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:11:24,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:25,791 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:26,438 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:11:26,439 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:11:26,791 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:26,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:11:26,859 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:11:26,859 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:11:26,861 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:11:27,788 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:11:27,788 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:11:27,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:28,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:29,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:29,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:11:30,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:31,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:32,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:33,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:34,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:35,008 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:11:35,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:36,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:37,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:38,704 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:11:38,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:39,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:40,008 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:11:40,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:41,438 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:11:41,439 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:11:41,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:42,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:43,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:44,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:45,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:11:45,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:46,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:47,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:48,794 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:49,794 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:50,794 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:51,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:11:51,794 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:52,794 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:53,794 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:54,794 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:55,795 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:56,017 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:11:56,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:11:56,440 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:11:56,794 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:57,795 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:58,795 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:58,937 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:11:58,939 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:11:58,939 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:11:58,941 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:11:59,795 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:11:59,819 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:11:59,819 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:12:00,795 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:01,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:12:01,795 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:02,795 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:03,795 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:04,795 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:05,795 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:06,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:12:06,795 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:07,796 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:08,708 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:12:08,796 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:09,796 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:10,926 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:11,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:12:11,440 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:12:11,582 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:12:11,840 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:12,840 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:13,840 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:14,840 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:15,840 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:16,840 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:17,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:12:17,840 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:18,840 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:19,840 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:20,840 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:21,841 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:22,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:23,021 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:12:23,844 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:24,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:25,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:26,443 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:12:26,444 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:12:26,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:27,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:28,294 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:12:28,295 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:12:28,295 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:12:28,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:12:28,297 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:12:28,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:28,851 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:12:29,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:29,852 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:12:30,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:31,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:32,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:33,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:34,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:12:34,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:35,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:36,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:37,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:38,712 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:12:38,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:39,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:12:39,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:40,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:41,443 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:12:41,444 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:12:41,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:42,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:43,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:44,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:45,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:12:45,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:46,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:47,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:48,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:49,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:50,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:51,024 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:12:51,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:52,900 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:53,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:54,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:55,900 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:56,444 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:12:56,445 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:12:56,575 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:12:56,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:12:56,661 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:12:56,662 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:12:56,663 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:12:56,879 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:12:57,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:57,880 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:12:58,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:12:59,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:00,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:01,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:02,023 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:13:02,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:03,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:04,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:05,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:06,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:07,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:13:07,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:08,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:08,716 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:13:09,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:10,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:11,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:11,444 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:13:11,444 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:13:12,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:12,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:13:13,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:14,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:15,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:16,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:17,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:18,024 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:13:18,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:19,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:20,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:21,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:22,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:23,034 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:13:23,106 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:24,104 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:25,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:26,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:26,447 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:13:26,451 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:13:27,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:28,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:28,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:13:28,461 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:13:28,461 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:13:28,461 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:13:28,462 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:13:28,905 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:13:29,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:29,906 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:13:30,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:31,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:32,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:33,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:34,026 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:13:34,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:35,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:36,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:37,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:38,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:38,720 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:13:39,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:39,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:13:40,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:41,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:41,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:13:41,448 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:13:42,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:43,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:44,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:45,025 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:13:45,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:46,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:47,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:48,112 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:49,112 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:50,112 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:51,034 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:13:51,121 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:52,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:53,115 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:54,115 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:55,115 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:56,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:13:56,115 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:56,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:13:56,448 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:13:56,592 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:13:56,594 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:13:56,594 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:13:56,596 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:13:56,927 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:13:57,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:57,927 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:13:58,117 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:13:59,117 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:00,117 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:01,117 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:02,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:14:02,117 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:03,117 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:04,117 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:05,117 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:06,117 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:07,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:08,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:14:08,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:08,725 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:14:09,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:10,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:11,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:11,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:14:11,448 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:14:12,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:13,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:13,521 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:14:14,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:15,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:16,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:17,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:18,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:19,033 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:14:19,119 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:20,119 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:21,119 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:22,132 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:23,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:24,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:14:24,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:25,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:26,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:26,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:14:26,448 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:14:27,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:27,317 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:14:27,318 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:14:27,319 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:14:27,320 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:14:27,956 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:14:28,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:29,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:29,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:14:29,957 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:14:30,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:31,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:32,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:33,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:34,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:35,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:14:35,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:36,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:37,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:38,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:38,728 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:14:39,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:40,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:14:40,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:41,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:41,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:14:41,449 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:14:42,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:43,127 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:44,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:45,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:46,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:14:46,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:47,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:48,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:49,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:50,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:51,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:52,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:14:52,128 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:53,129 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:54,136 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:55,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:56,133 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:56,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:14:56,455 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:14:57,134 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:57,540 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:14:58,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:59,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:14:59,411 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:14:59,412 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:14:59,412 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:14:59,414 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:14:59,989 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:15:00,135 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:01,140 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:01,991 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:15:02,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:03,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:15:03,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:04,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:05,139 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:06,140 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:07,140 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:08,140 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:08,732 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:15:08,738 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:15:09,140 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:10,140 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:11,140 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:11,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:15:11,455 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:15:12,140 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:13,140 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:14,140 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:14,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:15:15,140 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:16,141 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:17,141 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:18,141 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:19,141 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:20,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:15:20,141 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:21,141 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:22,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:23,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:24,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:25,044 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:15:25,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:26,146 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:26,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:15:26,458 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:15:27,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:27,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:15:27,887 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:15:27,888 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:15:27,889 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:15:28,021 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:15:28,299 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:29,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:30,023 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:15:30,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:31,044 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:15:31,296 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:32,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:33,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:34,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:35,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:36,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:37,044 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:15:37,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:38,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:38,736 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:15:39,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:40,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:41,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:41,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:15:41,458 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:15:42,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:42,561 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:15:43,297 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:44,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:45,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:46,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:47,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:48,045 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:15:48,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:49,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:50,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:51,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:52,304 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:53,302 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:54,048 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:15:54,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:55,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:56,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:56,461 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:15:56,462 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:15:57,303 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:58,226 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:15:58,227 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:15:58,227 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:15:58,228 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:15:58,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:15:59,057 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:15:59,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:15:59,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:00,059 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:16:00,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:01,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:02,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:03,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:04,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:05,050 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:16:05,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:06,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:07,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:08,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:08,740 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:16:09,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:10,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:11,050 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:16:11,462 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:16:11,462 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:16:11,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:12,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:13,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:14,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:15,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:16,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:16:16,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:17,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:18,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:19,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:20,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:21,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:22,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:16:22,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:23,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:24,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:25,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:26,462 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:16:26,463 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:16:26,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:27,536 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:16:27,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:28,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:29,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:30,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:31,023 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:16:31,024 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:16:31,024 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:16:31,026 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:16:31,095 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:16:31,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:32,097 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:16:32,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:33,065 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:16:33,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:34,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:35,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:36,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:37,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:38,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:38,745 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:16:38,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:16:39,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:40,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:41,462 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:16:41,463 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:16:41,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:42,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:43,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:44,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:16:44,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:45,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:46,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:47,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:48,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:49,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:50,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:16:50,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:51,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:52,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:53,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:54,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:55,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:56,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:16:56,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:16:56,469 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:16:56,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:57,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:58,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:16:59,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:00,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:01,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:17:01,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:02,180 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:17:02,181 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:17:02,182 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:17:02,184 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:17:02,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:03,120 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:17:03,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:04,121 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:17:04,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:05,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:06,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:07,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:17:07,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:08,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:08,748 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:17:09,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:10,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:11,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:17:11,470 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:17:11,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:12,603 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:17:12,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:13,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:14,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:15,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:16,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:17,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:18,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:17:18,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:19,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:20,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:21,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:22,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:23,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:24,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:17:24,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:25,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:26,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:17:26,474 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:17:26,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:27,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:28,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:29,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:17:29,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:30,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:31,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:31,836 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:17:31,838 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:17:31,838 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:17:31,839 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:17:32,150 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:17:32,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:33,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:34,152 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:17:34,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:34,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:17:35,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:36,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:37,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:38,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:38,752 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:17:39,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:40,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:17:40,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:41,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:17:41,474 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:17:41,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:42,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:43,720 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:44,720 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:45,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:17:45,720 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:46,720 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:47,720 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:48,720 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:49,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:50,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:17:50,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:51,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:52,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:53,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:54,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:55,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:17:55,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:57,948 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:17:58,165 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:17:58,336 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:17:58,336 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:18:00,428 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:18:04,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:05,428 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:18:05,680 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:06,648 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:08,756 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:18:10,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:18:12,422 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:13,066 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:18:13,067 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:18:13,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:14,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:15,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:18:20,791 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:18:21,802 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:23,221 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:26,324 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:18:30,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:18:30,476 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:18:30,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:31,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:18:36,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:18:38,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:38,760 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:18:39,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:41,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:18:46,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:18:51,018 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:18:51,019 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:18:51,058 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:51,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:52,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:18:52,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:52,820 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:18:53,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:54,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:54,821 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:18:55,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:56,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:57,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:57,178 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:18:58,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:18:59,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:00,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:01,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:01,198 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:19:01,199 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:19:02,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:02,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:19:03,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:04,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:05,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:06,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:07,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:08,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:08,178 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:19:08,765 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:19:09,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:10,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:11,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:12,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:13,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:13,178 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:19:14,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:15,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:16,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:16,198 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:19:16,199 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:19:17,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:18,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:18,327 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:19:19,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:20,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:21,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:22,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:23,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:24,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:24,178 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:19:25,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:26,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:27,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:28,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:29,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:30,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:30,178 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:19:31,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:31,199 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:19:31,199 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:19:32,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:33,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:34,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:35,155 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:35,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:19:36,155 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:37,155 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:38,155 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:38,767 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:19:39,155 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:40,155 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:40,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:19:41,155 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:42,155 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:43,155 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:44,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:45,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:46,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:19:46,179 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:19:46,199 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: stop_status 2024-12-27 02:19:46,199 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: stop_status 2024-12-27 02:19:51,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:19:56,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:20:01,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:20:05,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 02:20:05,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:06,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:06,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:20:07,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:08,771 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:20:08,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:09,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:10,779 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:11,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:12,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:12,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:20:13,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:14,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:15,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:16,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:17,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:18,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:18,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:20:19,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:20,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:21,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:22,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:23,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:24,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:24,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:20:25,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:26,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:27,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:28,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:29,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:29,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:20:30,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:31,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:32,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:33,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:34,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:35,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:35,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:20:36,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:37,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:38,775 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:20:38,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:39,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:40,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:41,782 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:41,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:20:42,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:43,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:44,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:45,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:46,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:46,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:20:47,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:48,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:49,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:50,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:51,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:51,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:20:52,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:53,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:54,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:55,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:56,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:57,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:57,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:20:58,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:20:59,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:00,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:01,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:02,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:03,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:03,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:21:04,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:05,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:06,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:07,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:08,779 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:21:08,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:09,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:21:09,785 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:10,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:11,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:12,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:13,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:14,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:14,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:21:15,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:16,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:17,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:18,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:19,786 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:19,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:21:20,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:21,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:22,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:23,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:24,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:24,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:21:25,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:26,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:27,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:28,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:29,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:30,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:21:30,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:31,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:32,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:33,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:34,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:35,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:36,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:21:36,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:37,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:38,783 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:21:38,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:39,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:40,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:41,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:21:41,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:42,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:43,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:44,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:45,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:46,787 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:21:46,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:47,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:21:51,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:21:56,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:22:01,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:22:06,808 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:22:08,786 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:22:12,788 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:22:17,791 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:22:22,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:22:27,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:22:27,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 02:22:27,982 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:28,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:30,168 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:22:30,221 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:31,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:32,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:32,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:22:33,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:34,171 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:22:34,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:35,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:36,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:37,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:37,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:22:38,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:38,791 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:22:39,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:40,218 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:41,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:42,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:42,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:22:43,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:44,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:45,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:46,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:47,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:47,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:22:48,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:49,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:50,219 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:51,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:52,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:53,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:53,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:22:54,220 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:55,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:56,227 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:57,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:58,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:22:58,958 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:22:59,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:00,224 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:00,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:23:00,866 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:23:00,866 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:23:00,867 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:23:01,200 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:23:01,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:02,201 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:23:02,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:03,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:04,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:04,949 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:23:05,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:06,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:07,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:08,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:08,794 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:23:09,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:10,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:10,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:23:11,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:12,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:13,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:14,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:15,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:15,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:23:16,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:17,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:18,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:19,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:20,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:21,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:21,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:23:22,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:23,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:24,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:25,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:26,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:26,958 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:23:27,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:28,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:29,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:30,340 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:30,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:23:30,826 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:23:30,827 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:23:30,828 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:23:31,229 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:23:31,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:32,230 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:23:32,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:32,867 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:23:33,498 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:34,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:35,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:36,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:37,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:37,954 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:23:38,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:38,798 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:23:39,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:40,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:41,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:42,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:42,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:23:43,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:44,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:45,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:46,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:47,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:48,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:48,955 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:23:49,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:50,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:51,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:52,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:53,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:54,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:54,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:23:55,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:56,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:57,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:58,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:59,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:23:59,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:24:00,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:00,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:24:00,471 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:24:00,471 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:24:00,473 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:24:01,268 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:24:01,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:02,269 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:24:02,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:03,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:04,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:05,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:05,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:24:06,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:07,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:08,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:08,802 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:24:09,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:10,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:11,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:11,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:24:12,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:13,562 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:14,562 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:15,562 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:16,562 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:16,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:24:17,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:18,562 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:19,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:20,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:21,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:22,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:22,957 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:24:23,563 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:24,575 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:25,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:26,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:27,575 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:27,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:24:28,575 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:29,573 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:30,186 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:24:30,187 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:24:30,188 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:24:30,189 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:24:30,299 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:24:30,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:31,600 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:32,304 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:24:32,600 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:32,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:24:33,600 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:34,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:35,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:36,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:37,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:37,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:24:38,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:38,806 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:24:39,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:40,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:41,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:42,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:43,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:43,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:24:44,601 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:45,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:46,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:47,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:48,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:48,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:24:49,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:50,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:51,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:52,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:53,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:54,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:54,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:24:55,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:56,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:57,679 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:58,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:59,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:24:59,972 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:25:00,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:01,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:02,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:03,574 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:25:03,576 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:25:03,576 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:25:03,577 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:25:03,983 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:04,335 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:25:04,720 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:05,336 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:25:05,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:25:05,720 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:06,720 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:07,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:08,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:08,811 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:25:09,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:10,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:10,813 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:25:11,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:12,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:13,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:14,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:15,721 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:15,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:25:16,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:17,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:18,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:19,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:20,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:21,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:21,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:25:22,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:23,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:24,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:25,722 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:26,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:27,723 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:27,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:25:28,731 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:29,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:30,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:31,731 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:32,727 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:32,969 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:25:33,728 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:34,402 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:25:34,403 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:25:34,403 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:25:34,404 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:25:34,821 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:35,361 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:25:35,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:36,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:37,362 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:25:37,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:37,969 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:25:38,814 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:25:38,821 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:39,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:40,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:41,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:42,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:43,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:43,969 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:25:44,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:45,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:46,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:47,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:48,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:48,969 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:25:49,816 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:50,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:51,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:52,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:53,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:54,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:54,970 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:25:55,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:56,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:57,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:58,827 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:59,820 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:25:59,974 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:26:00,821 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:01,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:02,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:03,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:04,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:04,835 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:26:04,837 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:26:04,838 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:26:04,839 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:26:05,387 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:26:05,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:05,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:26:06,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:07,389 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:26:07,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:08,818 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:26:08,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:09,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:10,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:11,821 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:26:11,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:12,822 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:13,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:14,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:15,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:16,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:16,974 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:26:17,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:18,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:19,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:20,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:21,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:22,206 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:26:22,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:23,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:24,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:25,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:26,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:27,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:28,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:26:28,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:29,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:30,832 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:31,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:32,825 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:33,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:26:33,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:34,828 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:35,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:35,939 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:26:35,941 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:26:35,941 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:26:35,943 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:26:36,420 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:26:36,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:37,421 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:26:37,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:38,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:26:38,822 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:26:38,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:39,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:40,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:41,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:42,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:43,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:44,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:26:44,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:45,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:46,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:47,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:48,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:49,830 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:50,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:26:51,017 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:52,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:53,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:54,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:55,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:56,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:56,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:26:57,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:58,016 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:26:59,017 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:00,017 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:01,017 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:02,017 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:02,111 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:27:03,026 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:04,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:05,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:06,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:07,023 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:07,122 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:27:08,020 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:08,827 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:27:08,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:27:08,845 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:27:08,845 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:27:08,847 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:27:09,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:09,450 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:27:10,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:11,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:11,452 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:27:12,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:12,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:27:13,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:14,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:15,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:16,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:17,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:18,114 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:27:18,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:19,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:20,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:21,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:22,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:23,115 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:27:23,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:24,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:25,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:26,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:27,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:28,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:29,115 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:27:29,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:30,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:31,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:32,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:33,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:34,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:27:34,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:35,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:36,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:37,271 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:37,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:27:37,953 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:27:37,953 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:27:37,955 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:27:38,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:38,486 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:27:38,831 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:27:39,366 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:39,487 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:27:39,833 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:27:40,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:41,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:42,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:43,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:44,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:45,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:27:45,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:46,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:47,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:48,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:49,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:50,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:27:50,368 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:51,368 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:52,368 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:53,368 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:54,368 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:55,368 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:56,367 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:27:56,368 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:57,368 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:58,368 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:27:59,368 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:00,368 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:01,372 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:28:01,382 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:02,371 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:03,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:04,377 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:05,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:06,375 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:07,281 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:28:07,282 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:28:07,283 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:28:07,283 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:28:07,284 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:28:07,519 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:28:07,694 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:08,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:08,834 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:28:09,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:09,520 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:28:10,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:11,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:12,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:12,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:28:13,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:14,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:15,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:16,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:17,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:18,368 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:28:18,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:19,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:20,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:21,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:22,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:23,369 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:28:23,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:24,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:25,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:26,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:27,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:28,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:29,376 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:28:29,444 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:30,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:31,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:32,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:33,443 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:34,443 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:35,194 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:28:35,196 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:28:35,196 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:28:35,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:28:35,198 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:28:35,545 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:28:35,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:36,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:37,547 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:28:37,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:38,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:38,838 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:28:39,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:40,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:40,841 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:28:41,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:42,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:43,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:44,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:45,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:46,373 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:28:46,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:47,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:48,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:49,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:50,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:51,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:28:51,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:52,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:53,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:54,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:55,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:56,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:57,374 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:28:57,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:58,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:28:59,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:00,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:01,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:02,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:03,373 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:29:03,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:04,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:05,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:06,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:07,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:08,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:29:08,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:08,845 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:29:09,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:10,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:10,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:29:10,781 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:29:10,782 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:29:10,783 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:29:11,580 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:29:11,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:12,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:13,581 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:29:13,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:14,377 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:29:14,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:15,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:16,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:17,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:18,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:19,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:20,377 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:29:20,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:21,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:22,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:23,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:24,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:25,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:29:25,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:26,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:27,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:28,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:29,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:30,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:31,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:29:31,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:32,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:33,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:34,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:35,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:36,385 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:29:36,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:37,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:38,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:38,844 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:29:39,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:39,809 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:29:39,811 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:29:39,812 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:29:39,814 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:29:40,608 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:29:40,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:41,609 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:29:41,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:41,851 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:29:42,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:43,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:44,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:45,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:46,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:47,379 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:29:47,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:48,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:49,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:50,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:51,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:52,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:53,379 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:29:53,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:54,760 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:55,757 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:56,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:57,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:58,379 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:29:58,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:29:59,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:00,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:01,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:02,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:03,380 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:30:03,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:04,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:05,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:06,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:07,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:08,386 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:30:08,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:08,849 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:30:09,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:10,762 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:11,763 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:12,372 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:30:12,373 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:30:12,373 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:30:12,375 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:30:12,643 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:30:13,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:30:13,644 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:30:13,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:14,652 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:15,652 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:16,652 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:17,652 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:18,652 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:30:18,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:19,652 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:20,652 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:21,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:22,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:23,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:24,652 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:30:24,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:25,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:26,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:27,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:28,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:29,652 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:30:29,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:30,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:31,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:32,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:33,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:34,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:35,652 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:30:35,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:36,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:37,662 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:38,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:38,860 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:30:39,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:40,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:40,862 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:30:41,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:42,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:42,843 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:30:42,845 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:30:42,845 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:30:42,846 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:30:43,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:43,675 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:30:44,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:45,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:45,676 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:30:46,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:30:46,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:47,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:48,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:49,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:50,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:51,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:52,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:30:52,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:53,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:54,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:55,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:56,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:57,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:30:57,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:58,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:30:59,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:00,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:01,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:02,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:03,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:31:03,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:04,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:05,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:06,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:07,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:08,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:08,859 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:31:08,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:31:09,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:10,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:11,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:12,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:13,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:14,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:31:14,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:15,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:16,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:16,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:31:16,785 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:31:16,785 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:31:16,788 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:31:17,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:17,707 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:31:18,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:19,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:19,709 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:31:20,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:31:20,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:21,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:22,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:23,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:24,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:25,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:26,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:31:26,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:27,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:28,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:29,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:30,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:31,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:32,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:31:32,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:33,673 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:34,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:35,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:36,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:37,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:31:37,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:38,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:38,863 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:31:39,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:40,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:41,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:42,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:42,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:31:43,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:44,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:45,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:46,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:47,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:47,860 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:31:47,861 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:31:47,862 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:31:47,867 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:31:47,901 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:31:48,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:48,738 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:31:49,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:49,738 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:31:50,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:51,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:52,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:53,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:31:53,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:54,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:55,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:56,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:57,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:58,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:31:59,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:31:59,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:00,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:01,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:02,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:03,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:04,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:32:04,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:05,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:06,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:07,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:08,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:08,867 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:32:09,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:09,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:32:10,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:11,686 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:12,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:13,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:14,685 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:15,681 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:32:15,683 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:15,859 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:32:15,861 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:32:15,861 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:32:15,863 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:32:16,684 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:16,770 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:32:17,684 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:17,770 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:32:18,684 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:19,684 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:20,684 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:21,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:32:21,684 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:22,684 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:23,684 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:24,684 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:25,684 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:26,685 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:27,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:32:27,685 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:28,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:29,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:30,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:31,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:32,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:33,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:32:33,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:34,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:35,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:36,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:37,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:38,676 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:32:38,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:38,871 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:32:39,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:40,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:41,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:42,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:43,684 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:32:43,754 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:44,757 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:45,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:46,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:47,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:47,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:32:47,887 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:32:47,887 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:32:47,889 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:32:48,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:48,803 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:32:49,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:32:49,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:49,804 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:32:50,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:51,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:52,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:53,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:54,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:55,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:32:55,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:56,758 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:57,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:58,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:32:59,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:00,894 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:01,678 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:33:01,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:02,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:03,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:04,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:05,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:06,678 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:33:06,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:07,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:08,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:08,875 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:33:09,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:10,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:11,798 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:11,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:33:12,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:13,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:14,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:15,811 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:16,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:17,690 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:33:17,810 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:18,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:33:18,484 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:33:18,484 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:33:18,486 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:33:18,835 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:33:18,945 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:19,836 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:33:19,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:20,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:21,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:22,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:22,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:33:23,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:24,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:25,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:26,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:27,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:28,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:33:28,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:29,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:30,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:31,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:32,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:33,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:33:33,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:34,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:35,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:36,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:37,943 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:38,879 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:33:38,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:39,942 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:33:39,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:40,944 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:41,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:42,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:43,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:44,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:44,948 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:33:45,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:46,951 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:47,741 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:33:47,743 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:33:47,743 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:33:47,745 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:33:47,884 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:33:48,153 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:49,149 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:49,885 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:33:50,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:50,945 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:33:51,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:52,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:53,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:54,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:55,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:55,946 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:33:56,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:57,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:58,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:33:59,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:00,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:01,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:01,946 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:34:02,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:03,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:04,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:05,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:06,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:07,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:07,946 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:34:08,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:08,883 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:34:09,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:10,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:11,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:12,152 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:12,947 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:34:13,159 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:14,154 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:15,160 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:16,155 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:17,163 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:17,959 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:34:18,158 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:19,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:19,246 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:34:19,247 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:34:19,248 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:34:19,249 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:34:19,917 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:34:20,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:21,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:21,919 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:34:22,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:23,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:23,287 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:34:24,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:25,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:26,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:27,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:28,156 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:28,949 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:34:29,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:30,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:31,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:32,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:33,321 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:33,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:34:34,228 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:35,228 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:36,228 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:37,228 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:38,228 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:38,887 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:34:39,228 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:39,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:34:40,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:41,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:42,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:43,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:44,229 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:44,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:34:45,231 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:46,232 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:47,232 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:48,233 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:49,232 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:49,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:34:50,233 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:50,298 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:34:50,299 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:34:50,300 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:34:50,301 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:34:50,953 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:34:51,233 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:51,954 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:34:52,233 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:53,233 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:54,233 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:55,233 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:55,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:34:56,233 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:57,233 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:58,233 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:34:59,234 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:00,234 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:01,234 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:01,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:35:02,234 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:03,234 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:04,234 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:05,234 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:06,234 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:06,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:35:07,234 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:08,234 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:08,891 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:35:09,234 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:10,235 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:11,235 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:12,235 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:12,956 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:35:13,235 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:14,235 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:15,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:16,241 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:17,242 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:17,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:35:18,242 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:19,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:20,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:20,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:35:20,432 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:35:20,432 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:35:20,433 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:35:20,986 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:35:21,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:21,987 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:35:22,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:23,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:23,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:35:24,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:25,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:26,243 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:27,244 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:28,244 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:28,959 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:35:29,244 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:30,244 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:31,244 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:32,244 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:33,244 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:33,960 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:35:34,244 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:35,245 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:36,244 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:37,245 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:38,245 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:38,895 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:35:39,245 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:39,959 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:35:40,245 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:41,245 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:42,245 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:43,245 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:44,250 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:44,973 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:35:45,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:46,252 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:47,259 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:48,254 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:49,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:49,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:35:49,436 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:35:49,436 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:35:49,438 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:35:50,016 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:35:50,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:50,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:35:51,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:52,017 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:35:52,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:53,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:54,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:55,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:55,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:35:56,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:57,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:58,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:35:59,255 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:00,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:01,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:01,966 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:36:02,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:03,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:04,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:05,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:06,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:07,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:36:07,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:08,256 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:08,899 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:36:09,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:10,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:11,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:12,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:13,021 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:36:13,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:14,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:15,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:16,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:17,259 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:18,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:36:18,259 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:19,259 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:20,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:21,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:21,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:36:21,648 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:36:21,648 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:36:21,649 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:36:22,050 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:36:22,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:23,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:23,688 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:36:24,052 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:36:24,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:25,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:26,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:27,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:28,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:29,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:36:29,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:30,260 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:31,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:32,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:33,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:34,028 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:36:34,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:35,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:36,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:37,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:38,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:38,900 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:36:39,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:40,027 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:36:40,261 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:41,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:42,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:43,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:44,262 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:45,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:36:45,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:46,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:47,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:48,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:49,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:50,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:50,396 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:36:50,397 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:36:50,398 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:36:50,398 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:36:50,399 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:36:51,083 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:36:51,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:52,084 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:36:52,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:53,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:54,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:55,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:56,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:36:56,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:57,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:58,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:36:59,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:00,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:01,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:02,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:37:02,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:03,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:04,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:05,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:06,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:07,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:37:07,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:08,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:08,904 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:37:09,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:10,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:11,266 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:12,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:13,032 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:37:13,267 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:14,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:15,278 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:16,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:17,279 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:18,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:37:18,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:19,223 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:37:19,225 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:37:19,226 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:37:19,227 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:37:19,635 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:20,110 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:37:20,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:21,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:22,112 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:37:22,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:23,265 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:37:23,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:24,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:25,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:26,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:27,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:28,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:29,034 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:37:29,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:30,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:31,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:32,378 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:33,379 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:34,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:37:34,379 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:35,379 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:36,379 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:37,379 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:38,379 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:38,909 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:37:39,379 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:40,035 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:37:40,379 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:41,379 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:42,379 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:43,380 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:44,388 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:45,036 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:37:45,381 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:46,381 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:47,388 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:48,382 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:49,384 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:50,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:37:50,120 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:37:50,121 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:37:50,122 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:37:50,123 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:37:50,140 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:37:50,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:51,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:52,142 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:37:52,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:53,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:54,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:55,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:37:55,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:56,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:57,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:58,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:37:59,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:00,537 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:01,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:38:01,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:02,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:03,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:04,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:05,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:06,037 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:38:06,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:07,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:08,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:08,913 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:38:09,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:10,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:11,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:12,046 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:38:12,575 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:13,581 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:14,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:15,582 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:16,580 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:17,046 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:38:17,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:38:17,475 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:38:17,475 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:38:17,477 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:38:17,907 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:18,171 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:38:18,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:19,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:20,173 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:38:20,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:21,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:22,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:23,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:38:23,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:24,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:25,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:26,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:27,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:28,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:29,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:38:29,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:30,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:31,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:32,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:33,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:34,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:38:34,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:35,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:36,623 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:37,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:38,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:38,917 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:38:39,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:40,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:38:40,623 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:41,623 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:42,623 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:43,623 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:44,623 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:45,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:38:45,630 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:46,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:47,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:48,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:49,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:49,953 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:38:49,955 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:38:49,955 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:38:49,956 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:38:50,207 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:38:50,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:50,994 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:38:51,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:52,208 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:38:52,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:53,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:54,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:55,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:56,047 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:38:56,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:57,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:58,626 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:38:59,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:00,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:01,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:02,048 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:39:02,797 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:03,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:04,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:05,702 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:06,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:07,048 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:39:07,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:08,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:08,921 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:39:09,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:10,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:11,703 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:12,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:13,049 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:39:13,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:14,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:15,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:16,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:17,711 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:18,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:39:18,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:39:18,442 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:39:18,443 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:39:18,444 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:39:18,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:19,237 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:39:19,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:20,238 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:39:20,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:21,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:22,845 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:23,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:24,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:39:24,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:25,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:26,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:27,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:28,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:29,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:30,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:39:30,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:31,846 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:32,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:33,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:34,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:35,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:39:35,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:36,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:37,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:38,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:38,925 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:39:39,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:40,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:41,052 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:39:41,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:42,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:43,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:44,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:45,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:46,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:39:46,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:47,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:48,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:49,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:50,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:39:50,011 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:39:50,011 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:39:50,012 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:39:50,269 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:39:50,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:51,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:52,051 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:39:52,270 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:39:52,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:53,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:54,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:55,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:56,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:57,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:39:57,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:58,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:39:59,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:00,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:01,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:02,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:03,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:40:03,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:04,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:05,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:06,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:07,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:08,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:08,928 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:40:08,934 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:40:09,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:10,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:11,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:12,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:13,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:14,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:40:14,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:15,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:16,890 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:17,893 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:18,899 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:19,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:40:19,896 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:20,896 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:21,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:22,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:23,058 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:40:23,059 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:40:23,060 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:40:23,061 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:40:23,305 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:40:23,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:24,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:40:24,306 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:40:24,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:25,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:26,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:27,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:28,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:29,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:30,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:40:30,898 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:31,898 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:32,898 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:33,898 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:35,056 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:36,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:36,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:40:37,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:38,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:38,931 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:40:39,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:40,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:41,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:41,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:40:42,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:43,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:44,053 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:45,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:46,054 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:46,060 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:40:47,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:48,058 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:49,058 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:50,058 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:51,062 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:51,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:40:52,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:52,437 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:40:52,438 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:40:52,438 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:40:52,440 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:40:53,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:53,331 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:40:54,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:54,332 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:40:55,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:56,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:57,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:40:57,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:58,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:40:59,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:00,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:01,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:02,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:03,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:41:03,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:04,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:05,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:06,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:07,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:08,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:41:08,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:08,936 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:41:09,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:10,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:11,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:12,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:13,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:14,061 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:41:14,064 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:15,065 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:16,073 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:17,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:18,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:19,070 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:41:19,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:20,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:21,071 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:21,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:41:21,819 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:41:21,819 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:41:21,820 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:41:22,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:22,357 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:41:23,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:24,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:24,359 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:41:25,065 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:41:25,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:26,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:27,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:28,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:29,268 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:30,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:31,065 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:41:31,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:32,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:33,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:34,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:35,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:36,066 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:41:36,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:37,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:38,269 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:38,941 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:41:39,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:40,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:41,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:42,066 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:41:42,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:43,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:44,270 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:45,272 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:46,277 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:47,069 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:41:47,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:48,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:49,273 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:50,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:50,631 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:41:50,631 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:41:50,632 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:41:50,634 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:41:51,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:51,390 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:41:52,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:52,390 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:41:52,672 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:41:53,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:54,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:55,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:56,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:57,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:58,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:41:58,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:41:59,274 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:00,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:01,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:02,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:03,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:04,068 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:42:04,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:05,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:06,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:07,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:08,275 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:08,944 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:42:09,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:10,067 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:42:10,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:11,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:12,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:13,276 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:14,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:15,082 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:42:15,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:16,286 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:17,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:18,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:19,285 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:19,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:42:19,869 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:42:19,869 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:42:19,871 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:42:20,292 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:20,418 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:42:20,910 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:42:21,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:22,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:22,419 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:42:23,290 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:24,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:25,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:26,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:42:26,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:27,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:28,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:29,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:30,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:31,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:32,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:42:32,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:33,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:34,291 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:35,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:36,334 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:37,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:42:37,334 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:38,334 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:38,948 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:42:39,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:40,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:41,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:42,072 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:42:42,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:43,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:44,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:45,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:46,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:47,335 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:48,079 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:42:48,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:49,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:50,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:51,345 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:52,344 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:53,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:53,594 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:42:53,595 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:42:53,596 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:42:53,596 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:42:53,597 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:42:54,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:54,445 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:42:55,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:56,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:56,447 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:42:57,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:58,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:42:59,073 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:42:59,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:00,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:01,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:02,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:03,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:04,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:05,074 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:43:05,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:06,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:07,412 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:08,412 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:08,952 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:43:09,412 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:10,074 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:43:10,412 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:11,412 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:12,412 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:13,412 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:14,412 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:15,074 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:43:15,412 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:16,412 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:17,413 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:18,420 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:19,415 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:20,077 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:43:20,417 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:21,422 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:22,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:23,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:24,110 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:43:24,111 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:43:24,112 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:43:24,113 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:43:24,481 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:43:24,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:25,151 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:43:25,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:26,483 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:43:26,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:27,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:28,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:29,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:30,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:31,076 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:43:31,513 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:32,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:33,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:34,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:35,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:36,076 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:43:36,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:37,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:38,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:38,956 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:43:39,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:40,514 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:41,077 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:43:41,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:42,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:43,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:44,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:45,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:46,077 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:43:46,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:47,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:48,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:49,515 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:50,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:51,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:43:51,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:52,523 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:53,521 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:54,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:55,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:55,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:43:55,824 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:43:55,824 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:43:55,826 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:43:56,511 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:43:56,511 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:43:56,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:56,864 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:43:57,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:58,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:43:59,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:00,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:01,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:02,082 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:44:02,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:03,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:04,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:05,523 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:06,523 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:07,741 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:08,083 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:44:08,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:08,961 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:44:09,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:10,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:11,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:12,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:13,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:14,083 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:44:14,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:15,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:16,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:17,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:18,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:19,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:20,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:44:20,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:21,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:22,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:23,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:24,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:25,092 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:44:25,093 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:44:25,094 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:44:25,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:44:25,096 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:44:25,536 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:44:25,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:26,537 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:44:26,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:27,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:28,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:29,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:30,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:31,086 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:44:31,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:32,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:33,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:34,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:35,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:36,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:37,086 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:44:37,655 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:38,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:38,964 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:44:39,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:40,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:41,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:42,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:43,087 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:44:43,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:44,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:45,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:46,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:47,656 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:48,087 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:44:48,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:49,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:50,663 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:51,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:52,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:53,088 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:44:53,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:54,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:55,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:56,059 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:44:56,060 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:44:56,061 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:44:56,063 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:44:56,562 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:44:56,563 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:44:56,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:57,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:58,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:44:58,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:44:59,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:00,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:01,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:02,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:03,658 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:04,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:45:04,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:05,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:06,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:07,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:08,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:08,968 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:45:09,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:10,090 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:45:10,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:11,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:12,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:13,659 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:14,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:15,091 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:45:15,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:16,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:17,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:18,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:19,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:20,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:21,091 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:45:21,660 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:22,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:23,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:24,664 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:25,665 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:26,098 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:45:26,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:27,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:28,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:45:28,402 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:45:28,403 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:45:28,404 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:45:28,593 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:45:28,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:29,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:30,595 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:45:30,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:31,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:32,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:45:32,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:33,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:34,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:35,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:36,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:37,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:38,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:45:38,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:38,973 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:45:39,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:40,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:41,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:43,006 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:43,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:45:44,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:45,002 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:46,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:47,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:48,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:48,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:45:49,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:50,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:51,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:52,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:53,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:54,003 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:54,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:45:55,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:56,011 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:57,008 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:58,006 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:59,012 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:45:59,102 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:46:00,012 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:01,009 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:01,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:46:01,819 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:46:01,819 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:46:01,820 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:46:02,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:02,620 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:46:02,621 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:46:03,236 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:04,236 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:05,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:46:05,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:06,236 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:07,236 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:08,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:08,976 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:46:09,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:10,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:11,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:46:11,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:12,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:13,237 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:14,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:15,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:16,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:46:16,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:17,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:18,454 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:19,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:20,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:21,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:46:21,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:22,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:23,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:24,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:25,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:26,466 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:27,103 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:46:27,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:28,466 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:29,466 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:30,466 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:31,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:31,593 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:46:31,594 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:46:31,594 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:46:31,596 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:46:31,645 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:46:32,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:32,635 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:46:32,646 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:46:33,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:34,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:35,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:36,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:37,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:38,105 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:46:38,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:38,980 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:46:39,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:40,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:41,467 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:42,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:43,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:44,104 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:46:44,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:45,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:46,470 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:47,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:48,468 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:49,105 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:46:49,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:50,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:51,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:52,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:53,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:54,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:55,105 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:46:55,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:56,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:57,469 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:58,471 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:46:59,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:00,106 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:47:00,472 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:01,472 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:02,472 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:02,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:47:02,640 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:47:02,641 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:47:02,643 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:47:02,672 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:47:03,472 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:04,472 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:04,674 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:47:05,107 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:47:05,472 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:06,472 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:07,472 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:08,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:08,986 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:47:09,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:10,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:11,107 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:47:11,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:12,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:13,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:14,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:15,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:16,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:17,107 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:47:17,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:18,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:19,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:20,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:21,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:22,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:23,107 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:47:23,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:24,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:25,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:26,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:27,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:28,113 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:47:28,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:29,481 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:30,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:31,481 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:32,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:33,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:47:33,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:33,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:47:33,550 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:47:33,550 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:47:33,551 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:47:33,700 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:47:34,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:34,701 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:47:35,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:36,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:37,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:38,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:38,989 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:47:38,995 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:47:39,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:40,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:41,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:42,482 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:43,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:44,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:47:44,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:45,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:46,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:47,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:48,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:49,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:47:49,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:50,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:51,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:52,483 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:53,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:54,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:55,118 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:47:55,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:56,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:57,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:58,495 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:47:59,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:00,123 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:48:00,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:01,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:02,107 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:48:02,108 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:48:02,109 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:48:02,110 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:48:02,522 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:02,731 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:48:02,732 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:48:03,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:04,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:05,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:48:05,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:06,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:07,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:08,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:08,993 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:48:09,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:10,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:11,125 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:48:11,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:12,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:13,520 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:14,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:15,565 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:16,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:48:16,564 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:17,565 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:18,565 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:19,565 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:20,565 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:21,565 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:22,126 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:48:22,565 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:23,565 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:24,565 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:25,565 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:26,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:27,129 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:48:27,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:28,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:29,568 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:30,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:31,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:31,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:48:31,890 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:48:31,890 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:48:31,891 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:48:32,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:32,754 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:48:32,755 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:48:32,931 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:48:33,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:34,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:35,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:36,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:37,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:38,131 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:48:38,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:38,997 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:48:39,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:40,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:41,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:42,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:43,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:44,130 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:48:44,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:45,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:46,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:47,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:48,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:49,131 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:48:49,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:50,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:51,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:52,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:53,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:54,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:55,131 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:48:55,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:56,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:57,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:58,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:48:59,585 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:00,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:49:00,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:01,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:02,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:03,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:03,667 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:49:03,669 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:49:03,669 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:49:03,670 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:49:03,783 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:49:04,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:04,784 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:49:05,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:06,138 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:49:06,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:07,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:08,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:08,999 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:49:09,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:10,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:11,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:12,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:49:12,578 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:13,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:14,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:15,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:16,579 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:17,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:18,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:49:18,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:19,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:20,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:21,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:22,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:23,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:24,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:49:24,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:25,610 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:26,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:27,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:28,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:29,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:49:29,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:30,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:31,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:32,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:33,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:34,150 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:49:34,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:35,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:36,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:37,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:49:37,110 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:49:37,110 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:49:37,111 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:49:37,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:37,816 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:49:38,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:38,817 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:49:39,004 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:49:39,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:40,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:49:40,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:41,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:42,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:43,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:44,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:45,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:46,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:49:46,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:47,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:48,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:49,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:50,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:51,143 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:49:51,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:52,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:53,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:54,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:55,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:56,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:49:56,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:57,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:58,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:49:59,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:00,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:01,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:02,144 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:50:02,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:03,627 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:04,628 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:05,629 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:06,629 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:07,145 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:50:07,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:08,637 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:08,909 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:50:08,910 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:50:08,910 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:50:08,912 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:50:09,008 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:50:09,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:09,843 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:50:10,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:10,844 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:50:11,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:12,331 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:50:12,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:13,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:14,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:15,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:16,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:17,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:18,328 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:50:18,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:19,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:20,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:21,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:22,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:23,328 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:50:23,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:24,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:25,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:26,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:27,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:28,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:29,328 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:50:29,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:30,649 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:31,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:32,643 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:33,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:34,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:50:34,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:35,647 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:36,328 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:50:36,329 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:50:36,330 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:50:36,331 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:50:36,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:36,868 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:50:37,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:37,869 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:50:38,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:39,011 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:50:39,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:50:39,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:40,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:41,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:42,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:43,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:44,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:45,332 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:50:45,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:46,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:47,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:48,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:49,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:50,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:50:50,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:51,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:52,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:53,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:54,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:55,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:56,333 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:50:56,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:57,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:58,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:50:59,743 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:00,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:01,339 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:51:01,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:02,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:03,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:04,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:05,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:06,341 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:51:06,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:51:06,432 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:51:06,432 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:51:06,433 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:51:06,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:06,894 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:51:07,836 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:07,895 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:51:08,836 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:09,015 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:51:09,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:10,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:11,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:12,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:51:12,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:13,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:14,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:15,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:16,837 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:17,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:18,337 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:51:18,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:19,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:20,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:21,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:22,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:23,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:51:23,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:24,838 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:25,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:26,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:27,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:28,338 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:51:28,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:29,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:30,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:31,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:32,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:33,342 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:51:33,839 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:34,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:35,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:36,847 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:37,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:38,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:51:38,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:38,916 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:51:38,917 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:51:38,917 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:51:38,919 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:51:38,921 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:51:39,019 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:51:39,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:40,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:41,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:41,924 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:51:42,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:43,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:44,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:51:44,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:45,848 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:46,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:47,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:48,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:49,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:50,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:51:50,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:51,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:52,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:53,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:54,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:55,344 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:51:55,849 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:56,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:57,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:58,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:51:59,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:00,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:01,344 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:52:01,850 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:02,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:03,855 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:04,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:05,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:06,357 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:52:06,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:07,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:08,263 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:52:08,264 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:52:08,265 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:52:08,266 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:52:08,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:08,947 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:52:09,023 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:52:09,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:09,948 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:52:10,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:11,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:12,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:52:12,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:13,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:14,852 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:15,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:16,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:17,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:18,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:52:18,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:19,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:20,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:21,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:22,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:23,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:24,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:52:24,853 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:25,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:26,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:27,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:28,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:29,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:30,347 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:52:30,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:31,854 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:32,866 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:33,858 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:34,865 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:35,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:52:35,866 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:36,866 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:37,867 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:38,045 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:52:38,047 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:52:38,047 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:52:38,048 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:52:38,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:38,977 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:52:39,028 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:52:39,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:39,978 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:52:40,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:41,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:52:41,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:42,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:43,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:44,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:45,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:46,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:47,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:52:47,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:48,868 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:49,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:50,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:51,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:52,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:53,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:52:53,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:54,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:55,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:56,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:57,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:58,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:52:58,869 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:52:59,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:00,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:01,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:02,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:03,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:04,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:53:04,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:05,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:06,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:07,875 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:08,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:09,031 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:53:09,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:10,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:53:10,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:53:10,452 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:53:10,452 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:53:10,454 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:53:10,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:11,008 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:53:11,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:12,009 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:53:12,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:13,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:14,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:15,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:53:15,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:16,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:17,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:18,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:19,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:20,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:21,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:53:21,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:22,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:23,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:24,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:25,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:26,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:53:26,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:27,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:28,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:29,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:30,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:31,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:53:31,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:32,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:33,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:34,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:35,889 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:36,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:53:36,896 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:37,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:38,895 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:39,033 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:53:39,896 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:40,896 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:41,165 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:53:41,166 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:53:41,166 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:53:41,168 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:53:42,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:42,036 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:53:42,207 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:53:43,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:44,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:44,037 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:53:45,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:46,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:47,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:47,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:53:48,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:49,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:50,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:51,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:52,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:53,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:53,351 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:53:54,004 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:55,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:56,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:57,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:58,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:53:58,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:53:59,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:00,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:01,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:02,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:03,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:04,005 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:04,352 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:54:05,006 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:06,006 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:07,019 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:08,007 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:09,007 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:09,037 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:54:09,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:54:10,008 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:11,008 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:12,008 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:12,866 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:54:12,867 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:54:12,867 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:54:12,868 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:54:13,059 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:54:13,399 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:14,014 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:14,060 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:54:15,014 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:15,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:54:16,014 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:17,014 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:18,014 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:19,014 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:20,014 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:21,014 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:21,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:54:22,015 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:23,015 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:24,015 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:25,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:26,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:26,353 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:54:27,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:28,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:29,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:30,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:31,038 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:31,354 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:54:32,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:33,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:34,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:35,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:36,039 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:37,042 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:37,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:54:38,042 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:39,046 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:54:39,047 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:40,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:41,044 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:42,044 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:42,370 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:54:42,946 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:54:42,947 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:54:42,947 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:54:42,949 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:54:43,085 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:54:43,415 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:44,086 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:54:44,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:45,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:46,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:47,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:48,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:48,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:54:49,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:50,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:51,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:52,099 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:53,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:54,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:54,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:54:55,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:56,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:57,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:58,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:59,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:54:59,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:55:00,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:01,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:02,100 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:03,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:04,101 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:05,108 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:05,366 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:55:06,104 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:07,109 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:08,115 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:09,047 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:55:09,115 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:10,115 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:10,366 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:55:10,925 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:55:10,927 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:55:10,927 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:55:10,931 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:55:11,108 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:55:11,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:12,109 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:55:12,612 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:13,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:14,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:15,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:16,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:55:16,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:17,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:18,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:19,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:20,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:21,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:22,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:55:22,613 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:23,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:24,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:25,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:26,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:27,389 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:55:27,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:28,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:29,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:30,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:31,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:32,614 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:33,388 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:55:33,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:34,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:35,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:36,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:37,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:38,394 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:55:38,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:39,051 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:55:39,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:40,619 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:41,620 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:41,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:55:41,777 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:55:41,777 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:55:41,781 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:55:42,137 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:55:42,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:43,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:43,817 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:55:44,139 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:55:44,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:45,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:46,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:47,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:48,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:49,392 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:55:49,805 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:50,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:51,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:52,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:53,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:54,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:55,392 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:55:55,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:56,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:57,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:58,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:55:59,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:00,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:56:00,806 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:01,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:02,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:03,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:04,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:05,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:06,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:56:06,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:07,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:08,807 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:09,056 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:56:09,820 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:10,813 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:11,399 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:56:11,818 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:12,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:13,826 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:14,829 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:15,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:15,950 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:56:15,951 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:56:15,951 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:56:15,954 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:56:16,165 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:56:16,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:16,992 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:56:17,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:18,166 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:56:18,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:19,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:20,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:21,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:22,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:56:22,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:23,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:24,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:25,823 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:26,824 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:27,979 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:28,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:56:28,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:29,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:30,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:31,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:32,873 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:33,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:34,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:56:34,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:35,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:36,874 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:37,884 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:38,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:39,065 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:56:39,403 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:56:39,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:40,880 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:41,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:42,896 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:43,479 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:56:43,481 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:56:43,482 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:56:43,483 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:56:43,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:44,199 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:56:44,896 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:45,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:56:45,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:46,200 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:56:46,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:47,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:48,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:49,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:50,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:51,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:56:51,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:52,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:53,897 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:55,010 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:55,913 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:56,400 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:56:56,913 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:57,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:58,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:56:59,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:00,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:01,401 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:57:01,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:02,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:03,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:04,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:05,914 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:06,918 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:07,407 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:57:07,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:08,915 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:09,066 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:57:09,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:10,920 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:11,921 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:12,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:57:12,452 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:57:12,452 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:57:12,452 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:57:12,453 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:57:13,225 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:57:13,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:14,226 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:57:14,345 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:15,345 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:16,345 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:17,345 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:18,345 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:18,405 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:57:19,345 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:20,345 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:21,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:22,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:23,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:24,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:24,404 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:57:25,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:26,346 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:27,581 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:28,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:29,405 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:57:29,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:30,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:31,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:32,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:33,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:34,405 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:57:34,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:35,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:36,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:37,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:38,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:39,073 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:57:39,410 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:57:39,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:40,494 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:41,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:42,492 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:43,463 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:57:43,465 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:57:43,465 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:57:43,467 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:57:43,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:44,256 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:57:44,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:45,407 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:57:45,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:46,258 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:57:46,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:47,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:48,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:49,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:50,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:51,407 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:57:51,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:52,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:53,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:54,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:55,870 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:56,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:57,408 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:57:57,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:58,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:57:59,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:00,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:01,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:02,408 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:58:02,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:03,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:04,871 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:05,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:06,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:07,408 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:58:07,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:08,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:09,080 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:58:09,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:10,872 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:11,876 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:12,429 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:58:12,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:13,881 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:14,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:15,137 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:58:15,138 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:58:15,139 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:58:15,140 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:58:15,283 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:58:15,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:16,284 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:58:16,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:17,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:18,179 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:58:18,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:19,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:20,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:21,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:22,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:23,411 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:58:23,877 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:24,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:25,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:26,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:27,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:28,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:29,411 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:58:29,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:30,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:31,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:32,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:33,878 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:34,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:35,411 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:58:35,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:36,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:37,879 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:38,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:39,090 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:58:39,884 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:40,419 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:58:40,884 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:41,884 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:42,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:43,885 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:44,815 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:58:44,817 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:58:44,817 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:58:44,819 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:58:45,233 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:45,311 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:58:45,856 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:58:45,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:46,312 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:58:46,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:47,962 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:48,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:49,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:50,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:51,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:58:51,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:52,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:53,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:54,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:55,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:56,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:57,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:58:57,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:58,963 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:58:59,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:00,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:01,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:02,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:03,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:59:03,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:04,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:05,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:06,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:07,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:08,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:59:08,964 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:09,083 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:59:09,965 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:10,965 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:11,965 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:12,965 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:13,969 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:14,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:59:14,967 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:15,971 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:16,970 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:17,970 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:18,971 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:19,781 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:59:19,783 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:59:19,784 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:59:19,784 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:59:19,786 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:59:20,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:20,353 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:59:21,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:22,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:22,354 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:59:23,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:24,188 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:25,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:25,414 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:59:26,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:27,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:28,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:29,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:30,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:30,415 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:59:31,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:32,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:33,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:34,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:35,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:35,638 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:59:36,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:37,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:38,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:39,087 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 02:59:39,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:40,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:41,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:41,548 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:59:42,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:43,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:44,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:45,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:46,200 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:46,550 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:59:47,201 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:48,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:49,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:50,199 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:51,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:51,484 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 02:59:51,485 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 02:59:51,485 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 02:59:51,487 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 02:59:52,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:52,385 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 02:59:52,385 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 02:59:52,526 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:59:53,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:54,198 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:55,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:56,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:57,195 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:57,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 02:59:58,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 02:59:59,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:00,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:01,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:02,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:03,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:03,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:00:04,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:05,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:06,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:07,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:08,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:08,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:00:09,091 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:00:09,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:10,196 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:11,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:12,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:13,197 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:13,555 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:00:14,206 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:15,200 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:16,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:17,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:18,214 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:18,565 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:00:19,203 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:19,949 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 03:00:19,951 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 03:00:19,951 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:00:19,952 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:00:20,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:20,408 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:00:21,364 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:22,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:22,409 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:00:23,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:23,992 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:00:24,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:25,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:26,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:27,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:28,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:29,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:29,560 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:00:30,359 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:31,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:32,360 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:33,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:34,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:34,561 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:00:35,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:36,424 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:37,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:38,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:39,096 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:00:39,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:40,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:40,560 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:00:41,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:42,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:43,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:44,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:45,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:46,426 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:46,565 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:00:47,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:48,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:49,426 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:50,429 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:51,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:51,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:00:52,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:52,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 03:00:52,457 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 03:00:52,457 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:00:52,462 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:00:53,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:53,437 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:00:54,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:54,438 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:00:55,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:56,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:57,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:57,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:00:58,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:00:59,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:00,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:01,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:02,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:03,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:03,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:01:04,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:05,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:06,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:07,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:08,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:08,567 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:01:09,098 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:01:09,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:10,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:11,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:12,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:13,431 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:14,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:14,566 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:01:15,432 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:16,443 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:17,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:18,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:19,435 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:19,575 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:01:20,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:21,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:21,919 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 03:01:21,921 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 03:01:21,922 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:01:21,923 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:01:22,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:22,466 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:01:23,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:24,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:24,467 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:01:25,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:25,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:01:26,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:27,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:28,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:29,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:30,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:31,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:31,569 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:01:32,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:33,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:34,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:35,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:36,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:36,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:01:37,440 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:38,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:39,102 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:01:39,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:40,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:41,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:42,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:42,570 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:01:43,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:44,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:45,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:46,441 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:47,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:47,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:01:48,447 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:49,447 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:50,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:51,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:52,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:52,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:01:52,710 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 03:01:52,711 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 03:01:52,712 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:01:52,714 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:01:53,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:53,497 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:01:54,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:54,497 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:01:55,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:56,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:57,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:58,448 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:01:58,573 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:01:59,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:00,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:01,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:02,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:03,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:04,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:04,572 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:02:05,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:06,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:07,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:08,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:09,106 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:02:09,449 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:09,573 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:02:10,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:11,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:12,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:13,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:14,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:15,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:15,573 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:02:16,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:17,450 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:18,453 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:19,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:20,455 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:20,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:02:21,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:22,460 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:23,456 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:23,978 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 03:02:23,979 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 03:02:23,980 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:02:23,981 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:02:24,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:24,529 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:02:25,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:26,457 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:26,530 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:02:26,576 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:02:27,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:28,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:29,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:30,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:31,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:32,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:32,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:02:33,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:34,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:35,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:36,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:37,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:37,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:02:38,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:39,110 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:02:39,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:40,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:41,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:42,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:43,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:43,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:02:44,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:45,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:46,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:47,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:48,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:48,577 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:02:49,460 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:50,471 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:51,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:52,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:53,473 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:53,580 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:02:54,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:55,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:55,941 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 03:02:55,942 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 03:02:55,943 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:02:55,944 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:02:56,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:56,557 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:02:57,474 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:58,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:02:58,559 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:02:58,984 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:02:59,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:00,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:01,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:02,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:03,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:04,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:04,584 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:03:05,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:06,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:07,475 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:08,476 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:09,115 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:03:09,476 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:09,584 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:03:10,476 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:11,476 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:12,476 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:13,476 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:14,476 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:14,585 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:03:15,476 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:16,476 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:17,476 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:18,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:19,477 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:20,480 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:20,584 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:03:21,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:22,488 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:23,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:24,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:25,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:25,571 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 03:03:25,572 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 03:03:25,572 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:03:25,574 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:03:25,585 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:03:25,611 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:03:26,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:26,586 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:03:27,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:28,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:29,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:30,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:31,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:31,588 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:03:32,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:33,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:34,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:35,489 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:36,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:37,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:37,588 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:03:38,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:39,118 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:03:39,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:40,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:41,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:42,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:42,590 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:03:43,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:44,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:45,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:46,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:47,490 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:48,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:48,589 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:03:49,491 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:50,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:51,502 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:52,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:53,499 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:53,603 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:03:54,503 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:55,500 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:56,115 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 03:03:56,116 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 03:03:56,117 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:03:56,118 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:03:56,529 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:56,620 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:03:57,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:58,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:03:58,622 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:03:59,157 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:03:59,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:00,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:01,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:02,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:03,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:04,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:04,593 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:04:05,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:06,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:07,709 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:08,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:09,122 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:04:09,593 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:04:09,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:10,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:11,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:12,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:13,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:14,593 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:04:14,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:15,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:16,615 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:17,625 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:18,618 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:19,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:04:19,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:20,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:21,621 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:22,622 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:23,232 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 03:04:23,233 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 03:04:23,233 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:04:23,235 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:04:23,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:23,647 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:04:24,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:24,648 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:04:25,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:04:25,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:26,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:27,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:28,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:29,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:30,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:31,598 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:04:31,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:32,644 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:33,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:34,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:35,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:36,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:37,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:04:37,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:38,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:39,127 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:04:39,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:40,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:41,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:42,645 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:43,599 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:04:43,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:44,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:45,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:46,646 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:47,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:48,600 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:04:48,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:49,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:50,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:51,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:52,651 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:53,517 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 03:04:53,518 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 03:04:53,518 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:04:53,520 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:04:53,674 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:04:53,929 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:54,559 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:04:54,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:54,676 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:04:55,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:56,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:57,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:58,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:04:59,604 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:04:59,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:00,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:01,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:02,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:03,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:04,604 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:05:04,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:05,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:06,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:07,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:08,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:09,131 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:05:09,605 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:05:09,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:10,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:11,670 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:12,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:13,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:14,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:15,605 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:05:15,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:16,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:17,671 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:18,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:19,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:20,608 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:05:20,679 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:21,679 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:22,683 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:23,686 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:24,680 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:24,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 03:05:24,751 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 03:05:24,751 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:05:24,753 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:05:25,680 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:25,708 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:05:25,792 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:05:26,680 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:26,709 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:05:27,680 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:28,680 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:29,680 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:30,680 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:31,605 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:05:31,680 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:32,680 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:33,681 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:34,681 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:35,681 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:36,681 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:37,605 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:05:37,681 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:38,681 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:39,135 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:05:39,681 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:40,681 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:41,681 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:42,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:05:42,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:43,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:44,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:45,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:46,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:47,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:48,639 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:05:48,682 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:49,685 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:50,688 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:51,689 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:52,689 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:53,657 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:05:53,694 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:54,689 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:55,306 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 03:05:55,307 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 03:05:55,307 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:05:55,309 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:05:55,731 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:55,748 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:05:56,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:56,749 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:05:57,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:58,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:05:59,348 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:05:59,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:00,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:01,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:02,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:03,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:04,642 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:06:04,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:05,730 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:06,731 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:07,731 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:08,731 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:09,140 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:06:09,764 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:06:09,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:10,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:11,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:12,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:13,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:14,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:15,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:06:15,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:16,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:17,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:18,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:19,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:20,759 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:21,669 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:06:21,760 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:22,765 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:23,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:24,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:25,769 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:26,678 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:06:26,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:27,774 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:28,355 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 03:06:28,357 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 03:06:28,357 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:06:28,359 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:06:28,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:28,780 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:06:29,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:30,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:30,781 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:06:31,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:32,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:06:32,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:33,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:34,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:35,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:36,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:37,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:38,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:06:38,775 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:39,143 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:06:39,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:40,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:41,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:42,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:43,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:44,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:06:44,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:45,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:46,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:47,776 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:48,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:49,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:50,674 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:06:50,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:51,777 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:52,780 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:53,783 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:54,789 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:55,679 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:06:55,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:56,791 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:57,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:57,970 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 03:06:57,972 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 03:06:57,973 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:06:57,974 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:06:58,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:06:58,809 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:06:58,810 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:06:59,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:00,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:01,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:07:01,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:02,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:03,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:04,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:05,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:06,790 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:07,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:07:07,791 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:08,791 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:09,147 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:07:09,791 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:10,802 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:11,801 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:12,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:07:12,801 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:13,801 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:14,801 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:15,801 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:16,801 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:17,801 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:18,678 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:07:18,802 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:19,802 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:20,802 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:21,802 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:22,823 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:07:24,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:24,142 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:07:29,142 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:07:30,358 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:07:30,650 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:32,697 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:35,080 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:07:38,461 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:07:38,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:07:39,151 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:07:40,573 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:07:40,793 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:45,574 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:07:46,861 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:07:47,799 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:49,189 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:51,204 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:07:55,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:07:55,694 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:07:56,205 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:07:57,439 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:01,606 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:08:03,559 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:08:04,116 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:05,961 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:07,552 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:08:09,155 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:08:12,458 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:08:13,158 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:08:16,745 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:16,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:17,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:18,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:18,766 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:08:19,172 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:08:19,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:20,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:21,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:22,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:23,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:24,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:08:24,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:25,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:26,747 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:27,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:28,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:29,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:08:29,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:30,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:31,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:32,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:33,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:34,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:08:34,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:35,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:36,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:37,748 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:38,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:39,159 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:08:39,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:40,162 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:08:40,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:41,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:42,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:43,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:44,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:45,459 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:08:45,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:46,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:47,749 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:48,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:49,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:50,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:51,460 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:08:51,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:52,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:53,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:54,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:55,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:56,460 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:08:56,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:57,750 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:58,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:08:59,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:00,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:01,460 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:09:01,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:02,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:03,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:04,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:05,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:06,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:07,460 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:09:07,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:08,751 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:09,163 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:09:09,752 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:13,165 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:09:18,168 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:09:23,169 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:09:28,170 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:09:30,701 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:09:30,741 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:31,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:32,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:33,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:33,742 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:09:34,704 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:35,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:36,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:37,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:38,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:39,167 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:09:39,173 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:09:39,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:40,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:41,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:42,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:43,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:44,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:09:44,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:45,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:46,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:47,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:48,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:49,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:50,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:09:50,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:51,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:52,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:53,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:54,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:55,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:09:55,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:56,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:57,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:58,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:09:59,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:00,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:10:00,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:01,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:02,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:03,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:04,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:05,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:10:05,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:06,707 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:07,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:08,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:09,171 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:10:09,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:10,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:11,705 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:10:11,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:12,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:13,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:14,708 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:15,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:16,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:10:16,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:17,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:18,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:19,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:20,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:21,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:22,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:10:22,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:23,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:24,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:25,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:26,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:27,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:28,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:10:28,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:29,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:30,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:31,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:32,713 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:33,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:34,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:10:34,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:35,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:36,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:37,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:38,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:39,175 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:10:39,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:40,177 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:10:40,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:41,714 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:42,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:43,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:44,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:45,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:10:45,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:46,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:47,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:48,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:49,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:50,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:10:50,715 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:51,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:52,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:53,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:54,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:55,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:10:55,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:56,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:57,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:58,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:10:59,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:00,716 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:01,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:11:01,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:02,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:03,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:04,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:05,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:06,706 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:11:06,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:07,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:08,717 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:09,177 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:11:12,179 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:11:17,180 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:11:22,180 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:11:27,190 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:11:32,198 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:11:37,202 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:11:39,207 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:11:42,209 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:11:46,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:11:46,133 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:47,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:47,998 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:11:48,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:48,433 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: partial_history 2024-12-27 03:11:48,435 DEBUG SenderThread:840649 [sender.py:send():379] send: telemetry 2024-12-27 03:11:48,435 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:11:48,437 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:11:48,438 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:11:48,439 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:11:48,439 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:11:48,441 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:11:48,441 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:11:48,442 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:11:48,443 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:11:48,444 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:11:48,444 DEBUG SenderThread:840649 [sender.py:send():379] send: history 2024-12-27 03:11:48,445 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: summary_record 2024-12-27 03:11:48,446 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:11:49,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:49,313 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:11:49,313 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:11:50,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:51,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:52,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:53,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:53,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:11:54,093 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:55,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:56,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:57,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:58,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:11:59,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:11:59,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:00,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:01,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:02,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:03,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:04,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:12:04,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:05,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:06,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:07,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:08,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:09,094 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:12:09,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:09,185 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:12:10,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:11,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:12,095 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:13,301 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:14,343 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:12:15,327 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:12:15,712 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:19,349 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:12:20,343 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/config.yaml 2024-12-27 03:12:22,299 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:12:22,434 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:23,661 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:24,653 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:12:24,677 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:29,654 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:12:30,519 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:12:31,554 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:32,698 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:35,063 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:12:39,189 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:12:39,464 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:12:39,635 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:40,192 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:12:40,903 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:42,350 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:45,592 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:12:48,586 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:12:49,257 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:50,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:51,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:12:56,528 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:12:56,757 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:12:56,981 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:58,240 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:12:59,369 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:13:02,334 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:13:07,334 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:13:09,190 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:13:09,634 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:13:09,675 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:13:10,636 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:13:11,389 DEBUG SenderThread:840649 [sender.py:send():379] send: exit 2024-12-27 03:13:11,389 INFO SenderThread:840649 [sender.py:send_exit():586] handling exit code: 0 2024-12-27 03:13:11,390 INFO SenderThread:840649 [sender.py:send_exit():588] handling runtime: 15874 2024-12-27 03:13:11,391 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:13:11,392 INFO SenderThread:840649 [sender.py:send_exit():594] send defer 2024-12-27 03:13:11,392 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: defer 2024-12-27 03:13:11,392 INFO HandlerThread:840649 [handler.py:handle_request_defer():184] handle defer: 0 2024-12-27 03:13:11,392 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: defer 2024-12-27 03:13:11,393 INFO SenderThread:840649 [sender.py:send_request_defer():610] handle sender defer: 0 2024-12-27 03:13:11,393 INFO SenderThread:840649 [sender.py:transition_state():614] send defer: 1 2024-12-27 03:13:11,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: defer 2024-12-27 03:13:11,393 INFO HandlerThread:840649 [handler.py:handle_request_defer():184] handle defer: 1 2024-12-27 03:13:11,393 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: defer 2024-12-27 03:13:11,393 INFO SenderThread:840649 [sender.py:send_request_defer():610] handle sender defer: 1 2024-12-27 03:13:11,393 INFO SenderThread:840649 [sender.py:transition_state():614] send defer: 2 2024-12-27 03:13:11,393 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: defer 2024-12-27 03:13:11,393 INFO HandlerThread:840649 [handler.py:handle_request_defer():184] handle defer: 2 2024-12-27 03:13:11,393 INFO HandlerThread:840649 [system_monitor.py:finish():203] Stopping system monitor 2024-12-27 03:13:11,394 DEBUG SystemMonitor:840649 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-12-27 03:13:11,394 DEBUG SystemMonitor:840649 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-12-27 03:13:11,395 INFO HandlerThread:840649 [interfaces.py:finish():200] Joined cpu monitor 2024-12-27 03:13:11,395 INFO HandlerThread:840649 [interfaces.py:finish():200] Joined disk monitor 2024-12-27 03:13:11,644 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:13:11,645 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:13:12,646 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:13:13,039 INFO HandlerThread:840649 [interfaces.py:finish():200] Joined gpu monitor 2024-12-27 03:13:13,040 INFO HandlerThread:840649 [interfaces.py:finish():200] Joined memory monitor 2024-12-27 03:13:13,040 INFO HandlerThread:840649 [interfaces.py:finish():200] Joined network monitor 2024-12-27 03:13:13,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-27 03:13:13,040 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:13:13,041 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: defer 2024-12-27 03:13:13,041 INFO SenderThread:840649 [sender.py:send_request_defer():610] handle sender defer: 2 2024-12-27 03:13:13,041 INFO SenderThread:840649 [sender.py:transition_state():614] send defer: 3 2024-12-27 03:13:13,041 DEBUG SenderThread:840649 [sender.py:send():379] send: stats 2024-12-27 03:13:13,041 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: defer 2024-12-27 03:13:13,042 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: poll_exit 2024-12-27 03:13:13,042 INFO HandlerThread:840649 [handler.py:handle_request_defer():184] handle defer: 3 2024-12-27 03:13:13,043 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: defer 2024-12-27 03:13:13,043 INFO SenderThread:840649 [sender.py:send_request_defer():610] handle sender defer: 3 2024-12-27 03:13:13,043 INFO SenderThread:840649 [sender.py:transition_state():614] send defer: 4 2024-12-27 03:13:13,043 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: defer 2024-12-27 03:13:13,043 INFO HandlerThread:840649 [handler.py:handle_request_defer():184] handle defer: 4 2024-12-27 03:13:13,043 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: defer 2024-12-27 03:13:13,043 INFO SenderThread:840649 [sender.py:send_request_defer():610] handle sender defer: 4 2024-12-27 03:13:13,043 INFO SenderThread:840649 [sender.py:transition_state():614] send defer: 5 2024-12-27 03:13:13,044 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: defer 2024-12-27 03:13:13,044 INFO HandlerThread:840649 [handler.py:handle_request_defer():184] handle defer: 5 2024-12-27 03:13:13,044 DEBUG SenderThread:840649 [sender.py:send():379] send: summary 2024-12-27 03:13:13,046 INFO SenderThread:840649 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-27 03:13:13,046 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: defer 2024-12-27 03:13:13,046 INFO SenderThread:840649 [sender.py:send_request_defer():610] handle sender defer: 5 2024-12-27 03:13:13,046 INFO SenderThread:840649 [sender.py:transition_state():614] send defer: 6 2024-12-27 03:13:13,047 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: defer 2024-12-27 03:13:13,047 INFO HandlerThread:840649 [handler.py:handle_request_defer():184] handle defer: 6 2024-12-27 03:13:13,047 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: defer 2024-12-27 03:13:13,047 INFO SenderThread:840649 [sender.py:send_request_defer():610] handle sender defer: 6 2024-12-27 03:13:13,047 INFO SenderThread:840649 [sender.py:transition_state():614] send defer: 7 2024-12-27 03:13:13,047 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:13:13,048 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: defer 2024-12-27 03:13:13,048 INFO HandlerThread:840649 [handler.py:handle_request_defer():184] handle defer: 7 2024-12-27 03:13:13,048 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: defer 2024-12-27 03:13:13,048 INFO SenderThread:840649 [sender.py:send_request_defer():610] handle sender defer: 7 2024-12-27 03:13:13,390 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-27 03:13:13,646 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:13:16,526 INFO SenderThread:840649 [sender.py:transition_state():614] send defer: 8 2024-12-27 03:13:16,526 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: poll_exit 2024-12-27 03:13:16,527 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: defer 2024-12-27 03:13:16,527 INFO HandlerThread:840649 [handler.py:handle_request_defer():184] handle defer: 8 2024-12-27 03:13:16,527 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: defer 2024-12-27 03:13:16,527 INFO SenderThread:840649 [sender.py:send_request_defer():610] handle sender defer: 8 2024-12-27 03:13:16,528 INFO SenderThread:840649 [job_builder.py:build():440] Attempting to build job artifact 2024-12-27 03:13:16,532 INFO SenderThread:840649 [job_builder.py:_get_source_type():569] is repo sourced job 2024-12-27 03:13:16,648 INFO Thread-12 :840649 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:13:17,392 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-27 03:13:22,394 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:13:27,395 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:13:31,646 INFO SenderThread:840649 [job_builder.py:build():545] adding wandb-job metadata file 2024-12-27 03:13:31,682 INFO SenderThread:840649 [sender.py:transition_state():614] send defer: 9 2024-12-27 03:13:31,683 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: poll_exit 2024-12-27 03:13:31,683 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: defer 2024-12-27 03:13:31,683 INFO HandlerThread:840649 [handler.py:handle_request_defer():184] handle defer: 9 2024-12-27 03:13:31,683 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: status_report 2024-12-27 03:13:31,684 DEBUG SenderThread:840649 [sender.py:send():379] send: artifact 2024-12-27 03:13:32,211 INFO wandb-upload_1:840649 [upload_job.py:push():85] Skipped uploading /home/ctpham_umass_edu/.local/share/wandb/artifacts/staging/tmp9hwv3y4n 2024-12-27 03:13:32,396 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: keepalive 2024-12-27 03:13:32,397 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-27 03:13:33,459 INFO wandb-upload_0:840649 [upload_job.py:push():88] Uploaded file /tmp/tmpf9q5vwsk/wandb-job.json 2024-12-27 03:13:34,124 INFO SenderThread:840649 [sender.py:send_artifact():1530] sent artifact job-https___github.com_chtmp223_BookGen-dev.git_prolong-final_finetune.py - {'id': 'QXJ0aWZhY3Q6MTQwODQxNDM3Mg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjQ5NzE2NzEyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQwNDAxNzI3OA==', 'versionIndex': 19}}} 2024-12-27 03:13:34,124 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: defer 2024-12-27 03:13:34,125 INFO SenderThread:840649 [sender.py:send_request_defer():610] handle sender defer: 9 2024-12-27 03:13:34,125 INFO SenderThread:840649 [dir_watcher.py:finish():358] shutting down directory watcher 2024-12-27 03:13:34,651 INFO SenderThread:840649 [dir_watcher.py:finish():388] scan: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files 2024-12-27 03:13:34,652 INFO SenderThread:840649 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/requirements.txt requirements.txt 2024-12-27 03:13:34,652 INFO SenderThread:840649 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log output.log 2024-12-27 03:13:34,652 INFO SenderThread:840649 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/config.yaml config.yaml 2024-12-27 03:13:34,653 INFO SenderThread:840649 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json wandb-summary.json 2024-12-27 03:13:34,653 INFO SenderThread:840649 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-metadata.json wandb-metadata.json 2024-12-27 03:13:34,660 INFO SenderThread:840649 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/conda-environment.yaml conda-environment.yaml 2024-12-27 03:13:34,665 INFO SenderThread:840649 [sender.py:transition_state():614] send defer: 10 2024-12-27 03:13:34,665 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: poll_exit 2024-12-27 03:13:34,666 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: defer 2024-12-27 03:13:34,666 INFO HandlerThread:840649 [handler.py:handle_request_defer():184] handle defer: 10 2024-12-27 03:13:34,669 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: defer 2024-12-27 03:13:34,669 INFO SenderThread:840649 [sender.py:send_request_defer():610] handle sender defer: 10 2024-12-27 03:13:34,669 INFO SenderThread:840649 [file_pusher.py:finish():169] shutting down file pusher 2024-12-27 03:13:34,864 INFO wandb-upload_2:840649 [upload_job.py:push():130] Uploaded file /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/requirements.txt 2024-12-27 03:13:34,942 INFO wandb-upload_0:840649 [upload_job.py:push():130] Uploaded file /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/wandb-summary.json 2024-12-27 03:13:34,953 INFO wandb-upload_4:840649 [upload_job.py:push():130] Uploaded file /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/conda-environment.yaml 2024-12-27 03:13:34,965 INFO wandb-upload_3:840649 [upload_job.py:push():130] Uploaded file /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/config.yaml 2024-12-27 03:13:34,976 INFO wandb-upload_1:840649 [upload_job.py:push():130] Uploaded file /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/files/output.log 2024-12-27 03:13:35,177 INFO Thread-11 (_thread_body):840649 [sender.py:transition_state():614] send defer: 11 2024-12-27 03:13:35,177 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: defer 2024-12-27 03:13:35,177 INFO HandlerThread:840649 [handler.py:handle_request_defer():184] handle defer: 11 2024-12-27 03:13:35,178 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: defer 2024-12-27 03:13:35,178 INFO SenderThread:840649 [sender.py:send_request_defer():610] handle sender defer: 11 2024-12-27 03:13:35,178 INFO SenderThread:840649 [file_pusher.py:join():175] waiting for file pusher 2024-12-27 03:13:35,178 INFO SenderThread:840649 [sender.py:transition_state():614] send defer: 12 2024-12-27 03:13:35,179 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: defer 2024-12-27 03:13:35,179 INFO HandlerThread:840649 [handler.py:handle_request_defer():184] handle defer: 12 2024-12-27 03:13:35,179 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: defer 2024-12-27 03:13:35,179 INFO SenderThread:840649 [sender.py:send_request_defer():610] handle sender defer: 12 2024-12-27 03:13:35,179 INFO SenderThread:840649 [file_stream.py:finish():601] file stream finish called 2024-12-27 03:13:35,398 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-27 03:13:35,415 INFO SenderThread:840649 [file_stream.py:finish():605] file stream finish is done 2024-12-27 03:13:35,415 INFO SenderThread:840649 [sender.py:transition_state():614] send defer: 13 2024-12-27 03:13:35,415 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: poll_exit 2024-12-27 03:13:35,415 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: defer 2024-12-27 03:13:35,415 INFO HandlerThread:840649 [handler.py:handle_request_defer():184] handle defer: 13 2024-12-27 03:13:35,416 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: defer 2024-12-27 03:13:35,416 INFO SenderThread:840649 [sender.py:send_request_defer():610] handle sender defer: 13 2024-12-27 03:13:35,416 INFO SenderThread:840649 [sender.py:transition_state():614] send defer: 14 2024-12-27 03:13:35,416 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: defer 2024-12-27 03:13:35,416 INFO HandlerThread:840649 [handler.py:handle_request_defer():184] handle defer: 14 2024-12-27 03:13:35,417 DEBUG SenderThread:840649 [sender.py:send():379] send: final 2024-12-27 03:13:35,417 DEBUG SenderThread:840649 [sender.py:send():379] send: footer 2024-12-27 03:13:35,417 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: defer 2024-12-27 03:13:35,417 INFO SenderThread:840649 [sender.py:send_request_defer():610] handle sender defer: 14 2024-12-27 03:13:35,418 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-27 03:13:35,418 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: poll_exit 2024-12-27 03:13:35,419 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: server_info 2024-12-27 03:13:35,419 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: server_info 2024-12-27 03:13:35,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-27 03:13:35,425 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: get_summary 2024-12-27 03:13:35,426 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: sampled_history 2024-12-27 03:13:35,430 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-27 03:13:35,483 DEBUG SenderThread:840649 [sender.py:send_request():406] send_request: poll_exit 2024-12-27 03:13:35,483 INFO MainThread:840649 [wandb_run.py:_footer_history_summary_info():3991] rendering history 2024-12-27 03:13:35,485 INFO MainThread:840649 [wandb_run.py:_footer_history_summary_info():4023] rendering summary 2024-12-27 03:13:35,485 INFO MainThread:840649 [wandb_run.py:_footer_sync_info():3950] logging synced files 2024-12-27 03:13:35,486 DEBUG HandlerThread:840649 [handler.py:handle_request():158] handle_request: shutdown 2024-12-27 03:13:35,486 INFO HandlerThread:840649 [handler.py:finish():882] shutting down handler 2024-12-27 03:13:36,425 INFO WriterThread:840649 [datastore.py:close():296] close: /scratch3/workspace/ctpham_umass_edu-ft/balanced_prolong-512K-base_bsz-16_lr-1e-6_epochs-1_/wandb/run-20241226_224836-fta6xses/run-fta6xses.wandb 2024-12-27 03:13:36,483 INFO SenderThread:840649 [sender.py:finish():1608] shutting down sender 2024-12-27 03:13:36,484 INFO SenderThread:840649 [file_pusher.py:finish():169] shutting down file pusher 2024-12-27 03:13:36,484 INFO SenderThread:840649 [file_pusher.py:join():175] waiting for file pusher