The following values were not passed to `accelerate launch` and had defaults used instead:
	`--num_processes` was set to a value of `8`
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
[2024-08-02 03:39:11,996] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-02 03:39:12,042] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-02 03:39:12,052] [INFO] [root.spawn:38] [PID:218800] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -c /tmp/tmpe6855lfo/test.c -o /tmp/tmpe6855lfo/test.o
[2024-08-02 03:39:12,068] [INFO] [root.spawn:38] [PID:218800] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat /tmp/tmpe6855lfo/test.o -laio -o /tmp/tmpe6855lfo/a.out
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
[2024-08-02 03:39:12,095] [INFO] [root.spawn:38] [PID:218805] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -c /tmp/tmp8r77j3wc/test.c -o /tmp/tmp8r77j3wc/test.o
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
[2024-08-02 03:39:12,112] [INFO] [root.spawn:38] [PID:218805] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat /tmp/tmp8r77j3wc/test.o -laio -o /tmp/tmp8r77j3wc/a.out
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
[2024-08-02 03:39:12,162] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-02 03:39:12,220] [INFO] [root.spawn:38] [PID:218799] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -c /tmp/tmp371f6uqc/test.c -o /tmp/tmp371f6uqc/test.o
[2024-08-02 03:39:12,236] [INFO] [root.spawn:38] [PID:218799] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat /tmp/tmp371f6uqc/test.o -laio -o /tmp/tmp371f6uqc/a.out
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
[2024-08-02 03:39:12,330] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-02 03:39:12,350] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-02 03:39:12,382] [INFO] [root.spawn:38] [PID:218806] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -c /tmp/tmp0i5_9h77/test.c -o /tmp/tmp0i5_9h77/test.o
[2024-08-02 03:39:12,400] [INFO] [root.spawn:38] [PID:218806] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat /tmp/tmp0i5_9h77/test.o -laio -o /tmp/tmp0i5_9h77/a.out
[2024-08-02 03:39:12,404] [INFO] [root.spawn:38] [PID:218803] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -c /tmp/tmpixlyoy3p/test.c -o /tmp/tmpixlyoy3p/test.o
[2024-08-02 03:39:12,419] [INFO] [root.spawn:38] [PID:218803] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat /tmp/tmpixlyoy3p/test.o -laio -o /tmp/tmpixlyoy3p/a.out
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
[2024-08-02 03:39:12,440] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
[2024-08-02 03:39:12,479] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-02 03:39:12,494] [INFO] [root.spawn:38] [PID:218802] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -c /tmp/tmpdgaq0fbv/test.c -o /tmp/tmpdgaq0fbv/test.o
[2024-08-02 03:39:12,504] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-08-02 03:39:12,509] [INFO] [root.spawn:38] [PID:218802] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat /tmp/tmpdgaq0fbv/test.o -laio -o /tmp/tmpdgaq0fbv/a.out
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
[2024-08-02 03:39:12,541] [INFO] [root.spawn:38] [PID:218804] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -c /tmp/tmp5oug0vaq/test.c -o /tmp/tmp5oug0vaq/test.o
[2024-08-02 03:39:12,555] [INFO] [root.spawn:38] [PID:218804] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat /tmp/tmp5oug0vaq/test.o -laio -o /tmp/tmp5oug0vaq/a.out
[2024-08-02 03:39:12,569] [INFO] [root.spawn:38] [PID:218801] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -c /tmp/tmp6reii5be/test.c -o /tmp/tmp6reii5be/test.o
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
[2024-08-02 03:39:12,584] [INFO] [root.spawn:38] [PID:218801] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat /tmp/tmp6reii5be/test.o -laio -o /tmp/tmp6reii5be/a.out
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
 [WARNING]  using untested triton version (2.3.1), only 1.0.0 is known to be compatible
[2024-08-02 03:39:13,229] [INFO] [axolotl.utils.config.models.input.check_eval_packing:963] [PID:218805] [RANK:6] explicitly setting `eval_sample_packing` to match `sample_packing`
[2024-08-02 03:39:13,229] [WARNING] [axolotl.utils.config.models.input.hint_trust_remote_code:327] [PID:218805] [RANK:6] `trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
[2024-08-02 03:39:13,230] [DEBUG] [axolotl.normalize_config:80] [PID:218805] [RANK:6] bf16 support detected, enabling for this configuration.
[2024-08-02 03:39:13,234] [INFO] [axolotl.utils.config.models.input.check_eval_packing:963] [PID:218800] [RANK:1] explicitly setting `eval_sample_packing` to match `sample_packing`
[2024-08-02 03:39:13,234] [WARNING] [axolotl.utils.config.models.input.hint_trust_remote_code:327] [PID:218800] [RANK:1] `trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
[2024-08-02 03:39:13,234] [DEBUG] [axolotl.normalize_config:80] [PID:218800] [RANK:1] bf16 support detected, enabling for this configuration.
[2024-08-02 03:39:13,291] [INFO] [axolotl.normalize_config:183] [PID:218805] [RANK:6] GPU memory usage baseline: 0.000GB (+0.652GB misc)
[2024-08-02 03:39:13,294] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-02 03:39:13,294] [INFO] [axolotl.normalize_config:183] [PID:218800] [RANK:1] GPU memory usage baseline: 0.000GB (+0.652GB misc)
[2024-08-02 03:39:13,297] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-02 03:39:13,325] [INFO] [axolotl.utils.config.models.input.check_eval_packing:963] [PID:218799] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing`
[2024-08-02 03:39:13,325] [WARNING] [axolotl.utils.config.models.input.hint_trust_remote_code:327] [PID:218799] [RANK:0] `trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
[2024-08-02 03:39:13,325] [DEBUG] [axolotl.normalize_config:80] [PID:218799] [RANK:0] bf16 support detected, enabling for this configuration.
[2024-08-02 03:39:13,390] [INFO] [axolotl.normalize_config:183] [PID:218799] [RANK:0] GPU memory usage baseline: 0.000GB (+0.652GB misc)
[2024-08-02 03:39:13,392] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-02 03:39:13,392] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
                                 dP            dP   dP 
                                 88            88   88 
      .d8888b. dP.  .dP .d8888b. 88 .d8888b. d8888P 88 
      88'  `88  `8bd8'  88'  `88 88 88'  `88   88   88 
      88.  .88  .d88b.  88.  .88 88 88.  .88   88   88 
      `88888P8 dP'  `dP `88888P' dP `88888P'   dP   dP 
                                                       
                                                       

****************************************
**** Axolotl Dependency Versions *****
  accelerate: 0.32.0         
        peft: 0.11.1         
transformers: 4.44.0.dev0    
         trl: 0.9.6          
       torch: 2.3.1+cu121    
bitsandbytes: 0.43.3         
****************************************
[2024-08-02 03:39:13,593] [INFO] [axolotl.utils.config.models.input.check_eval_packing:963] [PID:218803] [RANK:4] explicitly setting `eval_sample_packing` to match `sample_packing`
[2024-08-02 03:39:13,593] [WARNING] [axolotl.utils.config.models.input.hint_trust_remote_code:327] [PID:218803] [RANK:4] `trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
[2024-08-02 03:39:13,594] [DEBUG] [axolotl.normalize_config:80] [PID:218803] [RANK:4] bf16 support detected, enabling for this configuration.
[2024-08-02 03:39:13,666] [INFO] [axolotl.utils.config.models.input.check_eval_packing:963] [PID:218802] [RANK:3] explicitly setting `eval_sample_packing` to match `sample_packing`
[2024-08-02 03:39:13,666] [WARNING] [axolotl.utils.config.models.input.hint_trust_remote_code:327] [PID:218802] [RANK:3] `trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
[2024-08-02 03:39:13,667] [DEBUG] [axolotl.normalize_config:80] [PID:218802] [RANK:3] bf16 support detected, enabling for this configuration.
[2024-08-02 03:39:13,730] [INFO] [axolotl.utils.config.models.input.check_eval_packing:963] [PID:218806] [RANK:7] explicitly setting `eval_sample_packing` to match `sample_packing`
[2024-08-02 03:39:13,730] [WARNING] [axolotl.utils.config.models.input.hint_trust_remote_code:327] [PID:218806] [RANK:7] `trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
[2024-08-02 03:39:13,731] [DEBUG] [axolotl.normalize_config:80] [PID:218806] [RANK:7] bf16 support detected, enabling for this configuration.
[2024-08-02 03:39:13,736] [INFO] [axolotl.normalize_config:183] [PID:218802] [RANK:3] GPU memory usage baseline: 0.000GB (+0.652GB misc)
[2024-08-02 03:39:13,738] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-02 03:39:13,797] [INFO] [axolotl.normalize_config:183] [PID:218806] [RANK:7] GPU memory usage baseline: 0.000GB (+0.652GB misc)
[2024-08-02 03:39:13,799] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-02 03:39:13,817] [INFO] [axolotl.normalize_config:183] [PID:218803] [RANK:4] GPU memory usage baseline: 0.000GB (+0.652GB misc)
[2024-08-02 03:39:13,819] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-02 03:39:13,882] [INFO] [axolotl.utils.config.models.input.check_eval_packing:963] [PID:218804] [RANK:5] explicitly setting `eval_sample_packing` to match `sample_packing`
[2024-08-02 03:39:13,882] [WARNING] [axolotl.utils.config.models.input.hint_trust_remote_code:327] [PID:218804] [RANK:5] `trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
[2024-08-02 03:39:13,882] [DEBUG] [axolotl.normalize_config:80] [PID:218804] [RANK:5] bf16 support detected, enabling for this configuration.
[2024-08-02 03:39:13,909] [INFO] [axolotl.utils.config.models.input.check_eval_packing:963] [PID:218801] [RANK:2] explicitly setting `eval_sample_packing` to match `sample_packing`
[2024-08-02 03:39:13,909] [WARNING] [axolotl.utils.config.models.input.hint_trust_remote_code:327] [PID:218801] [RANK:2] `trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
[2024-08-02 03:39:13,910] [DEBUG] [axolotl.normalize_config:80] [PID:218801] [RANK:2] bf16 support detected, enabling for this configuration.
[2024-08-02 03:39:13,973] [INFO] [axolotl.normalize_config:183] [PID:218801] [RANK:2] GPU memory usage baseline: 0.000GB (+0.652GB misc)
[2024-08-02 03:39:13,975] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-02 03:39:14,003] [INFO] [axolotl.normalize_config:183] [PID:218804] [RANK:5] GPU memory usage baseline: 0.000GB (+0.652GB misc)
[2024-08-02 03:39:14,006] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-08-02 03:39:14,032] [DEBUG] [axolotl.load_tokenizer:282] [PID:218799] [RANK:0] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:14,032] [DEBUG] [axolotl.load_tokenizer:283] [PID:218799] [RANK:0] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:14,032] [DEBUG] [axolotl.load_tokenizer:284] [PID:218799] [RANK:0] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:14,032] [DEBUG] [axolotl.load_tokenizer:285] [PID:218799] [RANK:0] UNK: None / None
[2024-08-02 03:39:14,032] [INFO] [axolotl.load_tokenized_prepared_datasets:184] [PID:218799] [RANK:0] Unable to find prepared dataset in last_run_prepared/6e8f5c4b2563ce0d93325368db2c68fa
[2024-08-02 03:39:14,032] [INFO] [axolotl.load_tokenized_prepared_datasets:185] [PID:218799] [RANK:0] Loading raw datasets...
[2024-08-02 03:39:14,032] [WARNING] [axolotl.load_tokenized_prepared_datasets:187] [PID:218799] [RANK:0] Processing datasets during training can lead to VRAM instability. Please pre-process your dataset.
[2024-08-02 03:39:14,056] [DEBUG] [axolotl.load_tokenizer:282] [PID:218805] [RANK:6] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:14,056] [DEBUG] [axolotl.load_tokenizer:283] [PID:218805] [RANK:6] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:14,056] [DEBUG] [axolotl.load_tokenizer:284] [PID:218805] [RANK:6] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:14,056] [DEBUG] [axolotl.load_tokenizer:285] [PID:218805] [RANK:6] UNK: None / None
[2024-08-02 03:39:14,084] [DEBUG] [axolotl.load_tokenizer:282] [PID:218800] [RANK:1] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:14,084] [DEBUG] [axolotl.load_tokenizer:283] [PID:218800] [RANK:1] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:14,084] [DEBUG] [axolotl.load_tokenizer:284] [PID:218800] [RANK:1] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:14,084] [DEBUG] [axolotl.load_tokenizer:285] [PID:218800] [RANK:1] UNK: None / None
[2024-08-02 03:39:14,130] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218799] [RANK:0] Loading dataset with base_type: sharegpt and prompt_style: None
[2024-08-02 03:39:14,408] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218799] [RANK:0] Loading dataset with base_type: sharegpt and prompt_style: None
[2024-08-02 03:39:14,590] [INFO] [axolotl.load_tokenized_prepared_datasets:419] [PID:218799] [RANK:0] merging datasets
[2024-08-02 03:39:14,606] [DEBUG] [axolotl.load_tokenized_prepared_datasets:424] [PID:218799] [RANK:0] shuffle merged datasets
[2024-08-02 03:39:14,759] [DEBUG] [axolotl.load_tokenizer:282] [PID:218802] [RANK:3] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:14,760] [DEBUG] [axolotl.load_tokenizer:283] [PID:218802] [RANK:3] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:14,760] [DEBUG] [axolotl.load_tokenizer:284] [PID:218802] [RANK:3] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:14,760] [DEBUG] [axolotl.load_tokenizer:285] [PID:218802] [RANK:3] UNK: None / None
[2024-08-02 03:39:14,796] [DEBUG] [axolotl.load_tokenizer:282] [PID:218801] [RANK:2] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:14,796] [DEBUG] [axolotl.load_tokenizer:283] [PID:218801] [RANK:2] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:14,796] [DEBUG] [axolotl.load_tokenizer:284] [PID:218801] [RANK:2] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:14,796] [DEBUG] [axolotl.load_tokenizer:285] [PID:218801] [RANK:2] UNK: None / None
[2024-08-02 03:39:14,800] [DEBUG] [axolotl.load_tokenizer:282] [PID:218806] [RANK:7] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:14,800] [DEBUG] [axolotl.load_tokenizer:283] [PID:218806] [RANK:7] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:14,800] [DEBUG] [axolotl.load_tokenizer:284] [PID:218806] [RANK:7] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:14,800] [DEBUG] [axolotl.load_tokenizer:285] [PID:218806] [RANK:7] UNK: None / None
[2024-08-02 03:39:14,802] [DEBUG] [axolotl.load_tokenizer:282] [PID:218803] [RANK:4] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:14,802] [DEBUG] [axolotl.load_tokenizer:283] [PID:218803] [RANK:4] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:14,802] [DEBUG] [axolotl.load_tokenizer:284] [PID:218803] [RANK:4] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:14,803] [DEBUG] [axolotl.load_tokenizer:285] [PID:218803] [RANK:4] UNK: None / None
[2024-08-02 03:39:14,920] [DEBUG] [axolotl.load_tokenizer:282] [PID:218804] [RANK:5] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:14,921] [DEBUG] [axolotl.load_tokenizer:283] [PID:218804] [RANK:5] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:14,921] [DEBUG] [axolotl.load_tokenizer:284] [PID:218804] [RANK:5] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:14,921] [DEBUG] [axolotl.load_tokenizer:285] [PID:218804] [RANK:5] UNK: None / None

Dropping Long Sequences (num_proc=96):   0%|          | 0/266992 [00:00<?, ? examples/s]
Dropping Long Sequences (num_proc=96):   0%|          | 1000/266992 [00:00<02:17, 1940.39 examples/s]
Dropping Long Sequences (num_proc=96):   1%|▏         | 4000/266992 [00:00<00:38, 6763.57 examples/s]
Dropping Long Sequences (num_proc=96):   3%|▎         | 7000/266992 [00:00<00:24, 10487.04 examples/s]
Dropping Long Sequences (num_proc=96):   4%|▎         | 10000/266992 [00:00<00:17, 14499.30 examples/s]
Dropping Long Sequences (num_proc=96):   6%|▋         | 17000/266992 [00:01<00:09, 26729.39 examples/s]
Dropping Long Sequences (num_proc=96):   9%|▉         | 24000/266992 [00:01<00:06, 36060.27 examples/s]
Dropping Long Sequences (num_proc=96):  11%|█         | 29000/266992 [00:01<00:07, 33708.79 examples/s]
Dropping Long Sequences (num_proc=96):  13%|█▎        | 36000/266992 [00:01<00:05, 40817.11 examples/s]
Dropping Long Sequences (num_proc=96):  17%|█▋        | 45000/266992 [00:01<00:04, 46315.00 examples/s]
Dropping Long Sequences (num_proc=96):  21%|██        | 55000/266992 [00:01<00:03, 53539.83 examples/s]
Dropping Long Sequences (num_proc=96):  27%|██▋       | 72000/266992 [00:01<00:02, 79961.40 examples/s]
Dropping Long Sequences (num_proc=96):  32%|███▏      | 86000/266992 [00:01<00:01, 91290.33 examples/s]
Dropping Long Sequences (num_proc=96):  37%|███▋      | 100000/266992 [00:02<00:01, 103221.42 examples/s]
Dropping Long Sequences (num_proc=96):  42%|████▏     | 112000/266992 [00:02<00:01, 81257.54 examples/s] 
Dropping Long Sequences (num_proc=96):  46%|████▌     | 122000/266992 [00:02<00:02, 69854.57 examples/s]
Dropping Long Sequences (num_proc=96):  49%|████▉     | 130782/266992 [00:02<00:02, 63232.39 examples/s]
Dropping Long Sequences (num_proc=96):  52%|█████▏    | 138564/266992 [00:02<00:01, 65669.82 examples/s]
Dropping Long Sequences (num_proc=96):  55%|█████▍    | 146128/266992 [00:02<00:01, 66396.55 examples/s]
Dropping Long Sequences (num_proc=96):  58%|█████▊    | 153910/266992 [00:03<00:01, 68769.56 examples/s]
Dropping Long Sequences (num_proc=96):  65%|██████▌   | 174601/266992 [00:03<00:00, 103072.63 examples/s]
Dropping Long Sequences (num_proc=96):  71%|███████   | 189945/266992 [00:03<00:00, 115454.66 examples/s]
Dropping Long Sequences (num_proc=96):  76%|███████▌  | 202631/266992 [00:03<00:00, 115427.99 examples/s]
Dropping Long Sequences (num_proc=96):  81%|████████  | 216226/266992 [00:03<00:00, 116850.10 examples/s]
Dropping Long Sequences (num_proc=96):  86%|████████▌ | 228723/266992 [00:03<00:00, 105921.84 examples/s]
Dropping Long Sequences (num_proc=96):  91%|█████████ | 243562/266992 [00:03<00:00, 116861.77 examples/s]
Dropping Long Sequences (num_proc=96):  97%|█████████▋| 259963/266992 [00:03<00:00, 120241.46 examples/s]
Dropping Long Sequences (num_proc=96): 100%|██████████| 266992/266992 [00:04<00:00, 66676.37 examples/s] 

Add position_id column (Sample Packing) (num_proc=96):   0%|          | 0/266992 [00:00<?, ? examples/s]
Add position_id column (Sample Packing) (num_proc=96):   0%|          | 184/266992 [00:00<03:05, 1441.37 examples/s]
Add position_id column (Sample Packing) (num_proc=96):   1%|          | 1926/266992 [00:00<00:26, 9865.29 examples/s]
Add position_id column (Sample Packing) (num_proc=96):   2%|▏         | 5525/266992 [00:00<00:12, 20931.61 examples/s]
Add position_id column (Sample Packing) (num_proc=96):   3%|▎         | 8760/266992 [00:00<00:10, 25015.54 examples/s]
Add position_id column (Sample Packing) (num_proc=96):   4%|▍         | 11637/266992 [00:00<00:09, 25835.58 examples/s]
Add position_id column (Sample Packing) (num_proc=96):   6%|▌         | 14811/266992 [00:00<00:09, 27131.52 examples/s]
Add position_id column (Sample Packing) (num_proc=96):   7%|▋         | 18280/266992 [00:00<00:08, 28590.94 examples/s]
Add position_id column (Sample Packing) (num_proc=96):  17%|█▋        | 46129/266992 [00:00<00:02, 102784.03 examples/s]
Add position_id column (Sample Packing) (num_proc=96):  27%|██▋       | 72049/266992 [00:00<00:01, 149494.83 examples/s]
Add position_id column (Sample Packing) (num_proc=96):  33%|███▎      | 87975/266992 [00:01<00:01, 152365.49 examples/s]
Add position_id column (Sample Packing) (num_proc=96):  39%|███▉      | 103462/266992 [00:01<00:01, 152752.38 examples/s]
Add position_id column (Sample Packing) (num_proc=96):  46%|████▌     | 123443/266992 [00:01<00:00, 166792.89 examples/s]
Add position_id column (Sample Packing) (num_proc=96):  53%|█████▎    | 140308/266992 [00:01<00:00, 151622.73 examples/s]
Add position_id column (Sample Packing) (num_proc=96):  59%|█████▉    | 156980/266992 [00:01<00:00, 155809.96 examples/s]
Add position_id column (Sample Packing) (num_proc=96):  65%|██████▌   | 174851/266992 [00:01<00:00, 162133.28 examples/s]
Add position_id column (Sample Packing) (num_proc=96):  72%|███████▏  | 192258/266992 [00:01<00:00, 164137.44 examples/s]
Add position_id column (Sample Packing) (num_proc=96):  78%|███████▊  | 208989/266992 [00:01<00:00, 158085.28 examples/s]
Add position_id column (Sample Packing) (num_proc=96):  84%|████████▍ | 224982/266992 [00:01<00:00, 148572.54 examples/s]
Add position_id column (Sample Packing) (num_proc=96):  90%|████████▉ | 240145/266992 [00:02<00:00, 143721.53 examples/s]
Add position_id column (Sample Packing) (num_proc=96):  95%|█████████▌| 254710/266992 [00:02<00:00, 136613.49 examples/s]
Add position_id column (Sample Packing) (num_proc=96): 100%|██████████| 266992/266992 [00:02<00:00, 104117.27 examples/s]
[2024-08-02 03:39:26,305] [INFO] [axolotl.load_tokenized_prepared_datasets:184] [PID:218801] [RANK:2] Unable to find prepared dataset in last_run_prepared/6e8f5c4b2563ce0d93325368db2c68fa
[2024-08-02 03:39:26,305] [INFO] [axolotl.load_tokenized_prepared_datasets:184] [PID:218802] [RANK:3] Unable to find prepared dataset in last_run_prepared/6e8f5c4b2563ce0d93325368db2c68fa
[2024-08-02 03:39:26,305] [INFO] [axolotl.load_tokenized_prepared_datasets:185] [PID:218801] [RANK:2] Loading raw datasets...
[2024-08-02 03:39:26,305] [INFO] [axolotl.load_tokenized_prepared_datasets:185] [PID:218802] [RANK:3] Loading raw datasets...
[2024-08-02 03:39:26,305] [WARNING] [axolotl.load_tokenized_prepared_datasets:187] [PID:218802] [RANK:3] Processing datasets during training can lead to VRAM instability. Please pre-process your dataset.
[2024-08-02 03:39:26,305] [WARNING] [axolotl.load_tokenized_prepared_datasets:187] [PID:218801] [RANK:2] Processing datasets during training can lead to VRAM instability. Please pre-process your dataset.
[2024-08-02 03:39:26,305] [INFO] [axolotl.load_tokenized_prepared_datasets:184] [PID:218804] [RANK:5] Unable to find prepared dataset in last_run_prepared/6e8f5c4b2563ce0d93325368db2c68fa
[2024-08-02 03:39:26,305] [INFO] [axolotl.load_tokenized_prepared_datasets:184] [PID:218803] [RANK:4] Unable to find prepared dataset in last_run_prepared/6e8f5c4b2563ce0d93325368db2c68fa
[2024-08-02 03:39:26,305] [INFO] [axolotl.load_tokenized_prepared_datasets:184] [PID:218806] [RANK:7] Unable to find prepared dataset in last_run_prepared/6e8f5c4b2563ce0d93325368db2c68fa
[2024-08-02 03:39:26,305] [INFO] [axolotl.load_tokenized_prepared_datasets:185] [PID:218804] [RANK:5] Loading raw datasets...
[2024-08-02 03:39:26,306] [INFO] [axolotl.load_tokenized_prepared_datasets:185] [PID:218803] [RANK:4] Loading raw datasets...
[2024-08-02 03:39:26,305] [INFO] [axolotl.load_tokenized_prepared_datasets:184] [PID:218800] [RANK:1] Unable to find prepared dataset in last_run_prepared/6e8f5c4b2563ce0d93325368db2c68fa
[2024-08-02 03:39:26,306] [INFO] [axolotl.load_tokenized_prepared_datasets:185] [PID:218806] [RANK:7] Loading raw datasets...
[2024-08-02 03:39:26,306] [WARNING] [axolotl.load_tokenized_prepared_datasets:187] [PID:218804] [RANK:5] Processing datasets during training can lead to VRAM instability. Please pre-process your dataset.
[2024-08-02 03:39:26,306] [WARNING] [axolotl.load_tokenized_prepared_datasets:187] [PID:218803] [RANK:4] Processing datasets during training can lead to VRAM instability. Please pre-process your dataset.
[2024-08-02 03:39:26,306] [INFO] [axolotl.load_tokenized_prepared_datasets:185] [PID:218800] [RANK:1] Loading raw datasets...
[2024-08-02 03:39:26,306] [WARNING] [axolotl.load_tokenized_prepared_datasets:187] [PID:218806] [RANK:7] Processing datasets during training can lead to VRAM instability. Please pre-process your dataset.
[2024-08-02 03:39:26,306] [WARNING] [axolotl.load_tokenized_prepared_datasets:187] [PID:218800] [RANK:1] Processing datasets during training can lead to VRAM instability. Please pre-process your dataset.
[2024-08-02 03:39:26,305] [INFO] [axolotl.load_tokenized_prepared_datasets:184] [PID:218805] [RANK:6] Unable to find prepared dataset in last_run_prepared/6e8f5c4b2563ce0d93325368db2c68fa
[2024-08-02 03:39:26,306] [INFO] [axolotl.load_tokenized_prepared_datasets:185] [PID:218805] [RANK:6] Loading raw datasets...
[2024-08-02 03:39:26,306] [WARNING] [axolotl.load_tokenized_prepared_datasets:187] [PID:218805] [RANK:6] Processing datasets during training can lead to VRAM instability. Please pre-process your dataset.
[2024-08-02 03:39:26,309] [INFO] [axolotl.load_tokenized_prepared_datasets:432] [PID:218799] [RANK:0] Saving merged prepared dataset to disk... last_run_prepared/6e8f5c4b2563ce0d93325368db2c68fa

Saving the dataset (0/6 shards):   0%|          | 0/266992 [00:00<?, ? examples/s][2024-08-02 03:39:26,440] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218801] [RANK:2] Loading dataset with base_type: sharegpt and prompt_style: None
[2024-08-02 03:39:26,440] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218802] [RANK:3] Loading dataset with base_type: sharegpt and prompt_style: None
[2024-08-02 03:39:26,446] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218803] [RANK:4] Loading dataset with base_type: sharegpt and prompt_style: None
[2024-08-02 03:39:26,449] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218806] [RANK:7] Loading dataset with base_type: sharegpt and prompt_style: None
[2024-08-02 03:39:26,451] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218800] [RANK:1] Loading dataset with base_type: sharegpt and prompt_style: None
[2024-08-02 03:39:26,454] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218805] [RANK:6] Loading dataset with base_type: sharegpt and prompt_style: None

Saving the dataset (0/6 shards):   7%|▋         | 19000/266992 [00:00<00:01, 126438.09 examples/s][2024-08-02 03:39:26,502] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218804] [RANK:5] Loading dataset with base_type: sharegpt and prompt_style: None

Saving the dataset (0/6 shards):  15%|█▌        | 41000/266992 [00:00<00:01, 166288.64 examples/s]
Saving the dataset (1/6 shards):  17%|█▋        | 44499/266992 [00:00<00:01, 166288.64 examples/s][2024-08-02 03:39:26,769] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218801] [RANK:2] Loading dataset with base_type: sharegpt and prompt_style: None
[2024-08-02 03:39:26,785] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218800] [RANK:1] Loading dataset with base_type: sharegpt and prompt_style: None
[2024-08-02 03:39:26,799] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218802] [RANK:3] Loading dataset with base_type: sharegpt and prompt_style: None
[2024-08-02 03:39:26,809] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218806] [RANK:7] Loading dataset with base_type: sharegpt and prompt_style: None
[2024-08-02 03:39:26,814] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218803] [RANK:4] Loading dataset with base_type: sharegpt and prompt_style: None
[2024-08-02 03:39:26,826] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218805] [RANK:6] Loading dataset with base_type: sharegpt and prompt_style: None

Saving the dataset (1/6 shards):  25%|██▍       | 65499/266992 [00:00<00:01, 128256.86 examples/s][2024-08-02 03:39:26,885] [INFO] [axolotl.get_dataset_wrapper:545] [PID:218804] [RANK:5] Loading dataset with base_type: sharegpt and prompt_style: None

Saving the dataset (1/6 shards):  33%|███▎      | 87499/266992 [00:00<00:01, 150592.04 examples/s][2024-08-02 03:39:26,983] [INFO] [axolotl.load_tokenized_prepared_datasets:419] [PID:218801] [RANK:2] merging datasets
[2024-08-02 03:39:26,995] [INFO] [axolotl.load_tokenized_prepared_datasets:419] [PID:218800] [RANK:1] merging datasets
[2024-08-02 03:39:26,999] [DEBUG] [axolotl.load_tokenized_prepared_datasets:424] [PID:218801] [RANK:2] shuffle merged datasets
[2024-08-02 03:39:27,011] [DEBUG] [axolotl.load_tokenized_prepared_datasets:424] [PID:218800] [RANK:1] shuffle merged datasets
[2024-08-02 03:39:27,026] [INFO] [axolotl.load_tokenized_prepared_datasets:419] [PID:218802] [RANK:3] merging datasets

Saving the dataset (2/6 shards):  33%|███▎      | 88998/266992 [00:00<00:01, 150592.04 examples/s][2024-08-02 03:39:27,045] [DEBUG] [axolotl.load_tokenized_prepared_datasets:424] [PID:218802] [RANK:3] shuffle merged datasets
[2024-08-02 03:39:27,053] [INFO] [axolotl.load_tokenized_prepared_datasets:419] [PID:218806] [RANK:7] merging datasets
[2024-08-02 03:39:27,058] [INFO] [axolotl.load_tokenized_prepared_datasets:419] [PID:218803] [RANK:4] merging datasets
[2024-08-02 03:39:27,072] [DEBUG] [axolotl.load_tokenized_prepared_datasets:424] [PID:218806] [RANK:7] shuffle merged datasets
[2024-08-02 03:39:27,075] [INFO] [axolotl.load_tokenized_prepared_datasets:419] [PID:218805] [RANK:6] merging datasets
[2024-08-02 03:39:27,078] [DEBUG] [axolotl.load_tokenized_prepared_datasets:424] [PID:218803] [RANK:4] shuffle merged datasets
[2024-08-02 03:39:27,091] [DEBUG] [axolotl.load_tokenized_prepared_datasets:424] [PID:218805] [RANK:6] shuffle merged datasets
[2024-08-02 03:39:27,126] [INFO] [axolotl.load_tokenized_prepared_datasets:419] [PID:218804] [RANK:5] merging datasets
[2024-08-02 03:39:27,145] [DEBUG] [axolotl.load_tokenized_prepared_datasets:424] [PID:218804] [RANK:5] shuffle merged datasets

Saving the dataset (2/6 shards):  41%|████      | 109998/266992 [00:00<00:01, 129950.96 examples/s]
Saving the dataset (2/6 shards):  49%|████▉     | 131998/266992 [00:00<00:00, 148404.21 examples/s]
Saving the dataset (3/6 shards):  50%|█████     | 133497/266992 [00:01<00:00, 148404.21 examples/s]
Saving the dataset (3/6 shards):  58%|█████▊    | 154497/266992 [00:01<00:00, 131015.24 examples/s]
Saving the dataset (3/6 shards):  66%|██████▌   | 176497/266992 [00:01<00:00, 148520.68 examples/s]
Saving the dataset (4/6 shards):  67%|██████▋   | 177996/266992 [00:01<00:00, 148520.68 examples/s]
Saving the dataset (4/6 shards):  75%|███████▍  | 198996/266992 [00:01<00:00, 135036.28 examples/s]
Saving the dataset (4/6 shards):  83%|████████▎ | 220996/266992 [00:01<00:00, 151265.79 examples/s]
Saving the dataset (5/6 shards):  83%|████████▎ | 222494/266992 [00:01<00:00, 151265.79 examples/s]
Saving the dataset (5/6 shards):  90%|█████████ | 241494/266992 [00:01<00:00, 133400.88 examples/s]
Saving the dataset (5/6 shards):  98%|█████████▊| 261494/266992 [00:01<00:00, 146832.56 examples/s]
Saving the dataset (6/6 shards): 100%|██████████| 266992/266992 [00:01<00:00, 146832.56 examples/s]
Saving the dataset (6/6 shards): 100%|██████████| 266992/266992 [00:01<00:00, 137484.99 examples/s]
[2024-08-02 03:39:28,347] [DEBUG] [axolotl.calculate_total_num_steps:298] [PID:218799] [RANK:0] total_num_tokens: 1_193_282
[2024-08-02 03:39:28,374] [DEBUG] [axolotl.calculate_total_num_steps:311] [PID:218799] [RANK:0] `total_supervised_tokens: 654_792`
[2024-08-02 03:39:32,535] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 1.0 total_num_tokens per device: 149160
[2024-08-02 03:39:32,536] [DEBUG] [axolotl.calculate_total_num_steps:363] [PID:218799] [RANK:0] data_loader_len: 2
[2024-08-02 03:39:33,274] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 1.0 total_num_tokens per device: 149160
[2024-08-02 03:39:33,290] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 1.0 total_num_tokens per device: 149160
[2024-08-02 03:39:33,670] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 1.0 total_num_tokens per device: 149160
[2024-08-02 03:39:33,678] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 1.0 total_num_tokens per device: 149160
[2024-08-02 03:39:33,731] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 1.0 total_num_tokens per device: 149160
[2024-08-02 03:39:33,735] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 1.0 total_num_tokens per device: 149160
[2024-08-02 03:39:33,826] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 1.0 total_num_tokens per device: 149160
[2024-08-02 03:39:33,943] [INFO] [axolotl.calc_sample_packing_eff_est:369] [PID:218799] [RANK:0] sample_packing_eff_est across ranks: [0.9710953831672668, 0.9710953831672668, 0.9710953831672668, 0.9710953831672668, 0.9710953831672668, 0.9710953831672668, 0.9710953831672668, 0.9842182993888855]
[2024-08-02 03:39:33,944] [DEBUG] [axolotl.calculate_total_num_steps:381] [PID:218799] [RANK:0] sample_packing_eff_est: None
[2024-08-02 03:39:33,944] [DEBUG] [axolotl.calculate_total_num_steps:389] [PID:218799] [RANK:0] total_num_steps: 6
[2024-08-02 03:39:34,306] [DEBUG] [axolotl.calculate_total_num_steps:298] [PID:218799] [RANK:0] total_num_tokens: 117_563_908
[2024-08-02 03:39:36,357] [DEBUG] [axolotl.calculate_total_num_steps:311] [PID:218799] [RANK:0] `total_supervised_tokens: 64_640_556`
[2024-08-02 03:39:36,490] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 1.0 total_num_tokens per device: 14695488
[2024-08-02 03:39:36,490] [DEBUG] [axolotl.calculate_total_num_steps:363] [PID:218799] [RANK:0] data_loader_len: 221
[2024-08-02 03:39:36,547] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 1.0 total_num_tokens per device: 14695488
[2024-08-02 03:39:36,548] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 1.0 total_num_tokens per device: 14695488
[2024-08-02 03:39:36,591] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 1.0 total_num_tokens per device: 14695488
[2024-08-02 03:39:37,034] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 1.0 total_num_tokens per device: 14695488
[2024-08-02 03:39:37,043] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 1.0 total_num_tokens per device: 14695488
[2024-08-02 03:39:37,064] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 1.0 total_num_tokens per device: 14695488
[2024-08-02 03:39:37,069] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 1.0 total_num_tokens per device: 14695488
[2024-08-02 03:39:37,070] [INFO] [axolotl.calc_sample_packing_eff_est:369] [PID:218799] [RANK:0] sample_packing_eff_est across ranks: [0.9829494953155518, 0.9826803207397461, 0.9830842018127441, 0.9826803207397461, 0.9828149080276489, 0.9826803207397461, 0.9829494953155518, 0.9830842018127441]
[2024-08-02 03:39:37,079] [DEBUG] [axolotl.calculate_total_num_steps:381] [PID:218799] [RANK:0] sample_packing_eff_est: 0.99
[2024-08-02 03:39:37,079] [DEBUG] [axolotl.calculate_total_num_steps:389] [PID:218799] [RANK:0] total_num_steps: 663
[2024-08-02 03:39:37,100] [DEBUG] [axolotl.train.train:66] [PID:218799] [RANK:0] loading tokenizer... meta-llama/Meta-Llama-3.1-8B
[2024-08-02 03:39:37,427] [DEBUG] [axolotl.load_tokenizer:282] [PID:218806] [RANK:7] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:37,427] [DEBUG] [axolotl.load_tokenizer:283] [PID:218806] [RANK:7] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:37,428] [DEBUG] [axolotl.load_tokenizer:284] [PID:218806] [RANK:7] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:37,428] [DEBUG] [axolotl.load_tokenizer:285] [PID:218806] [RANK:7] UNK: None / None
[2024-08-02 03:39:37,458] [DEBUG] [axolotl.load_tokenizer:282] [PID:218799] [RANK:0] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:37,458] [DEBUG] [axolotl.load_tokenizer:283] [PID:218799] [RANK:0] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:37,458] [DEBUG] [axolotl.load_tokenizer:284] [PID:218799] [RANK:0] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:37,458] [DEBUG] [axolotl.load_tokenizer:285] [PID:218799] [RANK:0] UNK: None / None
[2024-08-02 03:39:37,459] [INFO] [axolotl.train.train:86] [PID:218799] [RANK:0] Using Auto-resume functionality to start with checkpoint at llama_3.1_8b_function_calling/checkpoint-112
[2024-08-02 03:39:37,459] [DEBUG] [axolotl.train.train:95] [PID:218799] [RANK:0] loading model
[2024-08-02 03:39:37,462] [DEBUG] [axolotl.load_tokenizer:282] [PID:218801] [RANK:2] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:37,462] [DEBUG] [axolotl.load_tokenizer:283] [PID:218801] [RANK:2] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:37,462] [DEBUG] [axolotl.load_tokenizer:284] [PID:218801] [RANK:2] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:37,462] [DEBUG] [axolotl.load_tokenizer:285] [PID:218801] [RANK:2] UNK: None / None
[2024-08-02 03:39:37,472] [DEBUG] [axolotl.load_tokenizer:282] [PID:218804] [RANK:5] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:37,472] [DEBUG] [axolotl.load_tokenizer:283] [PID:218804] [RANK:5] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:37,472] [DEBUG] [axolotl.load_tokenizer:284] [PID:218804] [RANK:5] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:37,472] [DEBUG] [axolotl.load_tokenizer:285] [PID:218804] [RANK:5] UNK: None / None
[2024-08-02 03:39:37,503] [DEBUG] [axolotl.load_tokenizer:282] [PID:218803] [RANK:4] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:37,503] [DEBUG] [axolotl.load_tokenizer:283] [PID:218803] [RANK:4] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:37,503] [DEBUG] [axolotl.load_tokenizer:284] [PID:218803] [RANK:4] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:37,503] [DEBUG] [axolotl.load_tokenizer:285] [PID:218803] [RANK:4] UNK: None / None
[2024-08-02 03:39:37,517] [DEBUG] [axolotl.load_tokenizer:282] [PID:218805] [RANK:6] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:37,517] [DEBUG] [axolotl.load_tokenizer:283] [PID:218805] [RANK:6] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:37,517] [DEBUG] [axolotl.load_tokenizer:284] [PID:218805] [RANK:6] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:37,517] [DEBUG] [axolotl.load_tokenizer:285] [PID:218805] [RANK:6] UNK: None / None
[2024-08-02 03:39:37,522] [DEBUG] [axolotl.load_tokenizer:282] [PID:218802] [RANK:3] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:37,522] [DEBUG] [axolotl.load_tokenizer:283] [PID:218802] [RANK:3] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:37,522] [DEBUG] [axolotl.load_tokenizer:284] [PID:218802] [RANK:3] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:37,522] [DEBUG] [axolotl.load_tokenizer:285] [PID:218802] [RANK:3] UNK: None / None
[2024-08-02 03:39:37,613] [DEBUG] [axolotl.load_tokenizer:282] [PID:218800] [RANK:1] EOS: 128256 / <|im_end|>
[2024-08-02 03:39:37,613] [DEBUG] [axolotl.load_tokenizer:283] [PID:218800] [RANK:1] BOS: 128000 / <|begin_of_text|>
[2024-08-02 03:39:37,613] [DEBUG] [axolotl.load_tokenizer:284] [PID:218800] [RANK:1] PAD: 128257 / <|endoftext|>
[2024-08-02 03:39:37,613] [DEBUG] [axolotl.load_tokenizer:285] [PID:218800] [RANK:1] UNK: None / None
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[2024-08-02 03:39:39,638] [INFO] [partition_parameters.py:345:__exit__] finished initializing model - num_params = 291, num_elems = 8.03B

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:05,  1.69s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:05,  1.69s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:05,  1.68s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:05,  1.69s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:05,  1.69s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:05,  1.70s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:05,  1.74s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:05,  1.78s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:03<00:03,  1.73s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:03<00:03,  1.74s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:03<00:03,  1.74s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:03<00:03,  1.74s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:03<00:03,  1.74s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:03<00:03,  1.75s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:03<00:03,  1.76s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:03<00:03,  1.77s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:05<00:01,  1.71s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:05<00:01,  1.72s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:05<00:01,  1.72s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:05<00:01,  1.72s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:05<00:01,  1.72s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:05<00:01,  1.72s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:05<00:01,  1.73s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:05<00:01,  1.74s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.27s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.44s/it]

Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.27s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.44s/it]

Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.28s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.44s/it]

Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.28s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.44s/it]

Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.27s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.45s/it]

Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.28s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.44s/it]

Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.28s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.44s/it]

Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.29s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00,  1.45s/it]
[2024-08-02 03:39:46,193] [INFO] [axolotl.load_model:791] [PID:218806] [RANK:7] GPU memory usage after model load: 1.957GB (+1.963GB cache, +1.132GB misc)
[2024-08-02 03:39:46,193] [INFO] [axolotl.load_model:791] [PID:218804] [RANK:5] GPU memory usage after model load: 1.957GB (+1.963GB cache, +1.132GB misc)
[2024-08-02 03:39:46,193] [INFO] [axolotl.load_model:791] [PID:218805] [RANK:6] GPU memory usage after model load: 1.957GB (+1.963GB cache, +1.155GB misc)
[2024-08-02 03:39:46,195] [INFO] [axolotl.load_model:856] [PID:218806] [RANK:7] converting modules to torch.bfloat16 for flash attention
[2024-08-02 03:39:46,196] [INFO] [axolotl.load_model:856] [PID:218804] [RANK:5] converting modules to torch.bfloat16 for flash attention
[2024-08-02 03:39:46,196] [INFO] [axolotl.load_model:856] [PID:218805] [RANK:6] converting modules to torch.bfloat16 for flash attention
[2024-08-02 03:39:46,197] [INFO] [axolotl.load_model:791] [PID:218799] [RANK:0] GPU memory usage after model load: 1.957GB (+1.963GB cache, +1.210GB misc)
[2024-08-02 03:39:46,198] [INFO] [axolotl.load_model:791] [PID:218800] [RANK:1] GPU memory usage after model load: 1.957GB (+1.963GB cache, +1.159GB misc)
[2024-08-02 03:39:46,199] [INFO] [axolotl.load_model:791] [PID:218801] [RANK:2] GPU memory usage after model load: 1.957GB (+1.963GB cache, +1.159GB misc)
[2024-08-02 03:39:46,200] [INFO] [axolotl.load_model:791] [PID:218802] [RANK:3] GPU memory usage after model load: 1.957GB (+1.963GB cache, +1.159GB misc)
[2024-08-02 03:39:46,200] [INFO] [axolotl.load_model:791] [PID:218803] [RANK:4] GPU memory usage after model load: 1.957GB (+1.963GB cache, +1.136GB misc)
[2024-08-02 03:39:46,200] [INFO] [axolotl.load_model:856] [PID:218799] [RANK:0] converting modules to torch.bfloat16 for flash attention
[2024-08-02 03:39:46,201] [INFO] [axolotl.load_model:856] [PID:218800] [RANK:1] converting modules to torch.bfloat16 for flash attention
[2024-08-02 03:39:46,202] [INFO] [axolotl.load_model:856] [PID:218801] [RANK:2] converting modules to torch.bfloat16 for flash attention
[2024-08-02 03:39:46,202] [INFO] [axolotl.load_model:856] [PID:218802] [RANK:3] converting modules to torch.bfloat16 for flash attention
[2024-08-02 03:39:46,202] [INFO] [axolotl.load_model:856] [PID:218803] [RANK:4] converting modules to torch.bfloat16 for flash attention
[2024-08-02 03:39:46,783] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.embed_tokens.weight
[2024-08-02 03:39:46,784] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.0.self_attn.q_proj.weight
[2024-08-02 03:39:46,784] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.0.mlp.gate_proj.weight
[2024-08-02 03:39:46,784] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.0.mlp.up_proj.weight
[2024-08-02 03:39:46,784] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.0.mlp.down_proj.weight
[2024-08-02 03:39:46,784] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.0.input_layernorm.weight
[2024-08-02 03:39:46,784] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.0.post_attention_layernorm.weight
[2024-08-02 03:39:46,784] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.1.self_attn.q_proj.weight
[2024-08-02 03:39:46,784] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.1.self_attn.v_proj.weight
[2024-08-02 03:39:46,784] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.1.mlp.gate_proj.weight
[2024-08-02 03:39:46,784] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.1.mlp.up_proj.weight
[2024-08-02 03:39:46,784] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.1.mlp.down_proj.weight
[2024-08-02 03:39:46,784] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.1.input_layernorm.weight
[2024-08-02 03:39:46,784] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.1.post_attention_layernorm.weight
[2024-08-02 03:39:46,784] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.2.mlp.gate_proj.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.2.mlp.up_proj.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.2.mlp.down_proj.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.2.input_layernorm.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.2.post_attention_layernorm.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.3.self_attn.v_proj.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.3.mlp.gate_proj.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.3.mlp.up_proj.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.3.mlp.down_proj.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.3.input_layernorm.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.3.post_attention_layernorm.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.4.self_attn.v_proj.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.4.mlp.gate_proj.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.4.mlp.up_proj.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.4.mlp.down_proj.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.4.input_layernorm.weight
[2024-08-02 03:39:46,785] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.4.post_attention_layernorm.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.5.self_attn.q_proj.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.5.self_attn.o_proj.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.5.mlp.gate_proj.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.5.mlp.up_proj.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.5.mlp.down_proj.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.5.input_layernorm.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.5.post_attention_layernorm.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.6.self_attn.q_proj.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.6.self_attn.v_proj.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.6.self_attn.o_proj.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.6.mlp.up_proj.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.6.mlp.down_proj.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.6.input_layernorm.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.6.post_attention_layernorm.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.7.self_attn.q_proj.weight
[2024-08-02 03:39:46,786] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.7.self_attn.o_proj.weight
[2024-08-02 03:39:46,787] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.7.mlp.up_proj.weight
[2024-08-02 03:39:46,787] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.7.input_layernorm.weight
[2024-08-02 03:39:46,787] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.7.post_attention_layernorm.weight
[2024-08-02 03:39:46,787] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.8.self_attn.q_proj.weight
[2024-08-02 03:39:46,787] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.8.self_attn.o_proj.weight
[2024-08-02 03:39:46,787] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.8.mlp.up_proj.weight
[2024-08-02 03:39:46,787] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.8.input_layernorm.weight
[2024-08-02 03:39:46,787] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.8.post_attention_layernorm.weight
[2024-08-02 03:39:46,787] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.9.self_attn.q_proj.weight
[2024-08-02 03:39:46,787] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.9.self_attn.o_proj.weight
[2024-08-02 03:39:46,787] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.9.mlp.up_proj.weight
[2024-08-02 03:39:46,787] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.9.input_layernorm.weight
[2024-08-02 03:39:46,787] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.9.post_attention_layernorm.weight
[2024-08-02 03:39:46,787] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.10.self_attn.q_proj.weight
[2024-08-02 03:39:46,788] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.10.self_attn.o_proj.weight
[2024-08-02 03:39:46,788] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.10.input_layernorm.weight
[2024-08-02 03:39:46,788] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.10.post_attention_layernorm.weight
[2024-08-02 03:39:46,788] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.11.self_attn.q_proj.weight
[2024-08-02 03:39:46,788] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.11.self_attn.k_proj.weight
[2024-08-02 03:39:46,788] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.11.self_attn.o_proj.weight
[2024-08-02 03:39:46,788] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.11.input_layernorm.weight
[2024-08-02 03:39:46,788] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.11.post_attention_layernorm.weight
[2024-08-02 03:39:46,788] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.12.self_attn.q_proj.weight
[2024-08-02 03:39:46,788] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.12.self_attn.o_proj.weight
[2024-08-02 03:39:46,788] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.12.mlp.up_proj.weight
[2024-08-02 03:39:46,788] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.12.input_layernorm.weight
[2024-08-02 03:39:46,788] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.12.post_attention_layernorm.weight
[2024-08-02 03:39:46,789] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.13.self_attn.q_proj.weight
[2024-08-02 03:39:46,789] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.13.self_attn.o_proj.weight
[2024-08-02 03:39:46,789] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.13.mlp.up_proj.weight
[2024-08-02 03:39:46,789] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.13.input_layernorm.weight
[2024-08-02 03:39:46,789] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.13.post_attention_layernorm.weight
[2024-08-02 03:39:46,789] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.14.self_attn.q_proj.weight
[2024-08-02 03:39:46,789] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.14.self_attn.k_proj.weight
[2024-08-02 03:39:46,789] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.14.self_attn.v_proj.weight
[2024-08-02 03:39:46,789] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.14.self_attn.o_proj.weight
[2024-08-02 03:39:46,789] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.14.mlp.up_proj.weight
[2024-08-02 03:39:46,789] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.14.input_layernorm.weight
[2024-08-02 03:39:46,789] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.14.post_attention_layernorm.weight
[2024-08-02 03:39:46,789] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.15.self_attn.q_proj.weight
[2024-08-02 03:39:46,789] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.15.self_attn.v_proj.weight
[2024-08-02 03:39:46,789] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.15.self_attn.o_proj.weight
[2024-08-02 03:39:46,790] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.15.mlp.up_proj.weight
[2024-08-02 03:39:46,790] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.15.input_layernorm.weight
[2024-08-02 03:39:46,790] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.15.post_attention_layernorm.weight
[2024-08-02 03:39:46,790] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.16.self_attn.q_proj.weight
[2024-08-02 03:39:46,790] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.16.self_attn.v_proj.weight
[2024-08-02 03:39:46,790] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.16.self_attn.o_proj.weight
[2024-08-02 03:39:46,790] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.17.self_attn.q_proj.weight
[2024-08-02 03:39:46,790] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.17.self_attn.k_proj.weight
[2024-08-02 03:39:46,790] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.17.self_attn.v_proj.weight
[2024-08-02 03:39:46,790] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.17.mlp.up_proj.weight
[2024-08-02 03:39:46,790] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.17.mlp.down_proj.weight
[2024-08-02 03:39:46,790] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.18.self_attn.k_proj.weight
[2024-08-02 03:39:46,791] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.19.self_attn.k_proj.weight
[2024-08-02 03:39:46,791] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.19.mlp.up_proj.weight
[2024-08-02 03:39:46,791] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.19.mlp.down_proj.weight
[2024-08-02 03:39:46,791] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.20.self_attn.k_proj.weight
[2024-08-02 03:39:46,791] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.20.self_attn.v_proj.weight
[2024-08-02 03:39:46,791] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.20.mlp.gate_proj.weight
[2024-08-02 03:39:46,791] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.20.mlp.down_proj.weight
[2024-08-02 03:39:46,791] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.21.self_attn.k_proj.weight
[2024-08-02 03:39:46,791] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.21.self_attn.v_proj.weight
[2024-08-02 03:39:46,791] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.21.self_attn.o_proj.weight
[2024-08-02 03:39:46,792] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.21.mlp.gate_proj.weight
[2024-08-02 03:39:46,792] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.21.mlp.down_proj.weight
[2024-08-02 03:39:46,792] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.22.self_attn.k_proj.weight
[2024-08-02 03:39:46,792] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.22.self_attn.v_proj.weight
[2024-08-02 03:39:46,792] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.22.mlp.gate_proj.weight
[2024-08-02 03:39:46,792] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.22.mlp.down_proj.weight
[2024-08-02 03:39:46,792] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.23.self_attn.k_proj.weight
[2024-08-02 03:39:46,792] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.23.self_attn.v_proj.weight
[2024-08-02 03:39:46,792] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.23.self_attn.o_proj.weight
[2024-08-02 03:39:46,792] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.23.mlp.gate_proj.weight
[2024-08-02 03:39:46,792] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.23.mlp.down_proj.weight
[2024-08-02 03:39:46,792] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.24.self_attn.k_proj.weight
[2024-08-02 03:39:46,793] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.24.self_attn.o_proj.weight
[2024-08-02 03:39:46,793] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.24.mlp.gate_proj.weight
[2024-08-02 03:39:46,793] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.25.self_attn.k_proj.weight
[2024-08-02 03:39:46,793] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.25.self_attn.v_proj.weight
[2024-08-02 03:39:46,793] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.25.self_attn.o_proj.weight
[2024-08-02 03:39:46,793] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.25.mlp.gate_proj.weight
[2024-08-02 03:39:46,793] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.26.self_attn.q_proj.weight
[2024-08-02 03:39:46,793] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.26.self_attn.k_proj.weight
[2024-08-02 03:39:46,793] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.26.self_attn.v_proj.weight
[2024-08-02 03:39:46,793] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.26.mlp.gate_proj.weight
[2024-08-02 03:39:46,793] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.27.self_attn.k_proj.weight
[2024-08-02 03:39:46,794] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.27.mlp.gate_proj.weight
[2024-08-02 03:39:46,794] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.28.self_attn.k_proj.weight
[2024-08-02 03:39:46,794] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.28.self_attn.v_proj.weight
[2024-08-02 03:39:46,794] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.28.mlp.gate_proj.weight
[2024-08-02 03:39:46,794] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.29.self_attn.k_proj.weight
[2024-08-02 03:39:46,794] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.29.self_attn.v_proj.weight
[2024-08-02 03:39:46,794] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.29.mlp.gate_proj.weight
[2024-08-02 03:39:46,794] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.29.mlp.down_proj.weight
[2024-08-02 03:39:46,795] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.30.mlp.down_proj.weight
[2024-08-02 03:39:46,795] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.31.self_attn.k_proj.weight
[2024-08-02 03:39:46,795] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen model.layers.31.mlp.down_proj.weight
[2024-08-02 03:39:46,795] [DEBUG] [axolotl.utils.freeze.freeze_layers_except:55] [PID:218799] [RANK:0] Unfrozen lm_head.weight
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/training_args.py:1525: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/training_args.py:1525: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/training_args.py:1525: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/training_args.py:1525: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/training_args.py:1525: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/training_args.py:1525: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/training_args.py:1525: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/training_args.py:1525: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
[2024-08-02 03:39:47,038] [INFO] [axolotl.train.train:173] [PID:218799] [RANK:0] Starting trainer...
[2024-08-02 03:39:47,370] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,389] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,444] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,458] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,468] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,477] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,492] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,512] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,557] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,569] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,583] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,594] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,602] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,602] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,616] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,635] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,682] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,693] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,707] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,719] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,726] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,727] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,739] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,759] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,807] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,817] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,832] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,844] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,850] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,851] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,930] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:47,971] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
Parameter Offload: Total persistent parameters: 266240 in 65 params
Warning: The following arguments do not match the ones in the `trainer_state.json` within the checkpoint directory: 
	eval_steps: 0.16666666666666666 (from args) != 112 (from trainer_state.json)
	save_steps: 0.16666666666666666 (from args) != 56 (from trainer_state.json)
Warning: The following arguments do not match the ones in the `trainer_state.json` within the checkpoint directory: 
	eval_steps: 0.16666666666666666 (from args) != 112 (from trainer_state.json)
	save_steps: 0.16666666666666666 (from args) != 56 (from trainer_state.json)
Warning: The following arguments do not match the ones in the `trainer_state.json` within the checkpoint directory: 
	eval_steps: 0.16666666666666666 (from args) != 112 (from trainer_state.json)
	save_steps: 0.16666666666666666 (from args) != 56 (from trainer_state.json)
Warning: The following arguments do not match the ones in the `trainer_state.json` within the checkpoint directory: 
	eval_steps: 0.16666666666666666 (from args) != 112 (from trainer_state.json)
	save_steps: 0.16666666666666666 (from args) != 56 (from trainer_state.json)
Warning: The following arguments do not match the ones in the `trainer_state.json` within the checkpoint directory: 
	eval_steps: 0.16666666666666666 (from args) != 112 (from trainer_state.json)
	save_steps: 0.16666666666666666 (from args) != 56 (from trainer_state.json)
Warning: The following arguments do not match the ones in the `trainer_state.json` within the checkpoint directory: 
	eval_steps: 0.16666666666666666 (from args) != 112 (from trainer_state.json)
	save_steps: 0.16666666666666666 (from args) != 56 (from trainer_state.json)
Warning: The following arguments do not match the ones in the `trainer_state.json` within the checkpoint directory: 
	eval_steps: 0.16666666666666666 (from args) != 112 (from trainer_state.json)
	save_steps: 0.16666666666666666 (from args) != 56 (from trainer_state.json)
Warning: The following arguments do not match the ones in the `trainer_state.json` within the checkpoint directory: 
	eval_steps: 0.16666666666666666 (from args) != 112 (from trainer_state.json)
	save_steps: 0.16666666666666666 (from args) != 56 (from trainer_state.json)
wandb: WARNING The `run_name` is currently set to the same value as `TrainingArguments.output_dir`. If this was not intended, please specify a different run name by setting the `TrainingArguments.run_name` parameter.
[2024-08-02 03:39:58,065] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:58,068] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:58,068] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:58,069] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:58,069] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:58,069] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:58,069] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:58,189] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:58,193] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:58,197] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:58,198] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:58,198] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:58,198] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:39:58,198] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
wandb: Currently logged in as: therealagi (therealagi-reviewramp). Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.17.5
wandb: Run data is saved locally in /root/wandb/run-20240802_033958-svh2equ1
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run llama_3.1_8b_function_calling
wandb: ⭐️ View project at https://wandb.ai/therealagi-reviewramp/function-calling-spectrum
wandb: 🚀 View run at https://wandb.ai/therealagi-reviewramp/function-calling-spectrum/runs/svh2equ1
wandb: WARNING Saving files without folders. If you want to preserve subdirectories pass base_path to wandb.save, i.e. wandb.save("/mnt/folder/file.h5", base_path="/mnt")
[2024-08-02 03:39:59,840] [INFO] [axolotl.callbacks.on_train_begin:785] [PID:218799] [RANK:0] The Axolotl config has been saved to the WandB run under files.

  0%|          | 0/669 [00:00<?, ?it/s][2024-08-02 03:39:59,963] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 03:40:00,082] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
[2024-08-02 03:40:47,392] [WARNING] [stage3.py:2069:step] 2 pytorch allocator cache flushes since last step. this happens when there is high memory pressure and is detrimental to performance. if this is happening frequently consider adjusting settings to reduce memory consumption. If you are unable to make the cache flushes go away consider adding get_accelerator().empty_cache() calls in your training loop to ensure that all ranks flush their caches at the same time
[2024-08-02 03:40:47,396] [INFO] [axolotl.callbacks.on_step_end:128] [PID:218800] [RANK:1] GPU memory usage while training: 7.450GB (+33.312GB cache, +1.255GB misc)
[2024-08-02 03:40:47,396] [INFO] [axolotl.callbacks.on_step_end:128] [PID:218802] [RANK:3] GPU memory usage while training: 7.450GB (+33.312GB cache, +1.255GB misc)
[2024-08-02 03:40:47,396] [INFO] [axolotl.callbacks.on_step_end:128] [PID:218805] [RANK:6] GPU memory usage while training: 7.450GB (+33.312GB cache, +1.251GB misc)
[2024-08-02 03:40:47,396] [INFO] [axolotl.callbacks.on_step_end:128] [PID:218806] [RANK:7] GPU memory usage while training: 7.450GB (+33.312GB cache, +1.204GB misc)
[2024-08-02 03:40:47,396] [INFO] [axolotl.callbacks.on_step_end:128] [PID:218804] [RANK:5] GPU memory usage while training: 7.450GB (+33.624GB cache, +1.204GB misc)
[2024-08-02 03:40:47,396] [INFO] [axolotl.callbacks.on_step_end:128] [PID:218801] [RANK:2] GPU memory usage while training: 7.450GB (+33.312GB cache, +1.255GB misc)
[2024-08-02 03:40:47,396] [INFO] [axolotl.callbacks.on_step_end:128] [PID:218803] [RANK:4] GPU memory usage while training: 7.450GB (+33.312GB cache, +1.208GB misc)
[2024-08-02 03:40:47,397] [INFO] [axolotl.callbacks.on_step_end:128] [PID:218799] [RANK:0] GPU memory usage while training: 7.450GB (+33.312GB cache, +1.282GB misc)

 17%|█▋        | 113/669 [00:47<03:53,  2.38it/s]
                                                 
{'loss': 0.515, 'grad_norm': 0.5662939933016762, 'learning_rate': 4.218512898330804e-06, 'epoch': 0.51}

 17%|█▋        | 113/669 [00:47<03:53,  2.38it/s]
 17%|█▋        | 114/669 [01:31<08:54,  1.04it/s]
                                                 
{'loss': 0.5038, 'grad_norm': 0.5225180760232837, 'learning_rate': 4.2109256449165405e-06, 'epoch': 0.51}

 17%|█▋        | 114/669 [01:31<08:54,  1.04it/s]
 17%|█▋        | 115/669 [02:15<15:50,  1.72s/it]
                                                 
{'loss': 0.5043, 'grad_norm': 0.5377410752157915, 'learning_rate': 4.203338391502276e-06, 'epoch': 0.51}

 17%|█▋        | 115/669 [02:15<15:50,  1.72s/it]
 17%|█▋        | 116/669 [02:59<25:19,  2.75s/it]
                                                 
{'loss': 0.5115, 'grad_norm': 0.5915094303426857, 'learning_rate': 4.195751138088012e-06, 'epoch': 0.52}

 17%|█▋        | 116/669 [02:59<25:19,  2.75s/it]
 17%|█▋        | 117/669 [03:43<38:03,  4.14s/it]
                                                 
{'loss': 0.5491, 'grad_norm': 0.5079762717076601, 'learning_rate': 4.1881638846737485e-06, 'epoch': 0.52}

 17%|█▋        | 117/669 [03:43<38:03,  4.14s/it]
 18%|█▊        | 118/669 [04:27<54:51,  5.97s/it]
                                                 
{'loss': 0.5095, 'grad_norm': 0.5933329970849119, 'learning_rate': 4.180576631259484e-06, 'epoch': 0.53}

 18%|█▊        | 118/669 [04:27<54:51,  5.97s/it]
 18%|█▊        | 119/669 [05:11<1:16:13,  8.31s/it]
                                                   
{'loss': 0.4881, 'grad_norm': 0.5053074762991974, 'learning_rate': 4.17298937784522e-06, 'epoch': 0.53}

 18%|█▊        | 119/669 [05:11<1:16:13,  8.31s/it]
 18%|█▊        | 120/669 [05:55<1:42:27, 11.20s/it]
                                                   
{'loss': 0.5069, 'grad_norm': 0.6515342691068152, 'learning_rate': 4.1654021244309564e-06, 'epoch': 0.54}

 18%|█▊        | 120/669 [05:55<1:42:27, 11.20s/it]
 18%|█▊        | 121/669 [06:39<2:13:14, 14.59s/it]
                                                   
{'loss': 0.5007, 'grad_norm': 0.5774345075195761, 'learning_rate': 4.157814871016692e-06, 'epoch': 0.54}

 18%|█▊        | 121/669 [06:39<2:13:14, 14.59s/it]
 18%|█▊        | 122/669 [07:23<2:47:31, 18.38s/it]
                                                   
{'loss': 0.5083, 'grad_norm': 0.6308991657157696, 'learning_rate': 4.150227617602428e-06, 'epoch': 0.55}

 18%|█▊        | 122/669 [07:23<2:47:31, 18.38s/it]
 18%|█▊        | 123/669 [08:07<3:23:28, 22.36s/it]
                                                   
{'loss': 0.5103, 'grad_norm': 0.5258198371155504, 'learning_rate': 4.142640364188164e-06, 'epoch': 0.55}

 18%|█▊        | 123/669 [08:07<3:23:28, 22.36s/it]
 19%|█▊        | 124/669 [08:51<3:58:40, 26.28s/it]
                                                   
{'loss': 0.5016, 'grad_norm': 0.5629243370057063, 'learning_rate': 4.1350531107739e-06, 'epoch': 0.55}

 19%|█▊        | 124/669 [08:51<3:58:40, 26.28s/it]
 19%|█▊        | 125/669 [09:35<4:31:09, 29.91s/it]
                                                   
{'loss': 0.491, 'grad_norm': 0.5440095922230602, 'learning_rate': 4.127465857359636e-06, 'epoch': 0.56}

 19%|█▊        | 125/669 [09:35<4:31:09, 29.91s/it]
 19%|█▉        | 126/669 [10:18<4:59:27, 33.09s/it]
                                                   
{'loss': 0.5239, 'grad_norm': 0.5126793634050918, 'learning_rate': 4.119878603945372e-06, 'epoch': 0.56}

 19%|█▉        | 126/669 [10:18<4:59:27, 33.09s/it]
 19%|█▉        | 127/669 [11:02<5:22:55, 35.75s/it]
                                                   
{'loss': 0.5176, 'grad_norm': 0.5476299968831635, 'learning_rate': 4.112291350531108e-06, 'epoch': 0.57}

 19%|█▉        | 127/669 [11:02<5:22:55, 35.75s/it]
 19%|█▉        | 128/669 [11:46<5:41:20, 37.86s/it]
                                                   
{'loss': 0.5084, 'grad_norm': 0.5570481457317527, 'learning_rate': 4.104704097116844e-06, 'epoch': 0.57}

 19%|█▉        | 128/669 [11:46<5:41:20, 37.86s/it]
 19%|█▉        | 129/669 [12:30<5:55:33, 39.51s/it]
                                                   
{'loss': 0.4966, 'grad_norm': 0.49480777896006917, 'learning_rate': 4.09711684370258e-06, 'epoch': 0.58}

 19%|█▉        | 129/669 [12:30<5:55:33, 39.51s/it]
 19%|█▉        | 130/669 [13:14<6:06:00, 40.74s/it]
                                                   
{'loss': 0.5059, 'grad_norm': 0.5089889598386482, 'learning_rate': 4.089529590288316e-06, 'epoch': 0.58}

 19%|█▉        | 130/669 [13:14<6:06:00, 40.74s/it]
 20%|█▉        | 131/669 [13:58<6:13:26, 41.65s/it]
                                                   
{'loss': 0.5166, 'grad_norm': 0.4954780811768282, 'learning_rate': 4.081942336874052e-06, 'epoch': 0.59}

 20%|█▉        | 131/669 [13:58<6:13:26, 41.65s/it]
 20%|█▉        | 132/669 [14:42<6:18:35, 42.30s/it]
                                                   
{'loss': 0.4857, 'grad_norm': 0.4809684003541764, 'learning_rate': 4.074355083459787e-06, 'epoch': 0.59}

 20%|█▉        | 132/669 [14:42<6:18:35, 42.30s/it]
 20%|█▉        | 133/669 [15:26<6:22:05, 42.77s/it]
                                                   
{'loss': 0.523, 'grad_norm': 0.5009748980192239, 'learning_rate': 4.066767830045524e-06, 'epoch': 0.59}

 20%|█▉        | 133/669 [15:26<6:22:05, 42.77s/it]
 20%|██        | 134/669 [16:10<6:24:35, 43.13s/it]
                                                   
{'loss': 0.4956, 'grad_norm': 0.48320950023108783, 'learning_rate': 4.05918057663126e-06, 'epoch': 0.6}

 20%|██        | 134/669 [16:10<6:24:35, 43.13s/it]
 20%|██        | 135/669 [16:54<6:25:55, 43.36s/it]
                                                   
{'loss': 0.5163, 'grad_norm': 0.49399181892388616, 'learning_rate': 4.051593323216995e-06, 'epoch': 0.6}

 20%|██        | 135/669 [16:54<6:25:55, 43.36s/it]
 20%|██        | 136/669 [17:38<6:26:46, 43.54s/it]
                                                   
{'loss': 0.4981, 'grad_norm': 0.4821527638840584, 'learning_rate': 4.044006069802732e-06, 'epoch': 0.61}

 20%|██        | 136/669 [17:38<6:26:46, 43.54s/it]
 20%|██        | 137/669 [19:33<9:35:07, 64.86s/it]
                                                   
{'loss': 0.5121, 'grad_norm': 0.5099314085164022, 'learning_rate': 4.036418816388468e-06, 'epoch': 0.61}

 20%|██        | 137/669 [19:33<9:35:07, 64.86s/it]
 21%|██        | 138/669 [20:17<8:38:17, 58.56s/it]
                                                   
{'loss': 0.5074, 'grad_norm': 0.5323495276796993, 'learning_rate': 4.028831562974203e-06, 'epoch': 0.62}

 21%|██        | 138/669 [20:17<8:38:17, 58.56s/it]
 21%|██        | 139/669 [21:01<7:58:43, 54.20s/it]
                                                   
{'loss': 0.52, 'grad_norm': 0.48851533865778524, 'learning_rate': 4.02124430955994e-06, 'epoch': 0.62}

 21%|██        | 139/669 [21:01<7:58:43, 54.20s/it]
 21%|██        | 140/669 [21:44<7:30:46, 51.13s/it]
                                                   
{'loss': 0.5039, 'grad_norm': 0.5281525574471102, 'learning_rate': 4.0136570561456756e-06, 'epoch': 0.63}

 21%|██        | 140/669 [21:44<7:30:46, 51.13s/it]
 21%|██        | 141/669 [22:28<7:10:57, 48.97s/it]
                                                   
{'loss': 0.4933, 'grad_norm': 0.5279406819094602, 'learning_rate': 4.006069802731411e-06, 'epoch': 0.63}

 21%|██        | 141/669 [22:28<7:10:57, 48.97s/it]
 21%|██        | 142/669 [23:12<6:56:56, 47.47s/it]
                                                   
{'loss': 0.5151, 'grad_norm': 0.45822529010842367, 'learning_rate': 3.998482549317148e-06, 'epoch': 0.63}

 21%|██        | 142/669 [23:12<6:56:56, 47.47s/it]
 21%|██▏       | 143/669 [23:56<6:46:48, 46.40s/it]
                                                   
{'loss': 0.4983, 'grad_norm': 0.5349296018392958, 'learning_rate': 3.9908952959028835e-06, 'epoch': 0.64}

 21%|██▏       | 143/669 [23:56<6:46:48, 46.40s/it]
 22%|██▏       | 144/669 [24:40<6:39:35, 45.67s/it]
                                                   
{'loss': 0.5344, 'grad_norm': 0.5316574655817645, 'learning_rate': 3.983308042488619e-06, 'epoch': 0.64}

 22%|██▏       | 144/669 [24:40<6:39:35, 45.67s/it]
 22%|██▏       | 145/669 [25:24<6:34:16, 45.15s/it]
                                                   
{'loss': 0.52, 'grad_norm': 0.5107562456487705, 'learning_rate': 3.975720789074356e-06, 'epoch': 0.65}

 22%|██▏       | 145/669 [25:24<6:34:16, 45.15s/it]
 22%|██▏       | 146/669 [26:08<6:30:31, 44.80s/it]
                                                   
{'loss': 0.4923, 'grad_norm': 0.5072583855988344, 'learning_rate': 3.9681335356600915e-06, 'epoch': 0.65}

 22%|██▏       | 146/669 [26:08<6:30:31, 44.80s/it]
 22%|██▏       | 147/669 [26:52<6:27:25, 44.53s/it]
                                                   
{'loss': 0.5295, 'grad_norm': 0.5836000010606872, 'learning_rate': 3.960546282245827e-06, 'epoch': 0.66}

 22%|██▏       | 147/669 [26:52<6:27:25, 44.53s/it]
 22%|██▏       | 148/669 [27:36<6:25:07, 44.35s/it]
                                                   
{'loss': 0.5216, 'grad_norm': 0.49884721978842206, 'learning_rate': 3.952959028831564e-06, 'epoch': 0.66}

 22%|██▏       | 148/669 [27:36<6:25:07, 44.35s/it]
 22%|██▏       | 149/669 [28:20<6:23:23, 44.24s/it]
                                                   
{'loss': 0.5414, 'grad_norm': 0.5120991222561042, 'learning_rate': 3.945371775417299e-06, 'epoch': 0.67}

 22%|██▏       | 149/669 [28:20<6:23:23, 44.24s/it]
 22%|██▏       | 150/669 [29:04<6:22:02, 44.17s/it]
                                                   
{'loss': 0.5381, 'grad_norm': 0.49838481084559894, 'learning_rate': 3.937784522003035e-06, 'epoch': 0.67}

 22%|██▏       | 150/669 [29:04<6:22:02, 44.17s/it]
 23%|██▎       | 151/669 [29:48<6:20:36, 44.09s/it]
                                                   
{'loss': 0.4896, 'grad_norm': 0.5398799415397277, 'learning_rate': 3.930197268588772e-06, 'epoch': 0.67}

 23%|██▎       | 151/669 [29:48<6:20:36, 44.09s/it]
 23%|██▎       | 152/669 [30:32<6:19:27, 44.04s/it]
                                                   
{'loss': 0.4985, 'grad_norm': 0.565748454568293, 'learning_rate': 3.922610015174507e-06, 'epoch': 0.68}

 23%|██▎       | 152/669 [30:32<6:19:27, 44.04s/it]
 23%|██▎       | 153/669 [31:16<6:18:33, 44.02s/it]
                                                   
{'loss': 0.5125, 'grad_norm': 0.5986782671726579, 'learning_rate': 3.915022761760243e-06, 'epoch': 0.68}

 23%|██▎       | 153/669 [31:16<6:18:33, 44.02s/it]
 23%|██▎       | 154/669 [32:00<6:17:39, 44.00s/it]
                                                   
{'loss': 0.5078, 'grad_norm': 0.5290750194980306, 'learning_rate': 3.907435508345979e-06, 'epoch': 0.69}

 23%|██▎       | 154/669 [32:00<6:17:39, 44.00s/it]
 23%|██▎       | 155/669 [32:44<6:16:46, 43.98s/it]
                                                   
{'loss': 0.5205, 'grad_norm': 0.5706479438251948, 'learning_rate': 3.899848254931715e-06, 'epoch': 0.69}

 23%|██▎       | 155/669 [32:44<6:16:46, 43.98s/it]
 23%|██▎       | 156/669 [33:28<6:15:55, 43.97s/it]
                                                   
{'loss': 0.4988, 'grad_norm': 0.5173864500214489, 'learning_rate': 3.892261001517451e-06, 'epoch': 0.7}

 23%|██▎       | 156/669 [33:28<6:15:55, 43.97s/it]
 23%|██▎       | 157/669 [34:11<6:15:00, 43.95s/it]
                                                   
{'loss': 0.4954, 'grad_norm': 0.4920045809108581, 'learning_rate': 3.884673748103187e-06, 'epoch': 0.7}

 23%|██▎       | 157/669 [34:11<6:15:00, 43.95s/it]
 24%|██▎       | 158/669 [34:55<6:14:15, 43.94s/it]
                                                   
{'loss': 0.5253, 'grad_norm': 0.5237901072069291, 'learning_rate': 3.877086494688923e-06, 'epoch': 0.71}

 24%|██▎       | 158/669 [34:55<6:14:15, 43.94s/it]
 24%|██▍       | 159/669 [35:39<6:13:36, 43.95s/it]
                                                   
{'loss': 0.5029, 'grad_norm': 0.5212795514388354, 'learning_rate': 3.869499241274659e-06, 'epoch': 0.71}

 24%|██▍       | 159/669 [35:39<6:13:36, 43.95s/it]
 24%|██▍       | 160/669 [36:23<6:12:51, 43.95s/it]
                                                   
{'loss': 0.5023, 'grad_norm': 0.48658569533086143, 'learning_rate': 3.861911987860395e-06, 'epoch': 0.72}

 24%|██▍       | 160/669 [36:23<6:12:51, 43.95s/it]
 24%|██▍       | 161/669 [37:07<6:12:19, 43.98s/it]
                                                   
{'loss': 0.5436, 'grad_norm': 0.5145837815914728, 'learning_rate': 3.854324734446131e-06, 'epoch': 0.72}

 24%|██▍       | 161/669 [37:07<6:12:19, 43.98s/it]
 24%|██▍       | 162/669 [39:03<9:13:05, 65.46s/it]
                                                   
{'loss': 0.4948, 'grad_norm': 0.49182827281373437, 'learning_rate': 3.846737481031867e-06, 'epoch': 0.72}

 24%|██▍       | 162/669 [39:03<9:13:05, 65.46s/it]
 24%|██▍       | 163/669 [39:47<8:17:05, 58.94s/it]
                                                   
{'loss': 0.4778, 'grad_norm': 0.5113640333759738, 'learning_rate': 3.839150227617603e-06, 'epoch': 0.73}

 24%|██▍       | 163/669 [39:47<8:17:05, 58.94s/it]
 25%|██▍       | 164/669 [40:31<7:38:15, 54.45s/it]
                                                   
{'loss': 0.5182, 'grad_norm': 0.47675402744999507, 'learning_rate': 3.831562974203339e-06, 'epoch': 0.73}

 25%|██▍       | 164/669 [40:31<7:38:15, 54.45s/it]
 25%|██▍       | 165/669 [41:15<7:10:47, 51.29s/it]
                                                   
{'loss': 0.5204, 'grad_norm': 0.5214515656344721, 'learning_rate': 3.823975720789075e-06, 'epoch': 0.74}

 25%|██▍       | 165/669 [41:15<7:10:47, 51.29s/it]
 25%|██▍       | 166/669 [41:58<6:51:25, 49.08s/it]
                                                   
{'loss': 0.492, 'grad_norm': 0.5130606167326404, 'learning_rate': 3.816388467374811e-06, 'epoch': 0.74}

 25%|██▍       | 166/669 [41:58<6:51:25, 49.08s/it]
 25%|██▍       | 167/669 [42:43<6:37:57, 47.57s/it]
                                                   
{'loss': 0.4997, 'grad_norm': 0.49678736596959705, 'learning_rate': 3.8088012139605467e-06, 'epoch': 0.75}

 25%|██▍       | 167/669 [42:43<6:37:57, 47.57s/it]
 25%|██▌       | 168/669 [43:26<6:27:53, 46.45s/it]
                                                   
{'loss': 0.5163, 'grad_norm': 0.5275415964748987, 'learning_rate': 3.801213960546283e-06, 'epoch': 0.75}

 25%|██▌       | 168/669 [43:26<6:27:53, 46.45s/it]/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(

 25%|██▌       | 169/669 [44:43<7:43:47, 55.65s/it]
                                                   
{'loss': 0.4846, 'grad_norm': 0.5323621998896121, 'learning_rate': 3.7936267071320185e-06, 'epoch': 0.76}

 25%|██▌       | 169/669 [44:43<7:43:47, 55.65s/it]
 25%|██▌       | 170/669 [45:27<7:13:36, 52.14s/it]
                                                   
{'loss': 0.5193, 'grad_norm': 0.5302675218909635, 'learning_rate': 3.7860394537177547e-06, 'epoch': 0.76}

 25%|██▌       | 170/669 [45:27<7:13:36, 52.14s/it]
 26%|██▌       | 171/669 [46:11<6:52:23, 49.69s/it]
                                                   
{'loss': 0.5248, 'grad_norm': 0.5097255453283106, 'learning_rate': 3.778452200303491e-06, 'epoch': 0.76}

 26%|██▌       | 171/669 [46:11<6:52:23, 49.69s/it]
 26%|██▌       | 172/669 [46:55<6:37:20, 47.97s/it]
                                                   
{'loss': 0.5217, 'grad_norm': 0.5221084637587756, 'learning_rate': 3.7708649468892265e-06, 'epoch': 0.77}

 26%|██▌       | 172/669 [46:55<6:37:20, 47.97s/it]
 26%|██▌       | 173/669 [47:39<6:26:30, 46.75s/it]
                                                   
{'loss': 0.5057, 'grad_norm': 0.5823307367451829, 'learning_rate': 3.7632776934749626e-06, 'epoch': 0.77}

 26%|██▌       | 173/669 [47:39<6:26:30, 46.75s/it]
 26%|██▌       | 174/669 [48:23<6:18:45, 45.91s/it]
                                                   
{'loss': 0.5372, 'grad_norm': 0.48586162675121547, 'learning_rate': 3.7556904400606987e-06, 'epoch': 0.78}

 26%|██▌       | 174/669 [48:23<6:18:45, 45.91s/it]
 26%|██▌       | 175/669 [49:07<6:13:14, 45.33s/it]
                                                   
{'loss': 0.4844, 'grad_norm': 0.5099958749272927, 'learning_rate': 3.748103186646434e-06, 'epoch': 0.78}

 26%|██▌       | 175/669 [49:07<6:13:14, 45.33s/it]
 26%|██▋       | 176/669 [49:51<6:09:10, 44.93s/it]
                                                   
{'loss': 0.4968, 'grad_norm': 0.5221594966770278, 'learning_rate': 3.74051593323217e-06, 'epoch': 0.79}

 26%|██▋       | 176/669 [49:51<6:09:10, 44.93s/it]
 26%|██▋       | 177/669 [50:35<6:06:07, 44.65s/it]
                                                   
{'loss': 0.4906, 'grad_norm': 0.5392129841421105, 'learning_rate': 3.732928679817906e-06, 'epoch': 0.79}

 26%|██▋       | 177/669 [50:35<6:06:07, 44.65s/it]
 27%|██▋       | 178/669 [51:19<6:03:49, 44.46s/it]
                                                   
{'loss': 0.5172, 'grad_norm': 0.5239629013619267, 'learning_rate': 3.725341426403642e-06, 'epoch': 0.8}

 27%|██▋       | 178/669 [51:19<6:03:49, 44.46s/it]
 27%|██▋       | 179/669 [52:03<6:01:39, 44.28s/it]
                                                   
{'loss': 0.4914, 'grad_norm': 0.537861417175095, 'learning_rate': 3.717754172989378e-06, 'epoch': 0.8}

 27%|██▋       | 179/669 [52:03<6:01:39, 44.28s/it]
 27%|██▋       | 180/669 [52:47<6:00:02, 44.18s/it]
                                                   
{'loss': 0.5014, 'grad_norm': 0.5001204419984062, 'learning_rate': 3.710166919575114e-06, 'epoch': 0.8}

 27%|██▋       | 180/669 [52:47<6:00:02, 44.18s/it]
 27%|██▋       | 181/669 [53:31<5:58:47, 44.11s/it]
                                                   
{'loss': 0.4979, 'grad_norm': 0.5243758405421232, 'learning_rate': 3.70257966616085e-06, 'epoch': 0.81}

 27%|██▋       | 181/669 [53:31<5:58:47, 44.11s/it]
 27%|██▋       | 182/669 [54:15<5:57:38, 44.06s/it]
                                                   
{'loss': 0.5003, 'grad_norm': 0.507560920806033, 'learning_rate': 3.6949924127465856e-06, 'epoch': 0.81}

 27%|██▋       | 182/669 [54:15<5:57:38, 44.06s/it]
 27%|██▋       | 183/669 [54:59<5:56:33, 44.02s/it]
                                                   
{'loss': 0.5011, 'grad_norm': 0.5623627396713589, 'learning_rate': 3.6874051593323218e-06, 'epoch': 0.82}

 27%|██▋       | 183/669 [54:59<5:56:33, 44.02s/it]
 28%|██▊       | 184/669 [55:43<5:55:38, 44.00s/it]
                                                   
{'loss': 0.5117, 'grad_norm': 0.5225320984203384, 'learning_rate': 3.679817905918058e-06, 'epoch': 0.82}

 28%|██▊       | 184/669 [55:43<5:55:38, 44.00s/it]
 28%|██▊       | 185/669 [56:27<5:54:42, 43.97s/it]
                                                   
{'loss': 0.5028, 'grad_norm': 0.5371208831947621, 'learning_rate': 3.6722306525037936e-06, 'epoch': 0.83}

 28%|██▊       | 185/669 [56:27<5:54:42, 43.97s/it]
 28%|██▊       | 186/669 [57:11<5:53:49, 43.95s/it]
                                                   
{'loss': 0.5111, 'grad_norm': 0.5412513689241548, 'learning_rate': 3.6646433990895297e-06, 'epoch': 0.83}

 28%|██▊       | 186/669 [57:11<5:53:49, 43.95s/it]
 28%|██▊       | 187/669 [59:08<8:49:12, 65.88s/it]
                                                   
{'loss': 0.5002, 'grad_norm': 0.49083593539874787, 'learning_rate': 3.657056145675266e-06, 'epoch': 0.84}

 28%|██▊       | 187/669 [59:08<8:49:12, 65.88s/it]
 28%|██▊       | 188/669 [59:51<7:55:00, 59.25s/it]
                                                   
{'loss': 0.4875, 'grad_norm': 0.5144646076400369, 'learning_rate': 3.6494688922610015e-06, 'epoch': 0.84}

 28%|██▊       | 188/669 [59:51<7:55:00, 59.25s/it]
 28%|██▊       | 189/669 [1:00:35<7:17:13, 54.65s/it]
                                                     
{'loss': 0.5072, 'grad_norm': 0.5385458846653849, 'learning_rate': 3.6418816388467377e-06, 'epoch': 0.84}

 28%|██▊       | 189/669 [1:00:35<7:17:13, 54.65s/it]
 28%|██▊       | 190/669 [1:01:19<6:50:40, 51.44s/it]
                                                     
{'loss': 0.549, 'grad_norm': 0.48088002744673064, 'learning_rate': 3.6342943854324738e-06, 'epoch': 0.85}

 28%|██▊       | 190/669 [1:01:19<6:50:40, 51.44s/it]
 29%|██▊       | 191/669 [1:02:03<6:31:49, 49.18s/it]
                                                     
{'loss': 0.5344, 'grad_norm': 0.563960197772806, 'learning_rate': 3.6267071320182095e-06, 'epoch': 0.85}

 29%|██▊       | 191/669 [1:02:03<6:31:49, 49.18s/it]
 29%|██▊       | 192/669 [1:02:47<6:18:41, 47.63s/it]
                                                     
{'loss': 0.5172, 'grad_norm': 0.5235289059912077, 'learning_rate': 3.6191198786039456e-06, 'epoch': 0.86}

 29%|██▊       | 192/669 [1:02:47<6:18:41, 47.63s/it]
 29%|██▉       | 193/669 [1:03:31<6:08:58, 46.51s/it]
                                                     
{'loss': 0.4908, 'grad_norm': 0.5059904029741168, 'learning_rate': 3.6115326251896813e-06, 'epoch': 0.86}

 29%|██▉       | 193/669 [1:03:31<6:08:58, 46.51s/it]
 29%|██▉       | 194/669 [1:04:15<6:02:18, 45.77s/it]
                                                     
{'loss': 0.4951, 'grad_norm': 0.507980780742306, 'learning_rate': 3.6039453717754174e-06, 'epoch': 0.87}

 29%|██▉       | 194/669 [1:04:15<6:02:18, 45.77s/it]
 29%|██▉       | 195/669 [1:04:59<5:57:18, 45.23s/it]
                                                     
{'loss': 0.505, 'grad_norm': 0.5552145890763519, 'learning_rate': 3.5963581183611536e-06, 'epoch': 0.87}

 29%|██▉       | 195/669 [1:04:59<5:57:18, 45.23s/it]
 29%|██▉       | 196/669 [1:05:43<5:53:28, 44.84s/it]
                                                     
{'loss': 0.52, 'grad_norm': 0.5379367693029108, 'learning_rate': 3.5887708649468893e-06, 'epoch': 0.88}

 29%|██▉       | 196/669 [1:05:43<5:53:28, 44.84s/it]
 29%|██▉       | 197/669 [1:06:27<5:50:31, 44.56s/it]
                                                     
{'loss': 0.4966, 'grad_norm': 0.5813281708032492, 'learning_rate': 3.5811836115326254e-06, 'epoch': 0.88}

 29%|██▉       | 197/669 [1:06:27<5:50:31, 44.56s/it]
 30%|██▉       | 198/669 [1:07:11<5:48:18, 44.37s/it]
                                                     
{'loss': 0.495, 'grad_norm': 0.5138017935214574, 'learning_rate': 3.5735963581183615e-06, 'epoch': 0.88}

 30%|██▉       | 198/669 [1:07:11<5:48:18, 44.37s/it]
 30%|██▉       | 199/669 [1:07:55<5:46:32, 44.24s/it]
                                                     
{'loss': 0.519, 'grad_norm': 0.5116405158423362, 'learning_rate': 3.5660091047040972e-06, 'epoch': 0.89}

 30%|██▉       | 199/669 [1:07:55<5:46:32, 44.24s/it]
 30%|██▉       | 200/669 [1:08:39<5:45:02, 44.14s/it]
                                                     
{'loss': 0.5002, 'grad_norm': 0.5696283061922035, 'learning_rate': 3.5584218512898333e-06, 'epoch': 0.89}

 30%|██▉       | 200/669 [1:08:39<5:45:02, 44.14s/it]
 30%|███       | 201/669 [1:09:23<5:43:44, 44.07s/it]
                                                     
{'loss': 0.5125, 'grad_norm': 0.5541549121266046, 'learning_rate': 3.5508345978755695e-06, 'epoch': 0.9}

 30%|███       | 201/669 [1:09:23<5:43:44, 44.07s/it]
 30%|███       | 202/669 [1:10:07<5:42:42, 44.03s/it]
                                                     
{'loss': 0.499, 'grad_norm': 0.5017863683272789, 'learning_rate': 3.543247344461305e-06, 'epoch': 0.9}

 30%|███       | 202/669 [1:10:07<5:42:42, 44.03s/it]
 30%|███       | 203/669 [1:10:51<5:41:46, 44.01s/it]
                                                     
{'loss': 0.4883, 'grad_norm': 0.5776107100347877, 'learning_rate': 3.5356600910470413e-06, 'epoch': 0.91}

 30%|███       | 203/669 [1:10:51<5:41:46, 44.01s/it]
 30%|███       | 204/669 [1:11:34<5:40:55, 43.99s/it]
                                                     
{'loss': 0.5052, 'grad_norm': 0.5350847562000018, 'learning_rate': 3.528072837632777e-06, 'epoch': 0.91}

 30%|███       | 204/669 [1:11:34<5:40:55, 43.99s/it]
 31%|███       | 205/669 [1:12:18<5:40:12, 43.99s/it]
                                                     
{'loss': 0.5422, 'grad_norm': 0.5012614076658582, 'learning_rate': 3.520485584218513e-06, 'epoch': 0.92}

 31%|███       | 205/669 [1:12:18<5:40:12, 43.99s/it]
 31%|███       | 206/669 [1:13:02<5:39:22, 43.98s/it]
                                                     
{'loss': 0.5204, 'grad_norm': 0.6107847324258918, 'learning_rate': 3.5128983308042493e-06, 'epoch': 0.92}

 31%|███       | 206/669 [1:13:02<5:39:22, 43.98s/it]
 31%|███       | 207/669 [1:13:46<5:38:31, 43.96s/it]
                                                     
{'loss': 0.5074, 'grad_norm': 0.49314942553653257, 'learning_rate': 3.505311077389985e-06, 'epoch': 0.93}

 31%|███       | 207/669 [1:13:46<5:38:31, 43.96s/it]
 31%|███       | 208/669 [1:14:30<5:37:40, 43.95s/it]
                                                     
{'loss': 0.4936, 'grad_norm': 0.5340833706807174, 'learning_rate': 3.497723823975721e-06, 'epoch': 0.93}

 31%|███       | 208/669 [1:14:30<5:37:40, 43.95s/it]
 31%|███       | 209/669 [1:15:14<5:36:49, 43.93s/it]
                                                     
{'loss': 0.4945, 'grad_norm': 0.5342137283430334, 'learning_rate': 3.490136570561457e-06, 'epoch': 0.93}

 31%|███       | 209/669 [1:15:14<5:36:49, 43.93s/it]
 31%|███▏      | 210/669 [1:15:58<5:36:17, 43.96s/it]
                                                     
{'loss': 0.4963, 'grad_norm': 0.5301769352005536, 'learning_rate': 3.482549317147193e-06, 'epoch': 0.94}

 31%|███▏      | 210/669 [1:15:58<5:36:17, 43.96s/it]
 32%|███▏      | 211/669 [1:16:42<5:35:33, 43.96s/it]
                                                     
{'loss': 0.4931, 'grad_norm': 0.5455545680194068, 'learning_rate': 3.474962063732929e-06, 'epoch': 0.94}

 32%|███▏      | 211/669 [1:16:42<5:35:33, 43.96s/it]
 32%|███▏      | 212/669 [1:18:39<8:20:59, 65.78s/it]
                                                     
{'loss': 0.4894, 'grad_norm': 0.5282021847913991, 'learning_rate': 3.467374810318665e-06, 'epoch': 0.95}

 32%|███▏      | 212/669 [1:18:39<8:20:59, 65.78s/it]
 32%|███▏      | 213/669 [1:19:23<7:29:41, 59.17s/it]
                                                     
{'loss': 0.5102, 'grad_norm': 0.5341400870442735, 'learning_rate': 3.459787556904401e-06, 'epoch': 0.95}

 32%|███▏      | 213/669 [1:19:23<7:29:41, 59.17s/it]
 32%|███▏      | 214/669 [1:20:06<6:53:59, 54.59s/it]
                                                     
{'loss': 0.501, 'grad_norm': 0.5313633005763995, 'learning_rate': 3.452200303490137e-06, 'epoch': 0.96}

 32%|███▏      | 214/669 [1:20:06<6:53:59, 54.59s/it]
 32%|███▏      | 215/669 [1:20:50<6:28:53, 51.40s/it]
                                                     
{'loss': 0.5021, 'grad_norm': 0.5774555510069799, 'learning_rate': 3.4446130500758727e-06, 'epoch': 0.96}

 32%|███▏      | 215/669 [1:20:50<6:28:53, 51.40s/it]
 32%|███▏      | 216/669 [1:21:34<6:11:00, 49.14s/it]
                                                     
{'loss': 0.5071, 'grad_norm': 0.521646351973494, 'learning_rate': 3.437025796661609e-06, 'epoch': 0.97}

 32%|███▏      | 216/669 [1:21:34<6:11:00, 49.14s/it]
 32%|███▏      | 217/669 [1:22:18<5:58:25, 47.58s/it]
                                                     
{'loss': 0.4993, 'grad_norm': 0.5335773786656196, 'learning_rate': 3.429438543247345e-06, 'epoch': 0.97}

 32%|███▏      | 217/669 [1:22:18<5:58:25, 47.58s/it]
 33%|███▎      | 218/669 [1:23:02<5:49:25, 46.49s/it]
                                                     
{'loss': 0.4802, 'grad_norm': 0.5774515212071393, 'learning_rate': 3.4218512898330806e-06, 'epoch': 0.97}

 33%|███▎      | 218/669 [1:23:02<5:49:25, 46.49s/it]
 33%|███▎      | 219/669 [1:23:46<5:42:55, 45.72s/it]
                                                     
{'loss': 0.483, 'grad_norm': 0.5159511449041201, 'learning_rate': 3.4142640364188168e-06, 'epoch': 0.98}

 33%|███▎      | 219/669 [1:23:46<5:42:55, 45.72s/it]
 33%|███▎      | 220/669 [1:24:30<5:38:14, 45.20s/it]
                                                     
{'loss': 0.4774, 'grad_norm': 0.509916149384416, 'learning_rate': 3.406676783004553e-06, 'epoch': 0.98}

 33%|███▎      | 220/669 [1:24:30<5:38:14, 45.20s/it]
 33%|███▎      | 221/669 [1:25:14<5:34:45, 44.83s/it]
                                                     
{'loss': 0.516, 'grad_norm': 0.5226659969898425, 'learning_rate': 3.3990895295902886e-06, 'epoch': 0.99}

 33%|███▎      | 221/669 [1:25:14<5:34:45, 44.83s/it]
 33%|███▎      | 222/669 [1:25:58<5:32:08, 44.58s/it]
                                                     
{'loss': 0.5346, 'grad_norm': 0.5632866199425641, 'learning_rate': 3.3915022761760247e-06, 'epoch': 0.99}

 33%|███▎      | 222/669 [1:25:58<5:32:08, 44.58s/it]
 33%|███▎      | 223/669 [1:26:42<5:30:07, 44.41s/it]
                                                     
{'loss': 0.5441, 'grad_norm': 0.5224220899066443, 'learning_rate': 3.383915022761761e-06, 'epoch': 1.0}

 33%|███▎      | 223/669 [1:26:42<5:30:07, 44.41s/it]
 33%|███▎      | 224/669 [1:27:26<5:28:22, 44.28s/it]
                                                     
{'loss': 0.512, 'grad_norm': 0.5965341004746808, 'learning_rate': 3.3763277693474965e-06, 'epoch': 1.0}

 33%|███▎      | 224/669 [1:27:26<5:28:22, 44.28s/it][2024-08-02 05:07:26,374] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:26,374] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:26,374] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:26,374] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:26,374] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:26,374] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:26,374] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:26,374] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:26,375] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:26,375] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:26,375] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:26,375] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:26,375] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:26,375] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:26,375] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:26,375] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:29,961] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:29,962] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:29,963] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:29,964] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


  0%|          | 0/8 [00:00<?, ?it/s][2024-08-02 05:07:33,246] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:33,247] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 25%|██▌       | 2/8 [00:03<00:09,  1.64s/it][2024-08-02 05:07:36,803] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:36,805] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 38%|███▊      | 3/8 [00:06<00:12,  2.44s/it][2024-08-02 05:07:40,354] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:40,355] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 50%|█████     | 4/8 [00:10<00:11,  2.85s/it][2024-08-02 05:07:44,100] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:44,101] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 62%|██████▎   | 5/8 [00:14<00:09,  3.16s/it][2024-08-02 05:07:47,645] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:47,646] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 75%|███████▌  | 6/8 [00:17<00:06,  3.29s/it][2024-08-02 05:07:51,186] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:51,187] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 88%|████████▊ | 7/8 [00:21<00:03,  3.37s/it][2024-08-02 05:07:54,741] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:54,742] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


100%|██████████| 8/8 [00:24<00:00,  3.43s/it][2024-08-02 05:07:58,390] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 05:07:58,391] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


9it [00:28,  3.50s/it]                       
                                                     

                      
{'eval_loss': 0.5086758732795715, 'eval_runtime': 32.322, 'eval_samples_per_second': 82.606, 'eval_steps_per_second': 5.167, 'epoch': 1.0}

 33%|███▎      | 224/669 [1:27:58<5:28:22, 44.28s/it]

9it [00:28,  3.50s/it]

                      /root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(

 34%|███▎      | 225/669 [1:29:16<7:53:36, 64.00s/it]
                                                     
{'loss': 0.5054, 'grad_norm': 0.5234315419190534, 'learning_rate': 3.3687405159332327e-06, 'epoch': 1.01}

 34%|███▎      | 225/669 [1:29:16<7:53:36, 64.00s/it]
 34%|███▍      | 226/669 [1:30:00<7:08:07, 57.99s/it]
                                                     
{'loss': 0.4878, 'grad_norm': 0.5575750854403085, 'learning_rate': 3.3611532625189684e-06, 'epoch': 1.01}

 34%|███▍      | 226/669 [1:30:00<7:08:07, 57.99s/it]
 34%|███▍      | 227/669 [1:30:44<6:36:06, 53.77s/it]
                                                     
{'loss': 0.5054, 'grad_norm': 0.5253448160628447, 'learning_rate': 3.3535660091047045e-06, 'epoch': 1.01}

 34%|███▍      | 227/669 [1:30:44<6:36:06, 53.77s/it]
 34%|███▍      | 228/669 [1:31:28<6:13:22, 50.80s/it]
                                                     
{'loss': 0.4818, 'grad_norm': 0.49020025660118194, 'learning_rate': 3.3459787556904406e-06, 'epoch': 1.02}

 34%|███▍      | 228/669 [1:31:28<6:13:22, 50.80s/it][2024-08-02 05:11:28,277] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 05:11:28,278] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 05:11:28,279] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 05:11:28,280] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 05:11:28,280] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 05:11:28,280] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 05:11:28,281] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 05:11:28,281] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 05:11:28,403] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 05:11:28,406] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 05:11:28,406] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 05:11:28,409] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 05:11:28,410] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 05:11:28,410] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 05:11:28,410] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 05:11:28,411] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488

 34%|███▍      | 229/669 [1:32:12<5:58:20, 48.87s/it]
                                                     
{'loss': 0.5008, 'grad_norm': 0.6277138162928673, 'learning_rate': 3.3383915022761763e-06, 'epoch': 1.0}

 34%|███▍      | 229/669 [1:32:12<5:58:20, 48.87s/it]
 34%|███▍      | 230/669 [1:32:56<5:46:53, 47.41s/it]
                                                     
{'loss': 0.4822, 'grad_norm': 0.5276575275655174, 'learning_rate': 3.3308042488619125e-06, 'epoch': 1.01}

 34%|███▍      | 230/669 [1:32:56<5:46:53, 47.41s/it]
 35%|███▍      | 231/669 [1:33:40<5:38:50, 46.42s/it]
                                                     
{'loss': 0.4783, 'grad_norm': 0.516729892988387, 'learning_rate': 3.3232169954476486e-06, 'epoch': 1.01}

 35%|███▍      | 231/669 [1:33:40<5:38:50, 46.42s/it]
 35%|███▍      | 232/669 [1:34:24<5:32:43, 45.68s/it]
                                                     
{'loss': 0.4927, 'grad_norm': 0.5648907712973277, 'learning_rate': 3.3156297420333843e-06, 'epoch': 1.02}

 35%|███▍      | 232/669 [1:34:24<5:32:43, 45.68s/it]
 35%|███▍      | 233/669 [1:35:08<5:28:19, 45.18s/it]
                                                     
{'loss': 0.4742, 'grad_norm': 0.5521221146190025, 'learning_rate': 3.3080424886191204e-06, 'epoch': 1.02}

 35%|███▍      | 233/669 [1:35:08<5:28:19, 45.18s/it]
 35%|███▍      | 234/669 [1:35:52<5:24:53, 44.81s/it]
                                                     
{'loss': 0.4863, 'grad_norm': 0.5288379239842262, 'learning_rate': 3.3004552352048565e-06, 'epoch': 1.03}

 35%|███▍      | 234/669 [1:35:52<5:24:53, 44.81s/it]
 35%|███▌      | 235/669 [1:36:36<5:22:17, 44.56s/it]
                                                     
{'loss': 0.4613, 'grad_norm': 0.4932863276651226, 'learning_rate': 3.2928679817905922e-06, 'epoch': 1.03}

 35%|███▌      | 235/669 [1:36:36<5:22:17, 44.56s/it]
 35%|███▌      | 236/669 [1:37:20<5:20:32, 44.42s/it]
                                                     
{'loss': 0.4969, 'grad_norm': 0.5069055178884726, 'learning_rate': 3.2852807283763284e-06, 'epoch': 1.04}

 35%|███▌      | 236/669 [1:37:20<5:20:32, 44.42s/it]
 35%|███▌      | 237/669 [1:39:19<8:01:02, 66.81s/it]
                                                     
{'loss': 0.4759, 'grad_norm': 0.49009866664123986, 'learning_rate': 3.277693474962064e-06, 'epoch': 1.04}

 35%|███▌      | 237/669 [1:39:19<8:01:02, 66.81s/it]
 36%|███▌      | 238/669 [1:40:03<7:10:15, 59.90s/it]
                                                     
{'loss': 0.4654, 'grad_norm': 0.5220649844244807, 'learning_rate': 3.2701062215478e-06, 'epoch': 1.04}

 36%|███▌      | 238/669 [1:40:03<7:10:15, 59.90s/it]
 36%|███▌      | 239/669 [1:40:47<6:35:00, 55.12s/it]
                                                     
{'loss': 0.5009, 'grad_norm': 0.5354627141925222, 'learning_rate': 3.2625189681335363e-06, 'epoch': 1.05}

 36%|███▌      | 239/669 [1:40:47<6:35:00, 55.12s/it]
 36%|███▌      | 240/669 [1:41:31<6:10:03, 51.76s/it]
                                                     
{'loss': 0.4564, 'grad_norm': 0.5561818122893151, 'learning_rate': 3.254931714719272e-06, 'epoch': 1.05}

 36%|███▌      | 240/669 [1:41:31<6:10:03, 51.76s/it]
 36%|███▌      | 241/669 [1:42:15<5:52:28, 49.41s/it]
                                                     
{'loss': 0.4924, 'grad_norm': 0.48719455414113416, 'learning_rate': 3.2473444613050077e-06, 'epoch': 1.06}

 36%|███▌      | 241/669 [1:42:15<5:52:28, 49.41s/it]
 36%|███▌      | 242/669 [1:42:59<5:40:05, 47.79s/it]
                                                     
{'loss': 0.4856, 'grad_norm': 0.5527905454518657, 'learning_rate': 3.2397572078907434e-06, 'epoch': 1.06}

 36%|███▌      | 242/669 [1:42:59<5:40:05, 47.79s/it]
 36%|███▋      | 243/669 [1:43:43<5:31:00, 46.62s/it]
                                                     
{'loss': 0.4935, 'grad_norm': 0.5381863443762583, 'learning_rate': 3.2321699544764795e-06, 'epoch': 1.07}

 36%|███▋      | 243/669 [1:43:43<5:31:00, 46.62s/it]
 36%|███▋      | 244/669 [1:44:27<5:24:33, 45.82s/it]
                                                     
{'loss': 0.4858, 'grad_norm': 0.5355360844432822, 'learning_rate': 3.2245827010622157e-06, 'epoch': 1.07}

 36%|███▋      | 244/669 [1:44:27<5:24:33, 45.82s/it]
 37%|███▋      | 245/669 [1:45:11<5:19:56, 45.28s/it]
                                                     
{'loss': 0.4819, 'grad_norm': 0.5191892602172652, 'learning_rate': 3.2169954476479514e-06, 'epoch': 1.08}

 37%|███▋      | 245/669 [1:45:11<5:19:56, 45.28s/it]
 37%|███▋      | 246/669 [1:45:55<5:16:23, 44.88s/it]
                                                     
{'loss': 0.5007, 'grad_norm': 0.4952672717234948, 'learning_rate': 3.2094081942336875e-06, 'epoch': 1.08}

 37%|███▋      | 246/669 [1:45:55<5:16:23, 44.88s/it]
 37%|███▋      | 247/669 [1:46:39<5:13:37, 44.59s/it]
                                                     
{'loss': 0.4874, 'grad_norm': 0.5331734326280092, 'learning_rate': 3.201820940819423e-06, 'epoch': 1.08}

 37%|███▋      | 247/669 [1:46:39<5:13:37, 44.59s/it]
 37%|███▋      | 248/669 [1:47:23<5:11:46, 44.43s/it]
                                                     
{'loss': 0.5045, 'grad_norm': 0.49333273079420603, 'learning_rate': 3.1942336874051593e-06, 'epoch': 1.09}

 37%|███▋      | 248/669 [1:47:23<5:11:46, 44.43s/it]
 37%|███▋      | 249/669 [1:48:07<5:10:02, 44.29s/it]
                                                     
{'loss': 0.5047, 'grad_norm': 0.5143326582021585, 'learning_rate': 3.1866464339908955e-06, 'epoch': 1.09}

 37%|███▋      | 249/669 [1:48:07<5:10:02, 44.29s/it]
 37%|███▋      | 250/669 [1:48:51<5:08:38, 44.20s/it]
                                                     
{'loss': 0.4803, 'grad_norm': 0.5027227799706213, 'learning_rate': 3.179059180576631e-06, 'epoch': 1.1}

 37%|███▋      | 250/669 [1:48:51<5:08:38, 44.20s/it]
 38%|███▊      | 251/669 [1:49:35<5:07:28, 44.13s/it]
                                                     
{'loss': 0.4654, 'grad_norm': 0.5257359502707071, 'learning_rate': 3.1714719271623673e-06, 'epoch': 1.1}

 38%|███▊      | 251/669 [1:49:35<5:07:28, 44.13s/it]
 38%|███▊      | 252/669 [1:50:19<5:06:30, 44.10s/it]
                                                     
{'loss': 0.5329, 'grad_norm': 0.5244897759431714, 'learning_rate': 3.1638846737481034e-06, 'epoch': 1.11}

 38%|███▊      | 252/669 [1:50:19<5:06:30, 44.10s/it]
 38%|███▊      | 253/669 [1:51:03<5:05:36, 44.08s/it]
                                                     
{'loss': 0.4721, 'grad_norm': 0.5028531015372357, 'learning_rate': 3.156297420333839e-06, 'epoch': 1.11}

 38%|███▊      | 253/669 [1:51:03<5:05:36, 44.08s/it]
 38%|███▊      | 254/669 [1:51:47<5:04:37, 44.04s/it]
                                                     
{'loss': 0.4962, 'grad_norm': 0.5342121642252061, 'learning_rate': 3.1487101669195752e-06, 'epoch': 1.12}

 38%|███▊      | 254/669 [1:51:47<5:04:37, 44.04s/it]
 38%|███▊      | 255/669 [1:52:31<5:03:39, 44.01s/it]
                                                     
{'loss': 0.4884, 'grad_norm': 0.49550125400926287, 'learning_rate': 3.1411229135053114e-06, 'epoch': 1.12}

 38%|███▊      | 255/669 [1:52:31<5:03:39, 44.01s/it]
 38%|███▊      | 256/669 [1:53:14<5:02:42, 43.98s/it]
                                                     
{'loss': 0.4676, 'grad_norm': 0.5724921757672813, 'learning_rate': 3.133535660091047e-06, 'epoch': 1.13}

 38%|███▊      | 256/669 [1:53:14<5:02:42, 43.98s/it]
 38%|███▊      | 257/669 [1:53:58<5:01:55, 43.97s/it]
                                                     
{'loss': 0.48, 'grad_norm': 0.5012184887171401, 'learning_rate': 3.125948406676783e-06, 'epoch': 1.13}

 38%|███▊      | 257/669 [1:53:58<5:01:55, 43.97s/it]
 39%|███▊      | 258/669 [1:54:42<5:01:05, 43.96s/it]
                                                     
{'loss': 0.494, 'grad_norm': 0.5049054849116781, 'learning_rate': 3.118361153262519e-06, 'epoch': 1.13}

 39%|███▊      | 258/669 [1:54:42<5:01:05, 43.96s/it]
 39%|███▊      | 259/669 [1:55:26<5:00:38, 44.00s/it]
                                                     
{'loss': 0.4787, 'grad_norm': 0.5291980859522503, 'learning_rate': 3.110773899848255e-06, 'epoch': 1.14}

 39%|███▊      | 259/669 [1:55:26<5:00:38, 44.00s/it]
 39%|███▉      | 260/669 [1:56:10<4:59:46, 43.98s/it]
                                                     
{'loss': 0.4745, 'grad_norm': 0.5235318263865767, 'learning_rate': 3.103186646433991e-06, 'epoch': 1.14}

 39%|███▉      | 260/669 [1:56:10<4:59:46, 43.98s/it]
 39%|███▉      | 261/669 [1:56:54<4:58:58, 43.97s/it]
                                                     
{'loss': 0.5863, 'grad_norm': 0.5067236081675356, 'learning_rate': 3.095599393019727e-06, 'epoch': 1.15}

 39%|███▉      | 261/669 [1:56:54<4:58:58, 43.97s/it]
 39%|███▉      | 262/669 [1:58:52<7:29:01, 66.20s/it]
                                                     
{'loss': 0.5264, 'grad_norm': 0.48763909341918976, 'learning_rate': 3.088012139605463e-06, 'epoch': 1.15}

 39%|███▉      | 262/669 [1:58:52<7:29:01, 66.20s/it]
 39%|███▉      | 263/669 [1:59:36<6:42:27, 59.48s/it]
                                                     
{'loss': 0.4716, 'grad_norm': 0.5198110315693464, 'learning_rate': 3.080424886191199e-06, 'epoch': 1.16}

 39%|███▉      | 263/669 [1:59:36<6:42:27, 59.48s/it]
 39%|███▉      | 264/669 [2:00:20<6:09:55, 54.80s/it]
                                                     
{'loss': 0.4768, 'grad_norm': 0.5093655208180958, 'learning_rate': 3.072837632776935e-06, 'epoch': 1.16}

 39%|███▉      | 264/669 [2:00:20<6:09:55, 54.80s/it]
 40%|███▉      | 265/669 [2:01:04<5:47:17, 51.58s/it]
                                                     
{'loss': 0.4765, 'grad_norm': 0.49489186143964614, 'learning_rate': 3.065250379362671e-06, 'epoch': 1.17}

 40%|███▉      | 265/669 [2:01:04<5:47:17, 51.58s/it]
 40%|███▉      | 266/669 [2:01:48<5:31:03, 49.29s/it]
                                                     
{'loss': 0.4869, 'grad_norm': 0.5209973994944476, 'learning_rate': 3.057663125948407e-06, 'epoch': 1.17}

 40%|███▉      | 266/669 [2:01:48<5:31:03, 49.29s/it]
 40%|███▉      | 267/669 [2:02:32<5:19:29, 47.69s/it]
                                                     
{'loss': 0.4623, 'grad_norm': 0.4873878645874658, 'learning_rate': 3.0500758725341427e-06, 'epoch': 1.17}

 40%|███▉      | 267/669 [2:02:32<5:19:29, 47.69s/it]
 40%|████      | 268/669 [2:03:16<5:11:14, 46.57s/it]
                                                     
{'loss': 0.4765, 'grad_norm': 0.5074910158328607, 'learning_rate': 3.042488619119879e-06, 'epoch': 1.18}

 40%|████      | 268/669 [2:03:16<5:11:14, 46.57s/it]
 40%|████      | 269/669 [2:04:00<5:05:12, 45.78s/it]
                                                     
{'loss': 0.4838, 'grad_norm': 0.5195087180758534, 'learning_rate': 3.0349013657056146e-06, 'epoch': 1.18}

 40%|████      | 269/669 [2:04:00<5:05:12, 45.78s/it]
 40%|████      | 270/669 [2:04:44<5:01:07, 45.28s/it]
                                                     
{'loss': 0.4638, 'grad_norm': 0.5264228931092361, 'learning_rate': 3.0273141122913507e-06, 'epoch': 1.19}

 40%|████      | 270/669 [2:04:44<5:01:07, 45.28s/it]
 41%|████      | 271/669 [2:05:28<4:57:45, 44.89s/it]
                                                     
{'loss': 0.4981, 'grad_norm': 0.4919380409463276, 'learning_rate': 3.019726858877087e-06, 'epoch': 1.19}

 41%|████      | 271/669 [2:05:28<4:57:45, 44.89s/it]
 41%|████      | 272/669 [2:06:12<4:55:08, 44.61s/it]
                                                     
{'loss': 0.477, 'grad_norm': 0.5060903656555795, 'learning_rate': 3.0121396054628225e-06, 'epoch': 1.2}

 41%|████      | 272/669 [2:06:12<4:55:08, 44.61s/it]
 41%|████      | 273/669 [2:06:56<4:53:08, 44.42s/it]
                                                     
{'loss': 0.5003, 'grad_norm': 0.4938697284936987, 'learning_rate': 3.0045523520485587e-06, 'epoch': 1.2}

 41%|████      | 273/669 [2:06:56<4:53:08, 44.42s/it]
 41%|████      | 274/669 [2:07:40<4:51:31, 44.28s/it]
                                                     
{'loss': 0.4673, 'grad_norm': 0.5599516302969035, 'learning_rate': 2.9969650986342948e-06, 'epoch': 1.21}

 41%|████      | 274/669 [2:07:40<4:51:31, 44.28s/it]
 41%|████      | 275/669 [2:08:24<4:50:14, 44.20s/it]
                                                     
{'loss': 0.4971, 'grad_norm': 0.5126307823030745, 'learning_rate': 2.9893778452200305e-06, 'epoch': 1.21}

 41%|████      | 275/669 [2:08:24<4:50:14, 44.20s/it]
 41%|████▏     | 276/669 [2:09:08<4:49:01, 44.13s/it]
                                                     
{'loss': 0.5006, 'grad_norm': 0.5927333276589611, 'learning_rate': 2.9817905918057666e-06, 'epoch': 1.21}

 41%|████▏     | 276/669 [2:09:08<4:49:01, 44.13s/it]
 41%|████▏     | 277/669 [2:09:52<4:47:52, 44.06s/it]
                                                     
{'loss': 0.5143, 'grad_norm': 0.5367579038483986, 'learning_rate': 2.9742033383915027e-06, 'epoch': 1.22}

 41%|████▏     | 277/669 [2:09:52<4:47:52, 44.06s/it]
 42%|████▏     | 278/669 [2:10:36<4:46:59, 44.04s/it]
                                                     
{'loss': 0.4733, 'grad_norm': 0.5179699664039845, 'learning_rate': 2.9666160849772384e-06, 'epoch': 1.22}

 42%|████▏     | 278/669 [2:10:36<4:46:59, 44.04s/it]
 42%|████▏     | 279/669 [2:11:20<4:46:03, 44.01s/it]
                                                     
{'loss': 0.4786, 'grad_norm': 0.5015385250607317, 'learning_rate': 2.9590288315629746e-06, 'epoch': 1.23}

 42%|████▏     | 279/669 [2:11:20<4:46:03, 44.01s/it]
 42%|████▏     | 280/669 [2:12:04<4:45:14, 44.00s/it]
                                                     
{'loss': 0.472, 'grad_norm': 0.4925764923779409, 'learning_rate': 2.9514415781487103e-06, 'epoch': 1.23}

 42%|████▏     | 280/669 [2:12:04<4:45:14, 44.00s/it]/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(

 42%|████▏     | 281/669 [2:13:21<5:49:20, 54.02s/it]
                                                     
{'loss': 0.4675, 'grad_norm': 0.5522056674324646, 'learning_rate': 2.9438543247344464e-06, 'epoch': 1.24}

 42%|████▏     | 281/669 [2:13:21<5:49:20, 54.02s/it]
 42%|████▏     | 282/669 [2:14:05<5:28:53, 50.99s/it]
                                                     
{'loss': 0.4778, 'grad_norm': 0.5128909667557014, 'learning_rate': 2.9362670713201825e-06, 'epoch': 1.24}

 42%|████▏     | 282/669 [2:14:05<5:28:53, 50.99s/it]
 42%|████▏     | 283/669 [2:14:49<5:14:24, 48.87s/it]
                                                     
{'loss': 0.4744, 'grad_norm': 0.49958051066954534, 'learning_rate': 2.9286798179059182e-06, 'epoch': 1.25}

 42%|████▏     | 283/669 [2:14:49<5:14:24, 48.87s/it]
 42%|████▏     | 284/669 [2:15:33<5:04:05, 47.39s/it]
                                                     
{'loss': 0.47, 'grad_norm': 0.5186801983796817, 'learning_rate': 2.9210925644916543e-06, 'epoch': 1.25}

 42%|████▏     | 284/669 [2:15:33<5:04:05, 47.39s/it]
 43%|████▎     | 285/669 [2:16:17<4:56:46, 46.37s/it]
                                                     
{'loss': 0.4693, 'grad_norm': 0.49479081933964797, 'learning_rate': 2.9135053110773905e-06, 'epoch': 1.25}

 43%|████▎     | 285/669 [2:16:17<4:56:46, 46.37s/it]
 43%|████▎     | 286/669 [2:17:01<4:51:31, 45.67s/it]
                                                     
{'loss': 0.5034, 'grad_norm': 0.5172924586559385, 'learning_rate': 2.905918057663126e-06, 'epoch': 1.26}

 43%|████▎     | 286/669 [2:17:01<4:51:31, 45.67s/it]
 43%|████▎     | 287/669 [2:19:00<7:10:45, 67.66s/it]
                                                     
{'loss': 0.4857, 'grad_norm': 0.49588955761543196, 'learning_rate': 2.8983308042488623e-06, 'epoch': 1.26}

 43%|████▎     | 287/669 [2:19:00<7:10:45, 67.66s/it]
 43%|████▎     | 288/669 [2:19:44<6:24:14, 60.51s/it]
                                                     
{'loss': 0.4845, 'grad_norm': 0.5651670438316321, 'learning_rate': 2.8907435508345984e-06, 'epoch': 1.27}

 43%|████▎     | 288/669 [2:19:44<6:24:14, 60.51s/it]
 43%|████▎     | 289/669 [2:20:28<5:51:43, 55.53s/it]
                                                     
{'loss': 0.4482, 'grad_norm': 0.47720259500943096, 'learning_rate': 2.883156297420334e-06, 'epoch': 1.27}

 43%|████▎     | 289/669 [2:20:28<5:51:43, 55.53s/it]
 43%|████▎     | 290/669 [2:21:12<5:28:49, 52.06s/it]
                                                     
{'loss': 0.4826, 'grad_norm': 0.5080092332022862, 'learning_rate': 2.8755690440060702e-06, 'epoch': 1.28}

 43%|████▎     | 290/669 [2:21:12<5:28:49, 52.06s/it]
 43%|████▎     | 291/669 [2:21:55<5:12:37, 49.62s/it]
                                                     
{'loss': 0.5592, 'grad_norm': 0.6016729996144168, 'learning_rate': 2.867981790591806e-06, 'epoch': 1.28}

 43%|████▎     | 291/669 [2:21:56<5:12:37, 49.62s/it]
 44%|████▎     | 292/669 [2:22:40<5:01:19, 47.96s/it]
                                                     
{'loss': 0.4821, 'grad_norm': 0.5456040494522852, 'learning_rate': 2.860394537177542e-06, 'epoch': 1.29}

 44%|████▎     | 292/669 [2:22:40<5:01:19, 47.96s/it]
 44%|████▍     | 293/669 [2:23:23<4:52:57, 46.75s/it]
                                                     
{'loss': 0.5089, 'grad_norm': 0.5858331665900632, 'learning_rate': 2.852807283763278e-06, 'epoch': 1.29}

 44%|████▍     | 293/669 [2:23:23<4:52:57, 46.75s/it]
 44%|████▍     | 294/669 [2:24:08<4:47:23, 45.98s/it]
                                                     
{'loss': 0.5101, 'grad_norm': 0.5895630636847224, 'learning_rate': 2.845220030349014e-06, 'epoch': 1.29}

 44%|████▍     | 294/669 [2:24:08<4:47:23, 45.98s/it]
 44%|████▍     | 295/669 [2:24:52<4:42:46, 45.37s/it]
                                                     
{'loss': 0.5008, 'grad_norm': 0.505172328049275, 'learning_rate': 2.83763277693475e-06, 'epoch': 1.3}

 44%|████▍     | 295/669 [2:24:52<4:42:46, 45.37s/it]
 44%|████▍     | 296/669 [2:25:36<4:39:27, 44.95s/it]
                                                     
{'loss': 0.4812, 'grad_norm': 0.5035894899292893, 'learning_rate': 2.830045523520486e-06, 'epoch': 1.3}

 44%|████▍     | 296/669 [2:25:36<4:39:27, 44.95s/it]
 44%|████▍     | 297/669 [2:26:20<4:36:52, 44.66s/it]
                                                     
{'loss': 0.4579, 'grad_norm': 0.5304264887862729, 'learning_rate': 2.822458270106222e-06, 'epoch': 1.31}

 44%|████▍     | 297/669 [2:26:20<4:36:52, 44.66s/it]
 45%|████▍     | 298/669 [2:27:04<4:34:50, 44.45s/it]
                                                     
{'loss': 0.4801, 'grad_norm': 0.4742288658363735, 'learning_rate': 2.814871016691958e-06, 'epoch': 1.31}

 45%|████▍     | 298/669 [2:27:04<4:34:50, 44.45s/it]
 45%|████▍     | 299/669 [2:27:47<4:33:08, 44.29s/it]
                                                     
{'loss': 0.4602, 'grad_norm': 0.5293412371049411, 'learning_rate': 2.807283763277694e-06, 'epoch': 1.32}

 45%|████▍     | 299/669 [2:27:47<4:33:08, 44.29s/it]
 45%|████▍     | 300/669 [2:28:31<4:31:44, 44.18s/it]
                                                     
{'loss': 0.4963, 'grad_norm': 0.5333831439310788, 'learning_rate': 2.79969650986343e-06, 'epoch': 1.32}

 45%|████▍     | 300/669 [2:28:31<4:31:44, 44.18s/it]
 45%|████▍     | 301/669 [2:29:15<4:30:28, 44.10s/it]
                                                     
{'loss': 0.4781, 'grad_norm': 0.4947844594727205, 'learning_rate': 2.792109256449166e-06, 'epoch': 1.33}

 45%|████▍     | 301/669 [2:29:15<4:30:28, 44.10s/it]
 45%|████▌     | 302/669 [2:29:59<4:29:37, 44.08s/it]
                                                     
{'loss': 0.4998, 'grad_norm': 0.5323122787674339, 'learning_rate': 2.7845220030349016e-06, 'epoch': 1.33}

 45%|████▌     | 302/669 [2:29:59<4:29:37, 44.08s/it]
 45%|████▌     | 303/669 [2:30:43<4:28:34, 44.03s/it]
                                                     
{'loss': 0.466, 'grad_norm': 0.48615316005162806, 'learning_rate': 2.7769347496206378e-06, 'epoch': 1.34}

 45%|████▌     | 303/669 [2:30:43<4:28:34, 44.03s/it]
 45%|████▌     | 304/669 [2:31:27<4:27:40, 44.00s/it]
                                                     
{'loss': 0.531, 'grad_norm': 0.5038897355673981, 'learning_rate': 2.769347496206374e-06, 'epoch': 1.34}

 45%|████▌     | 304/669 [2:31:27<4:27:40, 44.00s/it]
 46%|████▌     | 305/669 [2:32:11<4:26:48, 43.98s/it]
                                                     
{'loss': 0.4917, 'grad_norm': 0.47725378544672703, 'learning_rate': 2.7617602427921096e-06, 'epoch': 1.34}

 46%|████▌     | 305/669 [2:32:11<4:26:48, 43.98s/it]
 46%|████▌     | 306/669 [2:32:55<4:25:54, 43.95s/it]
                                                     
{'loss': 0.4673, 'grad_norm': 0.5241579403675019, 'learning_rate': 2.7541729893778457e-06, 'epoch': 1.35}

 46%|████▌     | 306/669 [2:32:55<4:25:54, 43.95s/it]
 46%|████▌     | 307/669 [2:33:39<4:25:05, 43.94s/it]
                                                     
{'loss': 0.467, 'grad_norm': 0.5252091486066182, 'learning_rate': 2.746585735963581e-06, 'epoch': 1.35}

 46%|████▌     | 307/669 [2:33:39<4:25:05, 43.94s/it]
 46%|████▌     | 308/669 [2:34:23<4:24:25, 43.95s/it]
                                                     
{'loss': 0.5257, 'grad_norm': 0.4803361684555166, 'learning_rate': 2.738998482549317e-06, 'epoch': 1.36}

 46%|████▌     | 308/669 [2:34:23<4:24:25, 43.95s/it]
 46%|████▌     | 309/669 [2:35:07<4:23:42, 43.95s/it]
                                                     
{'loss': 0.4889, 'grad_norm': 0.5300934772683465, 'learning_rate': 2.7314112291350532e-06, 'epoch': 1.36}

 46%|████▌     | 309/669 [2:35:07<4:23:42, 43.95s/it]
 46%|████▋     | 310/669 [2:35:51<4:22:53, 43.94s/it]
                                                     
{'loss': 0.4847, 'grad_norm': 0.5001374114396476, 'learning_rate': 2.723823975720789e-06, 'epoch': 1.37}

 46%|████▋     | 310/669 [2:35:51<4:22:53, 43.94s/it]
 46%|████▋     | 311/669 [2:36:35<4:22:14, 43.95s/it]
                                                     
{'loss': 0.4578, 'grad_norm': 0.5240875065710657, 'learning_rate': 2.716236722306525e-06, 'epoch': 1.37}

 46%|████▋     | 311/669 [2:36:35<4:22:14, 43.95s/it]
 47%|████▋     | 312/669 [2:38:34<6:36:26, 66.63s/it]
                                                     
{'loss': 0.4947, 'grad_norm': 0.4714761354385937, 'learning_rate': 2.708649468892261e-06, 'epoch': 1.38}

 47%|████▋     | 312/669 [2:38:34<6:36:26, 66.63s/it]
 47%|████▋     | 313/669 [2:39:18<5:54:40, 59.78s/it]
                                                     
{'loss': 0.5049, 'grad_norm': 0.5205222669608968, 'learning_rate': 2.701062215477997e-06, 'epoch': 1.38}

 47%|████▋     | 313/669 [2:39:18<5:54:40, 59.78s/it]
 47%|████▋     | 314/669 [2:40:02<5:25:36, 55.03s/it]
                                                     
{'loss': 0.4946, 'grad_norm': 0.5409150510364881, 'learning_rate': 2.693474962063733e-06, 'epoch': 1.38}

 47%|████▋     | 314/669 [2:40:02<5:25:36, 55.03s/it]
 47%|████▋     | 315/669 [2:40:46<5:05:16, 51.74s/it]
                                                     
{'loss': 0.5024, 'grad_norm': 0.49473773833387136, 'learning_rate': 2.6858877086494687e-06, 'epoch': 1.39}

 47%|████▋     | 315/669 [2:40:46<5:05:16, 51.74s/it]
 47%|████▋     | 316/669 [2:41:30<4:51:02, 49.47s/it]
                                                     
{'loss': 0.5129, 'grad_norm': 0.5322303350412809, 'learning_rate': 2.678300455235205e-06, 'epoch': 1.39}

 47%|████▋     | 316/669 [2:41:30<4:51:02, 49.47s/it]
 47%|████▋     | 317/669 [2:42:14<4:40:26, 47.80s/it]
                                                     
{'loss': 0.4852, 'grad_norm': 0.5249345159944246, 'learning_rate': 2.670713201820941e-06, 'epoch': 1.4}

 47%|████▋     | 317/669 [2:42:14<4:40:26, 47.80s/it]
 48%|████▊     | 318/669 [2:42:58<4:32:52, 46.65s/it]
                                                     
{'loss': 0.4755, 'grad_norm': 0.526125738227021, 'learning_rate': 2.6631259484066767e-06, 'epoch': 1.4}

 48%|████▊     | 318/669 [2:42:58<4:32:52, 46.65s/it]
 48%|████▊     | 319/669 [2:43:42<4:27:28, 45.85s/it]
                                                     
{'loss': 0.4708, 'grad_norm': 0.5077327993266544, 'learning_rate': 2.655538694992413e-06, 'epoch': 1.41}

 48%|████▊     | 319/669 [2:43:42<4:27:28, 45.85s/it]
 48%|████▊     | 320/669 [2:44:26<4:23:25, 45.29s/it]
                                                     
{'loss': 0.4574, 'grad_norm': 0.509006777595202, 'learning_rate': 2.647951441578149e-06, 'epoch': 1.41}

 48%|████▊     | 320/669 [2:44:26<4:23:25, 45.29s/it]
 48%|████▊     | 321/669 [2:45:10<4:20:28, 44.91s/it]
                                                     
{'loss': 0.4983, 'grad_norm': 0.5397063984545032, 'learning_rate': 2.6403641881638846e-06, 'epoch': 1.42}

 48%|████▊     | 321/669 [2:45:10<4:20:28, 44.91s/it]
 48%|████▊     | 322/669 [2:45:54<4:18:08, 44.64s/it]
                                                     
{'loss': 0.5063, 'grad_norm': 0.588039527577832, 'learning_rate': 2.6327769347496208e-06, 'epoch': 1.42}

 48%|████▊     | 322/669 [2:45:54<4:18:08, 44.64s/it]
 48%|████▊     | 323/669 [2:46:38<4:16:13, 44.43s/it]
                                                     
{'loss': 0.5439, 'grad_norm': 0.5086942941008908, 'learning_rate': 2.625189681335357e-06, 'epoch': 1.42}

 48%|████▊     | 323/669 [2:46:38<4:16:13, 44.43s/it]
 48%|████▊     | 324/669 [2:47:22<4:14:38, 44.29s/it]
                                                     
{'loss': 0.4789, 'grad_norm': 0.5593378373560197, 'learning_rate': 2.6176024279210926e-06, 'epoch': 1.43}

 48%|████▊     | 324/669 [2:47:22<4:14:38, 44.29s/it]
 49%|████▊     | 325/669 [2:48:06<4:13:21, 44.19s/it]
                                                     
{'loss': 0.4861, 'grad_norm': 0.5473461722884195, 'learning_rate': 2.6100151745068287e-06, 'epoch': 1.43}

 49%|████▊     | 325/669 [2:48:06<4:13:21, 44.19s/it]
 49%|████▊     | 326/669 [2:48:50<4:12:10, 44.11s/it]
                                                     
{'loss': 0.4818, 'grad_norm': 0.5744717523131797, 'learning_rate': 2.6024279210925644e-06, 'epoch': 1.44}

 49%|████▊     | 326/669 [2:48:50<4:12:10, 44.11s/it]
 49%|████▉     | 327/669 [2:49:34<4:11:06, 44.05s/it]
                                                     
{'loss': 0.4785, 'grad_norm': 0.5240335872273282, 'learning_rate': 2.5948406676783005e-06, 'epoch': 1.44}

 49%|████▉     | 327/669 [2:49:34<4:11:06, 44.05s/it]
 49%|████▉     | 328/669 [2:50:18<4:10:07, 44.01s/it]
                                                     
{'loss': 0.5042, 'grad_norm': 0.5608475565499077, 'learning_rate': 2.5872534142640367e-06, 'epoch': 1.45}

 49%|████▉     | 328/669 [2:50:18<4:10:07, 44.01s/it]
 49%|████▉     | 329/669 [2:51:02<4:09:18, 43.99s/it]
                                                     
{'loss': 0.4995, 'grad_norm': 0.5363050759442354, 'learning_rate': 2.5796661608497724e-06, 'epoch': 1.45}

 49%|████▉     | 329/669 [2:51:02<4:09:18, 43.99s/it]
 49%|████▉     | 330/669 [2:51:46<4:08:32, 43.99s/it]
                                                     
{'loss': 0.4954, 'grad_norm': 0.5027607995293213, 'learning_rate': 2.5720789074355085e-06, 'epoch': 1.46}

 49%|████▉     | 330/669 [2:51:46<4:08:32, 43.99s/it]
 49%|████▉     | 331/669 [2:52:30<4:07:39, 43.96s/it]
                                                     
{'loss': 0.4882, 'grad_norm': 0.4969954043343912, 'learning_rate': 2.5644916540212446e-06, 'epoch': 1.46}

 49%|████▉     | 331/669 [2:52:30<4:07:39, 43.96s/it]
 50%|████▉     | 332/669 [2:53:13<4:06:51, 43.95s/it]
                                                     
{'loss': 0.4726, 'grad_norm': 0.5718337637259987, 'learning_rate': 2.5569044006069803e-06, 'epoch': 1.46}

 50%|████▉     | 332/669 [2:53:13<4:06:51, 43.95s/it]
 50%|████▉     | 333/669 [2:53:57<4:06:07, 43.95s/it]
                                                     
{'loss': 0.4697, 'grad_norm': 0.5234285673769393, 'learning_rate': 2.5493171471927164e-06, 'epoch': 1.47}

 50%|████▉     | 333/669 [2:53:57<4:06:07, 43.95s/it]
 50%|████▉     | 334/669 [2:54:41<4:05:28, 43.97s/it]
                                                     
{'loss': 0.4776, 'grad_norm': 0.5219608223935474, 'learning_rate': 2.5417298937784526e-06, 'epoch': 1.47}

 50%|████▉     | 334/669 [2:54:41<4:05:28, 43.97s/it]
 50%|█████     | 335/669 [2:55:25<4:04:44, 43.96s/it]
                                                     
{'loss': 0.4756, 'grad_norm': 0.606768683027719, 'learning_rate': 2.5341426403641883e-06, 'epoch': 1.48}

 50%|█████     | 335/669 [2:55:25<4:04:44, 43.96s/it]
 50%|█████     | 336/669 [2:56:09<4:03:54, 43.95s/it]
                                                     
{'loss': 0.4619, 'grad_norm': 0.5432176614167162, 'learning_rate': 2.5265553869499244e-06, 'epoch': 1.48}

 50%|█████     | 336/669 [2:56:09<4:03:54, 43.95s/it][2024-08-02 06:36:09,661] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:09,661] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:09,661] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:09,661] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:09,661] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:09,661] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:09,661] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:09,661] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:09,662] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:09,662] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:09,662] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:09,662] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:09,662] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:09,662] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:09,662] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:09,662] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:12,926] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:12,927] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:12,928] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:12,929] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


  0%|          | 0/8 [00:00<?, ?it/s][2024-08-02 06:36:16,474] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:16,476] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 25%|██▌       | 2/8 [00:03<00:10,  1.77s/it][2024-08-02 06:36:20,016] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:20,017] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 38%|███▊      | 3/8 [00:07<00:12,  2.51s/it][2024-08-02 06:36:23,561] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:23,562] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 50%|█████     | 4/8 [00:10<00:11,  2.90s/it][2024-08-02 06:36:27,305] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:27,306] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 62%|██████▎   | 5/8 [00:14<00:09,  3.19s/it][2024-08-02 06:36:30,865] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:30,866] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 75%|███████▌  | 6/8 [00:17<00:06,  3.31s/it][2024-08-02 06:36:34,413] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:34,414] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 88%|████████▊ | 7/8 [00:21<00:03,  3.39s/it][2024-08-02 06:36:37,965] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:37,966] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


100%|██████████| 8/8 [00:25<00:00,  3.44s/it][2024-08-02 06:36:41,620] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 06:36:41,621] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


9it [00:28,  3.51s/it]                       
                                                     

                      
{'eval_loss': 0.5032872557640076, 'eval_runtime': 32.2653, 'eval_samples_per_second': 82.751, 'eval_steps_per_second': 5.176, 'epoch': 1.48}

 50%|█████     | 336/669 [2:56:42<4:03:54, 43.95s/it]

9it [00:28,  3.51s/it]

                      /root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(

 50%|█████     | 337/669 [2:59:12<7:52:50, 85.45s/it]
                                                     
{'loss': 0.4667, 'grad_norm': 0.5377224565720307, 'learning_rate': 2.51896813353566e-06, 'epoch': 1.49}

 50%|█████     | 337/669 [2:59:12<7:52:50, 85.45s/it]
 51%|█████     | 338/669 [2:59:56<6:42:42, 73.00s/it]
                                                     
{'loss': 0.4873, 'grad_norm': 0.5875842821019925, 'learning_rate': 2.5113808801213962e-06, 'epoch': 1.49}

 51%|█████     | 338/669 [2:59:56<6:42:42, 73.00s/it]
 51%|█████     | 339/669 [3:00:40<5:54:10, 64.40s/it]
                                                     
{'loss': 0.4626, 'grad_norm': 0.5624618850955563, 'learning_rate': 2.5037936267071324e-06, 'epoch': 1.5}

 51%|█████     | 339/669 [3:00:40<5:54:10, 64.40s/it]
 51%|█████     | 340/669 [3:01:24<5:19:50, 58.33s/it]
                                                     
{'loss': 0.4749, 'grad_norm': 0.5380665704623208, 'learning_rate': 2.496206373292868e-06, 'epoch': 1.5}

 51%|█████     | 340/669 [3:01:24<5:19:50, 58.33s/it]
 51%|█████     | 341/669 [3:02:08<4:55:40, 54.09s/it]
                                                     
{'loss': 0.4862, 'grad_norm': 0.5515048280645092, 'learning_rate': 2.488619119878604e-06, 'epoch': 1.51}

 51%|█████     | 341/669 [3:02:08<4:55:40, 54.09s/it]
 51%|█████     | 342/669 [3:02:53<4:38:46, 51.15s/it]
                                                     
{'loss': 0.4533, 'grad_norm': 0.5541495309875792, 'learning_rate': 2.4810318664643403e-06, 'epoch': 1.51}

 51%|█████     | 342/669 [3:02:53<4:38:46, 51.15s/it]
 51%|█████▏    | 343/669 [3:03:36<4:26:11, 48.99s/it]
                                                     
{'loss': 0.4585, 'grad_norm': 0.514569610923219, 'learning_rate': 2.473444613050076e-06, 'epoch': 1.51}

 51%|█████▏    | 343/669 [3:03:36<4:26:11, 48.99s/it]
 51%|█████▏    | 344/669 [3:04:20<4:17:13, 47.49s/it]
                                                     
{'loss': 0.4835, 'grad_norm': 0.5862261680615611, 'learning_rate': 2.465857359635812e-06, 'epoch': 1.52}

 51%|█████▏    | 344/669 [3:04:20<4:17:13, 47.49s/it]
 52%|█████▏    | 345/669 [3:05:04<4:10:50, 46.45s/it]
                                                     
{'loss': 0.4638, 'grad_norm': 0.536975433650959, 'learning_rate': 2.458270106221548e-06, 'epoch': 1.52}

 52%|█████▏    | 345/669 [3:05:04<4:10:50, 46.45s/it]
 52%|█████▏    | 346/669 [3:05:48<4:06:03, 45.71s/it]
                                                     
{'loss': 0.4832, 'grad_norm': 0.5142279881627296, 'learning_rate': 2.450682852807284e-06, 'epoch': 1.53}

 52%|█████▏    | 346/669 [3:05:48<4:06:03, 45.71s/it]
 52%|█████▏    | 347/669 [3:06:32<4:02:26, 45.18s/it]
                                                     
{'loss': 0.4857, 'grad_norm': 0.5124062345457664, 'learning_rate': 2.44309559939302e-06, 'epoch': 1.53}

 52%|█████▏    | 347/669 [3:06:32<4:02:26, 45.18s/it]
 52%|█████▏    | 348/669 [3:07:16<3:59:44, 44.81s/it]
                                                     
{'loss': 0.4807, 'grad_norm': 0.509905421315166, 'learning_rate': 2.435508345978756e-06, 'epoch': 1.54}

 52%|█████▏    | 348/669 [3:07:16<3:59:44, 44.81s/it]
 52%|█████▏    | 349/669 [3:08:00<3:57:36, 44.55s/it]
                                                     
{'loss': 0.4722, 'grad_norm': 0.5275507665643101, 'learning_rate': 2.427921092564492e-06, 'epoch': 1.54}

 52%|█████▏    | 349/669 [3:08:00<3:57:36, 44.55s/it]
 52%|█████▏    | 350/669 [3:08:44<3:55:49, 44.36s/it]
                                                     
{'loss': 0.4941, 'grad_norm': 0.5003951978639241, 'learning_rate': 2.420333839150228e-06, 'epoch': 1.55}

 52%|█████▏    | 350/669 [3:08:44<3:55:49, 44.36s/it]
 52%|█████▏    | 351/669 [3:09:28<3:54:28, 44.24s/it]
                                                     
{'loss': 0.5376, 'grad_norm': 0.5251614039487292, 'learning_rate': 2.4127465857359637e-06, 'epoch': 1.55}

 52%|█████▏    | 351/669 [3:09:28<3:54:28, 44.24s/it]
 53%|█████▎    | 352/669 [3:10:12<3:53:12, 44.14s/it]
                                                     
{'loss': 0.5001, 'grad_norm': 0.5128097582851606, 'learning_rate': 2.4051593323217e-06, 'epoch': 1.55}

 53%|█████▎    | 352/669 [3:10:12<3:53:12, 44.14s/it]
 53%|█████▎    | 353/669 [3:10:56<3:52:07, 44.07s/it]
                                                     
{'loss': 0.4721, 'grad_norm': 0.4971288144993544, 'learning_rate': 2.397572078907436e-06, 'epoch': 1.56}

 53%|█████▎    | 353/669 [3:10:56<3:52:07, 44.07s/it]
 53%|█████▎    | 354/669 [3:11:40<3:51:11, 44.04s/it]
                                                     
{'loss': 0.5192, 'grad_norm': 0.5111491573396423, 'learning_rate': 2.3899848254931717e-06, 'epoch': 1.56}

 53%|█████▎    | 354/669 [3:11:40<3:51:11, 44.04s/it]
 53%|█████▎    | 355/669 [3:12:24<3:50:22, 44.02s/it]
                                                     
{'loss': 0.4541, 'grad_norm': 0.5025703950184777, 'learning_rate': 2.382397572078908e-06, 'epoch': 1.57}

 53%|█████▎    | 355/669 [3:12:24<3:50:22, 44.02s/it]
 53%|█████▎    | 356/669 [3:13:08<3:49:36, 44.02s/it]
                                                     
{'loss': 0.4789, 'grad_norm': 0.5011840454618501, 'learning_rate': 2.3748103186646435e-06, 'epoch': 1.57}

 53%|█████▎    | 356/669 [3:13:08<3:49:36, 44.02s/it]
 53%|█████▎    | 357/669 [3:13:52<3:48:47, 44.00s/it]
                                                     
{'loss': 0.497, 'grad_norm': 0.4981690031939279, 'learning_rate': 2.3672230652503792e-06, 'epoch': 1.58}

 53%|█████▎    | 357/669 [3:13:52<3:48:47, 44.00s/it]
 54%|█████▎    | 358/669 [3:14:36<3:47:58, 43.98s/it]
                                                     
{'loss': 0.4655, 'grad_norm': 0.4994506284814992, 'learning_rate': 2.3596358118361154e-06, 'epoch': 1.58}

 54%|█████▎    | 358/669 [3:14:36<3:47:58, 43.98s/it]
 54%|█████▎    | 359/669 [3:15:20<3:47:08, 43.96s/it]
                                                     
{'loss': 0.4883, 'grad_norm': 0.49694019288140645, 'learning_rate': 2.3520485584218515e-06, 'epoch': 1.59}

 54%|█████▎    | 359/669 [3:15:20<3:47:08, 43.96s/it]
 54%|█████▍    | 360/669 [3:16:04<3:46:25, 43.96s/it]
                                                     
{'loss': 0.4615, 'grad_norm': 0.5103222760000443, 'learning_rate': 2.344461305007587e-06, 'epoch': 1.59}

 54%|█████▍    | 360/669 [3:16:04<3:46:25, 43.96s/it]
 54%|█████▍    | 361/669 [3:16:48<3:45:41, 43.97s/it]
                                                     
{'loss': 0.4748, 'grad_norm': 0.4960772434204748, 'learning_rate': 2.3368740515933233e-06, 'epoch': 1.59}

 54%|█████▍    | 361/669 [3:16:48<3:45:41, 43.97s/it]
 54%|█████▍    | 362/669 [3:18:46<5:39:35, 66.37s/it]
                                                     
{'loss': 0.461, 'grad_norm': 0.5372239368130248, 'learning_rate': 2.3292867981790594e-06, 'epoch': 1.6}

 54%|█████▍    | 362/669 [3:18:46<5:39:35, 66.37s/it]
 54%|█████▍    | 363/669 [3:19:30<5:03:57, 59.60s/it]
                                                     
{'loss': 0.4985, 'grad_norm': 0.5002183877541576, 'learning_rate': 2.321699544764795e-06, 'epoch': 1.6}

 54%|█████▍    | 363/669 [3:19:30<5:03:57, 59.60s/it]
 54%|█████▍    | 364/669 [3:20:14<4:39:08, 54.91s/it]
                                                     
{'loss': 0.4846, 'grad_norm': 0.5065612982842012, 'learning_rate': 2.3141122913505313e-06, 'epoch': 1.61}

 54%|█████▍    | 364/669 [3:20:14<4:39:08, 54.91s/it]
 55%|█████▍    | 365/669 [3:20:58<4:21:35, 51.63s/it]
                                                     
{'loss': 0.4769, 'grad_norm': 0.5294145045424744, 'learning_rate': 2.3065250379362674e-06, 'epoch': 1.61}

 55%|█████▍    | 365/669 [3:20:58<4:21:35, 51.63s/it]
 55%|█████▍    | 366/669 [3:21:42<4:09:04, 49.32s/it]
                                                     
{'loss': 0.4738, 'grad_norm': 0.5019842435412675, 'learning_rate': 2.298937784522003e-06, 'epoch': 1.62}

 55%|█████▍    | 366/669 [3:21:42<4:09:04, 49.32s/it]
 55%|█████▍    | 367/669 [3:22:26<4:00:07, 47.71s/it]
                                                     
{'loss': 0.4773, 'grad_norm': 0.4973490272845337, 'learning_rate': 2.291350531107739e-06, 'epoch': 1.62}

 55%|█████▍    | 367/669 [3:22:26<4:00:07, 47.71s/it]
 55%|█████▌    | 368/669 [3:23:10<3:53:47, 46.60s/it]
                                                     
{'loss': 0.4921, 'grad_norm': 0.4917027064554639, 'learning_rate': 2.283763277693475e-06, 'epoch': 1.63}

 55%|█████▌    | 368/669 [3:23:10<3:53:47, 46.60s/it]
 55%|█████▌    | 369/669 [3:23:54<3:48:58, 45.80s/it]
                                                     
{'loss': 0.4823, 'grad_norm': 0.5191932977137537, 'learning_rate': 2.276176024279211e-06, 'epoch': 1.63}

 55%|█████▌    | 369/669 [3:23:54<3:48:58, 45.80s/it]
 55%|█████▌    | 370/669 [3:24:38<3:45:27, 45.24s/it]
                                                     
{'loss': 0.4631, 'grad_norm': 0.5130853619632637, 'learning_rate': 2.268588770864947e-06, 'epoch': 1.63}

 55%|█████▌    | 370/669 [3:24:38<3:45:27, 45.24s/it]
 55%|█████▌    | 371/669 [3:25:22<3:42:46, 44.85s/it]
                                                     
{'loss': 0.456, 'grad_norm': 0.5402133951776769, 'learning_rate': 2.261001517450683e-06, 'epoch': 1.64}

 55%|█████▌    | 371/669 [3:25:22<3:42:46, 44.85s/it]
 56%|█████▌    | 372/669 [3:26:06<3:40:37, 44.57s/it]
                                                     
{'loss': 0.4715, 'grad_norm': 0.5136201214507364, 'learning_rate': 2.253414264036419e-06, 'epoch': 1.64}

 56%|█████▌    | 372/669 [3:26:06<3:40:37, 44.57s/it]
 56%|█████▌    | 373/669 [3:26:50<3:39:06, 44.41s/it]
                                                     
{'loss': 0.465, 'grad_norm': 0.5397223114168315, 'learning_rate': 2.245827010622155e-06, 'epoch': 1.65}

 56%|█████▌    | 373/669 [3:26:50<3:39:06, 44.41s/it]
 56%|█████▌    | 374/669 [3:27:34<3:37:38, 44.27s/it]
                                                     
{'loss': 0.4526, 'grad_norm': 0.5178224930093817, 'learning_rate': 2.238239757207891e-06, 'epoch': 1.65}

 56%|█████▌    | 374/669 [3:27:34<3:37:38, 44.27s/it]
 56%|█████▌    | 375/669 [3:28:18<3:36:21, 44.15s/it]
                                                     
{'loss': 0.4873, 'grad_norm': 0.5399618966600667, 'learning_rate': 2.230652503793627e-06, 'epoch': 1.66}

 56%|█████▌    | 375/669 [3:28:18<3:36:21, 44.15s/it]
 56%|█████▌    | 376/669 [3:29:02<3:35:22, 44.10s/it]
                                                     
{'loss': 0.4771, 'grad_norm': 0.5106529140250409, 'learning_rate': 2.223065250379363e-06, 'epoch': 1.66}

 56%|█████▌    | 376/669 [3:29:02<3:35:22, 44.10s/it]
 56%|█████▋    | 377/669 [3:29:45<3:34:21, 44.05s/it]
                                                     
{'loss': 0.4743, 'grad_norm': 0.5460841160518792, 'learning_rate': 2.2154779969650988e-06, 'epoch': 1.67}

 56%|█████▋    | 377/669 [3:29:45<3:34:21, 44.05s/it]
 57%|█████▋    | 378/669 [3:30:29<3:33:24, 44.00s/it]
                                                     
{'loss': 0.4552, 'grad_norm': 0.5846706720135237, 'learning_rate': 2.207890743550835e-06, 'epoch': 1.67}

 57%|█████▋    | 378/669 [3:30:29<3:33:24, 44.00s/it]
 57%|█████▋    | 379/669 [3:31:13<3:32:31, 43.97s/it]
                                                     
{'loss': 0.495, 'grad_norm': 0.534699591229056, 'learning_rate': 2.2003034901365706e-06, 'epoch': 1.67}

 57%|█████▋    | 379/669 [3:31:13<3:32:31, 43.97s/it]
 57%|█████▋    | 380/669 [3:31:57<3:31:47, 43.97s/it]
                                                     
{'loss': 0.4824, 'grad_norm': 0.5377842578832556, 'learning_rate': 2.1927162367223067e-06, 'epoch': 1.68}

 57%|█████▋    | 380/669 [3:31:57<3:31:47, 43.97s/it]
 57%|█████▋    | 381/669 [3:32:41<3:30:59, 43.96s/it]
                                                     
{'loss': 0.4708, 'grad_norm': 0.5799148072307362, 'learning_rate': 2.185128983308043e-06, 'epoch': 1.68}

 57%|█████▋    | 381/669 [3:32:41<3:30:59, 43.96s/it]
 57%|█████▋    | 382/669 [3:33:25<3:30:11, 43.94s/it]
                                                     
{'loss': 0.4631, 'grad_norm': 0.5076181067753225, 'learning_rate': 2.1775417298937786e-06, 'epoch': 1.69}

 57%|█████▋    | 382/669 [3:33:25<3:30:11, 43.94s/it]
 57%|█████▋    | 383/669 [3:34:09<3:29:29, 43.95s/it]
                                                     
{'loss': 0.4676, 'grad_norm': 0.5025103762077108, 'learning_rate': 2.1699544764795147e-06, 'epoch': 1.69}

 57%|█████▋    | 383/669 [3:34:09<3:29:29, 43.95s/it]
 57%|█████▋    | 384/669 [3:34:53<3:28:51, 43.97s/it]
                                                     
{'loss': 0.4694, 'grad_norm': 0.48231447315943815, 'learning_rate': 2.162367223065251e-06, 'epoch': 1.7}

 57%|█████▋    | 384/669 [3:34:53<3:28:51, 43.97s/it]
 58%|█████▊    | 385/669 [3:35:37<3:28:03, 43.96s/it]
                                                     
{'loss': 0.4607, 'grad_norm': 0.5003437712619672, 'learning_rate': 2.1547799696509865e-06, 'epoch': 1.7}

 58%|█████▊    | 385/669 [3:35:37<3:28:03, 43.96s/it]
 58%|█████▊    | 386/669 [3:36:21<3:27:18, 43.95s/it]
                                                     
{'loss': 0.5253, 'grad_norm': 0.5149834201342615, 'learning_rate': 2.1471927162367226e-06, 'epoch': 1.71}

 58%|█████▊    | 386/669 [3:36:21<3:27:18, 43.95s/it]
 58%|█████▊    | 387/669 [3:38:19<5:11:43, 66.32s/it]
                                                     
{'loss': 0.5352, 'grad_norm': 0.5406857228194807, 'learning_rate': 2.1396054628224588e-06, 'epoch': 1.71}

 58%|█████▊    | 387/669 [3:38:19<5:11:43, 66.32s/it]
 58%|█████▊    | 388/669 [3:39:03<4:38:56, 59.56s/it]
                                                     
{'loss': 0.4936, 'grad_norm': 0.5204717082880796, 'learning_rate': 2.1320182094081945e-06, 'epoch': 1.72}

 58%|█████▊    | 388/669 [3:39:03<4:38:56, 59.56s/it]
 58%|█████▊    | 389/669 [3:39:47<4:16:07, 54.88s/it]
                                                     
{'loss': 0.4594, 'grad_norm': 0.5202749448515015, 'learning_rate': 2.12443095599393e-06, 'epoch': 1.72}

 58%|█████▊    | 389/669 [3:39:47<4:16:07, 54.88s/it]
 58%|█████▊    | 390/669 [3:40:31<3:59:55, 51.60s/it]
                                                     
{'loss': 0.4793, 'grad_norm': 0.4952910050954741, 'learning_rate': 2.1168437025796663e-06, 'epoch': 1.72}

 58%|█████▊    | 390/669 [3:40:31<3:59:55, 51.60s/it]
 58%|█████▊    | 391/669 [3:41:15<3:48:28, 49.31s/it]
                                                     
{'loss': 0.4745, 'grad_norm': 0.48637344511275765, 'learning_rate': 2.109256449165402e-06, 'epoch': 1.73}

 58%|█████▊    | 391/669 [3:41:15<3:48:28, 49.31s/it]
 59%|█████▊    | 392/669 [3:41:59<3:40:12, 47.70s/it]
                                                     
{'loss': 0.4652, 'grad_norm': 0.4906271558417189, 'learning_rate': 2.101669195751138e-06, 'epoch': 1.73}

 59%|█████▊    | 392/669 [3:41:59<3:40:12, 47.70s/it]/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(

 59%|█████▊    | 393/669 [3:43:16<4:20:20, 56.60s/it]
                                                     
{'loss': 0.5298, 'grad_norm': 0.5090727761430808, 'learning_rate': 2.0940819423368742e-06, 'epoch': 1.74}

 59%|█████▊    | 393/669 [3:43:16<4:20:20, 56.60s/it]
 59%|█████▉    | 394/669 [3:44:00<4:02:01, 52.81s/it]
                                                     
{'loss': 0.4838, 'grad_norm': 0.4866596430096002, 'learning_rate': 2.08649468892261e-06, 'epoch': 1.74}

 59%|█████▉    | 394/669 [3:44:00<4:02:01, 52.81s/it]
 59%|█████▉    | 395/669 [3:44:44<3:49:00, 50.15s/it]
                                                     
{'loss': 0.4737, 'grad_norm': 0.49259128094078225, 'learning_rate': 2.078907435508346e-06, 'epoch': 1.75}

 59%|█████▉    | 395/669 [3:44:44<3:49:00, 50.15s/it]
 59%|█████▉    | 396/669 [3:45:28<3:39:41, 48.28s/it]
                                                     
{'loss': 0.4777, 'grad_norm': 0.5031034927365164, 'learning_rate': 2.071320182094082e-06, 'epoch': 1.75}

 59%|█████▉    | 396/669 [3:45:28<3:39:41, 48.28s/it]
 59%|█████▉    | 397/669 [3:46:12<3:32:54, 46.97s/it]
                                                     
{'loss': 0.454, 'grad_norm': 0.48520021809068, 'learning_rate': 2.063732928679818e-06, 'epoch': 1.76}

 59%|█████▉    | 397/669 [3:46:12<3:32:54, 46.97s/it]
 59%|█████▉    | 398/669 [3:46:56<3:28:20, 46.13s/it]
                                                     
{'loss': 0.5199, 'grad_norm': 0.5217902917498033, 'learning_rate': 2.056145675265554e-06, 'epoch': 1.76}

 59%|█████▉    | 398/669 [3:46:56<3:28:20, 46.13s/it]
 60%|█████▉    | 399/669 [3:47:40<3:24:34, 45.46s/it]
                                                     
{'loss': 0.4649, 'grad_norm': 0.4963706354662438, 'learning_rate': 2.04855842185129e-06, 'epoch': 1.76}

 60%|█████▉    | 399/669 [3:47:40<3:24:34, 45.46s/it]
 60%|█████▉    | 400/669 [3:48:24<3:22:01, 45.06s/it]
                                                     
{'loss': 0.4887, 'grad_norm': 0.49552580249683814, 'learning_rate': 2.040971168437026e-06, 'epoch': 1.77}

 60%|█████▉    | 400/669 [3:48:24<3:22:01, 45.06s/it]
 60%|█████▉    | 401/669 [3:49:08<3:19:46, 44.73s/it]
                                                     
{'loss': 0.4757, 'grad_norm': 0.49468231483436587, 'learning_rate': 2.033383915022762e-06, 'epoch': 1.77}

 60%|█████▉    | 401/669 [3:49:08<3:19:46, 44.73s/it]
 60%|██████    | 402/669 [3:49:52<3:18:02, 44.50s/it]
                                                     
{'loss': 0.4846, 'grad_norm': 0.47502769186897137, 'learning_rate': 2.0257966616084977e-06, 'epoch': 1.78}

 60%|██████    | 402/669 [3:49:52<3:18:02, 44.50s/it]
 60%|██████    | 403/669 [3:50:36<3:16:28, 44.32s/it]
                                                     
{'loss': 0.4692, 'grad_norm': 0.5089134108324753, 'learning_rate': 2.018209408194234e-06, 'epoch': 1.78}

 60%|██████    | 403/669 [3:50:36<3:16:28, 44.32s/it]
 60%|██████    | 404/669 [3:51:20<3:15:15, 44.21s/it]
                                                     
{'loss': 0.4665, 'grad_norm': 0.4981731250179096, 'learning_rate': 2.01062215477997e-06, 'epoch': 1.79}

 60%|██████    | 404/669 [3:51:20<3:15:15, 44.21s/it]
 61%|██████    | 405/669 [3:52:04<3:14:09, 44.13s/it]
                                                     
{'loss': 0.4606, 'grad_norm': 0.4690174096990647, 'learning_rate': 2.0030349013657056e-06, 'epoch': 1.79}

 61%|██████    | 405/669 [3:52:04<3:14:09, 44.13s/it]
 61%|██████    | 406/669 [3:52:48<3:13:12, 44.08s/it]
                                                     
{'loss': 0.4847, 'grad_norm': 0.4914411018326139, 'learning_rate': 1.9954476479514418e-06, 'epoch': 1.8}

 61%|██████    | 406/669 [3:52:48<3:13:12, 44.08s/it]
 61%|██████    | 407/669 [3:53:32<3:12:19, 44.04s/it]
                                                     
{'loss': 0.488, 'grad_norm': 0.5513335287615446, 'learning_rate': 1.987860394537178e-06, 'epoch': 1.8}

 61%|██████    | 407/669 [3:53:32<3:12:19, 44.04s/it]
 61%|██████    | 408/669 [3:54:16<3:11:31, 44.03s/it]
                                                     
{'loss': 0.465, 'grad_norm': 0.48992847389625854, 'learning_rate': 1.9802731411229136e-06, 'epoch': 1.8}

 61%|██████    | 408/669 [3:54:16<3:11:31, 44.03s/it]
 61%|██████    | 409/669 [3:55:00<3:10:38, 43.99s/it]
                                                     
{'loss': 0.4872, 'grad_norm': 0.5185912675971102, 'learning_rate': 1.9726858877086497e-06, 'epoch': 1.81}

 61%|██████    | 409/669 [3:55:00<3:10:38, 43.99s/it]
 61%|██████▏   | 410/669 [3:55:44<3:10:05, 44.04s/it]
                                                     
{'loss': 0.4534, 'grad_norm': 0.47300994432774746, 'learning_rate': 1.965098634294386e-06, 'epoch': 1.81}

 61%|██████▏   | 410/669 [3:55:44<3:10:05, 44.04s/it]
 61%|██████▏   | 411/669 [3:56:28<3:09:11, 44.00s/it]
                                                     
{'loss': 0.4668, 'grad_norm': 0.5034275716173061, 'learning_rate': 1.9575113808801215e-06, 'epoch': 1.82}

 61%|██████▏   | 411/669 [3:56:28<3:09:11, 44.00s/it]
 62%|██████▏   | 412/669 [3:58:27<4:44:32, 66.43s/it]
                                                     
{'loss': 0.491, 'grad_norm': 0.46435516708821234, 'learning_rate': 1.9499241274658577e-06, 'epoch': 1.82}

 62%|██████▏   | 412/669 [3:58:27<4:44:32, 66.43s/it]
 62%|██████▏   | 413/669 [3:59:10<4:14:24, 59.63s/it]
                                                     
{'loss': 0.4793, 'grad_norm': 0.4915994950607221, 'learning_rate': 1.9423368740515934e-06, 'epoch': 1.83}

 62%|██████▏   | 413/669 [3:59:10<4:14:24, 59.63s/it]
 62%|██████▏   | 414/669 [3:59:54<3:53:28, 54.94s/it]
                                                     
{'loss': 0.4897, 'grad_norm': 0.49834550842622505, 'learning_rate': 1.9347496206373295e-06, 'epoch': 1.83}

 62%|██████▏   | 414/669 [3:59:54<3:53:28, 54.94s/it]
 62%|██████▏   | 415/669 [4:00:38<3:38:42, 51.66s/it]
                                                     
{'loss': 0.5167, 'grad_norm': 0.5275123854480922, 'learning_rate': 1.9271623672230656e-06, 'epoch': 1.84}

 62%|██████▏   | 415/669 [4:00:38<3:38:42, 51.66s/it]
 62%|██████▏   | 416/669 [4:01:22<3:28:09, 49.37s/it]
                                                     
{'loss': 0.5522, 'grad_norm': 0.501447167095016, 'learning_rate': 1.9195751138088013e-06, 'epoch': 1.84}

 62%|██████▏   | 416/669 [4:01:22<3:28:09, 49.37s/it]
 62%|██████▏   | 417/669 [4:02:06<3:20:29, 47.74s/it]
                                                     
{'loss': 0.464, 'grad_norm': 0.4929358533949542, 'learning_rate': 1.9119878603945374e-06, 'epoch': 1.84}

 62%|██████▏   | 417/669 [4:02:06<3:20:29, 47.74s/it]
 62%|██████▏   | 418/669 [4:02:50<3:14:57, 46.60s/it]
                                                     
{'loss': 0.4707, 'grad_norm': 0.4753893398726578, 'learning_rate': 1.9044006069802734e-06, 'epoch': 1.85}

 62%|██████▏   | 418/669 [4:02:50<3:14:57, 46.60s/it]
 63%|██████▎   | 419/669 [4:03:34<3:10:50, 45.80s/it]
                                                     
{'loss': 0.4674, 'grad_norm': 0.49249135995729143, 'learning_rate': 1.8968133535660093e-06, 'epoch': 1.85}

 63%|██████▎   | 419/669 [4:03:34<3:10:50, 45.80s/it]
 63%|██████▎   | 420/669 [4:04:18<3:07:48, 45.26s/it]
                                                     
{'loss': 0.4731, 'grad_norm': 0.4959065755228421, 'learning_rate': 1.8892261001517454e-06, 'epoch': 1.86}

 63%|██████▎   | 420/669 [4:04:18<3:07:48, 45.26s/it]
 63%|██████▎   | 421/669 [4:05:02<3:05:23, 44.85s/it]
                                                     
{'loss': 0.4688, 'grad_norm': 0.4830053823448673, 'learning_rate': 1.8816388467374813e-06, 'epoch': 1.86}

 63%|██████▎   | 421/669 [4:05:02<3:05:23, 44.85s/it]
 63%|██████▎   | 422/669 [4:05:46<3:03:34, 44.59s/it]
                                                     
{'loss': 0.4777, 'grad_norm': 0.5026774544021604, 'learning_rate': 1.874051593323217e-06, 'epoch': 1.87}

 63%|██████▎   | 422/669 [4:05:46<3:03:34, 44.59s/it]
 63%|██████▎   | 423/669 [4:06:30<3:02:03, 44.41s/it]
                                                     
{'loss': 0.4532, 'grad_norm': 0.48376669794821286, 'learning_rate': 1.866464339908953e-06, 'epoch': 1.87}

 63%|██████▎   | 423/669 [4:06:30<3:02:03, 44.41s/it]
 63%|██████▎   | 424/669 [4:07:14<3:00:45, 44.27s/it]
                                                     
{'loss': 0.4749, 'grad_norm': 0.5036910075540445, 'learning_rate': 1.858877086494689e-06, 'epoch': 1.88}

 63%|██████▎   | 424/669 [4:07:14<3:00:45, 44.27s/it]
 64%|██████▎   | 425/669 [4:07:58<2:59:37, 44.17s/it]
                                                     
{'loss': 0.4582, 'grad_norm': 0.464687327497754, 'learning_rate': 1.851289833080425e-06, 'epoch': 1.88}

 64%|██████▎   | 425/669 [4:07:58<2:59:37, 44.17s/it]
 64%|██████▎   | 426/669 [4:08:42<2:58:37, 44.11s/it]
                                                     
{'loss': 0.4954, 'grad_norm': 0.4713537790626329, 'learning_rate': 1.8437025796661609e-06, 'epoch': 1.88}

 64%|██████▎   | 426/669 [4:08:42<2:58:37, 44.11s/it]
 64%|██████▍   | 427/669 [4:09:26<2:57:50, 44.09s/it]
                                                     
{'loss': 0.4777, 'grad_norm': 0.4964027557908742, 'learning_rate': 1.8361153262518968e-06, 'epoch': 1.89}

 64%|██████▍   | 427/669 [4:09:26<2:57:50, 44.09s/it]
 64%|██████▍   | 428/669 [4:10:10<2:56:55, 44.05s/it]
                                                     
{'loss': 0.4723, 'grad_norm': 0.5022160749373462, 'learning_rate': 1.828528072837633e-06, 'epoch': 1.89}

 64%|██████▍   | 428/669 [4:10:10<2:56:55, 44.05s/it]
 64%|██████▍   | 429/669 [4:10:54<2:56:04, 44.02s/it]
                                                     
{'loss': 0.4619, 'grad_norm': 0.4971783807561644, 'learning_rate': 1.8209408194233688e-06, 'epoch': 1.9}

 64%|██████▍   | 429/669 [4:10:54<2:56:04, 44.02s/it]
 64%|██████▍   | 430/669 [4:11:38<2:55:13, 43.99s/it]
                                                     
{'loss': 0.4923, 'grad_norm': 0.5020482529233811, 'learning_rate': 1.8133535660091047e-06, 'epoch': 1.9}

 64%|██████▍   | 430/669 [4:11:38<2:55:13, 43.99s/it]
 64%|██████▍   | 431/669 [4:12:22<2:54:24, 43.97s/it]
                                                     
{'loss': 0.4936, 'grad_norm': 0.5027930539648167, 'learning_rate': 1.8057663125948407e-06, 'epoch': 1.91}

 64%|██████▍   | 431/669 [4:12:22<2:54:24, 43.97s/it]
 65%|██████▍   | 432/669 [4:13:06<2:53:43, 43.98s/it]
                                                     
{'loss': 0.4647, 'grad_norm': 0.5048109425957825, 'learning_rate': 1.7981790591805768e-06, 'epoch': 1.91}

 65%|██████▍   | 432/669 [4:13:06<2:53:43, 43.98s/it]
 65%|██████▍   | 433/669 [4:13:50<2:52:54, 43.96s/it]
                                                     
{'loss': 0.4962, 'grad_norm': 0.5152625504278344, 'learning_rate': 1.7905918057663127e-06, 'epoch': 1.92}

 65%|██████▍   | 433/669 [4:13:50<2:52:54, 43.96s/it]
 65%|██████▍   | 434/669 [4:14:34<2:52:11, 43.96s/it]
                                                     
{'loss': 0.4643, 'grad_norm': 0.5486472960937676, 'learning_rate': 1.7830045523520486e-06, 'epoch': 1.92}

 65%|██████▍   | 434/669 [4:14:34<2:52:11, 43.96s/it]
 65%|██████▌   | 435/669 [4:15:18<2:51:27, 43.96s/it]
                                                     
{'loss': 0.4838, 'grad_norm': 0.5074159325604022, 'learning_rate': 1.7754172989377847e-06, 'epoch': 1.93}

 65%|██████▌   | 435/669 [4:15:18<2:51:27, 43.96s/it]
 65%|██████▌   | 436/669 [4:16:02<2:50:42, 43.96s/it]
                                                     
{'loss': 0.4953, 'grad_norm': 0.487237277239409, 'learning_rate': 1.7678300455235207e-06, 'epoch': 1.93}

 65%|██████▌   | 436/669 [4:16:02<2:50:42, 43.96s/it]
 65%|██████▌   | 437/669 [4:18:02<4:18:30, 66.86s/it]
                                                     
{'loss': 0.4669, 'grad_norm': 0.5030243156211434, 'learning_rate': 1.7602427921092566e-06, 'epoch': 1.93}

 65%|██████▌   | 437/669 [4:18:02<4:18:30, 66.86s/it]
 65%|██████▌   | 438/669 [4:18:46<3:50:42, 59.92s/it]
                                                     
{'loss': 0.4645, 'grad_norm': 0.54024162061195, 'learning_rate': 1.7526555386949925e-06, 'epoch': 1.94}

 65%|██████▌   | 438/669 [4:18:46<3:50:42, 59.92s/it]
 66%|██████▌   | 439/669 [4:19:30<3:31:18, 55.13s/it]
                                                     
{'loss': 0.487, 'grad_norm': 0.4924548318787792, 'learning_rate': 1.7450682852807286e-06, 'epoch': 1.94}

 66%|██████▌   | 439/669 [4:19:30<3:31:18, 55.13s/it]
 66%|██████▌   | 440/669 [4:20:13<3:17:33, 51.76s/it]
                                                     
{'loss': 0.4904, 'grad_norm': 0.5002700607677886, 'learning_rate': 1.7374810318664645e-06, 'epoch': 1.95}

 66%|██████▌   | 440/669 [4:20:13<3:17:33, 51.76s/it]
 66%|██████▌   | 441/669 [4:20:57<3:07:46, 49.41s/it]
                                                     
{'loss': 0.4721, 'grad_norm': 0.49052978017153487, 'learning_rate': 1.7298937784522004e-06, 'epoch': 1.95}

 66%|██████▌   | 441/669 [4:20:57<3:07:46, 49.41s/it]
 66%|██████▌   | 442/669 [4:21:41<3:00:45, 47.78s/it]
                                                     
{'loss': 0.4674, 'grad_norm': 0.5330277427973065, 'learning_rate': 1.7223065250379363e-06, 'epoch': 1.96}

 66%|██████▌   | 442/669 [4:21:41<3:00:45, 47.78s/it]
 66%|██████▌   | 443/669 [4:22:25<2:55:36, 46.62s/it]
                                                     
{'loss': 0.4735, 'grad_norm': 0.49644600432443986, 'learning_rate': 1.7147192716236725e-06, 'epoch': 1.96}

 66%|██████▌   | 443/669 [4:22:25<2:55:36, 46.62s/it]
 66%|██████▋   | 444/669 [4:23:09<2:51:55, 45.85s/it]
                                                     
{'loss': 0.4758, 'grad_norm': 0.4763428511189237, 'learning_rate': 1.7071320182094084e-06, 'epoch': 1.97}

 66%|██████▋   | 444/669 [4:23:09<2:51:55, 45.85s/it]
 67%|██████▋   | 445/669 [4:23:53<2:49:03, 45.28s/it]
                                                     
{'loss': 0.5025, 'grad_norm': 0.49662051116381534, 'learning_rate': 1.6995447647951443e-06, 'epoch': 1.97}

 67%|██████▋   | 445/669 [4:23:53<2:49:03, 45.28s/it]
 67%|██████▋   | 446/669 [4:24:37<2:46:49, 44.89s/it]
                                                     
{'loss': 0.4708, 'grad_norm': 0.5368027446646121, 'learning_rate': 1.6919575113808804e-06, 'epoch': 1.97}

 67%|██████▋   | 446/669 [4:24:37<2:46:49, 44.89s/it]
 67%|██████▋   | 447/669 [4:25:21<2:44:58, 44.59s/it]
                                                     
{'loss': 0.4859, 'grad_norm': 0.49270124177132796, 'learning_rate': 1.6843702579666163e-06, 'epoch': 1.98}

 67%|██████▋   | 447/669 [4:25:21<2:44:58, 44.59s/it]
 67%|██████▋   | 448/669 [4:26:05<2:43:36, 44.42s/it]
                                                     
{'loss': 0.4542, 'grad_norm': 0.4886244120662818, 'learning_rate': 1.6767830045523523e-06, 'epoch': 1.98}

 67%|██████▋   | 448/669 [4:26:05<2:43:36, 44.42s/it][2024-08-02 08:06:05,525] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:05,525] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:05,525] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:05,525] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:05,525] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:05,525] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:05,525] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:05,525] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:05,526] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:05,526] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:05,526] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:05,526] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:05,526] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:05,526] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:05,526] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:05,527] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:08,790] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:08,791] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:08,792] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:08,793] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


  0%|          | 0/8 [00:00<?, ?it/s][2024-08-02 08:06:12,354] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:12,355] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 25%|██▌       | 2/8 [00:03<00:10,  1.78s/it][2024-08-02 08:06:15,903] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:15,904] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 38%|███▊      | 3/8 [00:07<00:12,  2.52s/it][2024-08-02 08:06:19,451] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:19,452] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 50%|█████     | 4/8 [00:10<00:11,  2.90s/it][2024-08-02 08:06:23,183] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:23,184] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 62%|██████▎   | 5/8 [00:14<00:09,  3.19s/it][2024-08-02 08:06:26,725] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:26,726] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 75%|███████▌  | 6/8 [00:17<00:06,  3.31s/it][2024-08-02 08:06:30,269] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:30,270] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 88%|████████▊ | 7/8 [00:21<00:03,  3.38s/it][2024-08-02 08:06:33,841] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:33,842] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


100%|██████████| 8/8 [00:25<00:00,  3.44s/it][2024-08-02 08:06:37,489] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 08:06:37,490] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


9it [00:28,  3.51s/it]                       
                                                     

                      
{'eval_loss': 0.49883514642715454, 'eval_runtime': 32.2692, 'eval_samples_per_second': 82.741, 'eval_steps_per_second': 5.175, 'epoch': 1.98}

 67%|██████▋   | 448/669 [4:26:37<2:43:36, 44.42s/it]

9it [00:28,  3.51s/it]

                      /root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(

 67%|██████▋   | 449/669 [4:27:55<3:54:39, 64.00s/it]
                                                     
{'loss': 0.4752, 'grad_norm': 0.48474622370426634, 'learning_rate': 1.6691957511380882e-06, 'epoch': 1.99}

 67%|██████▋   | 449/669 [4:27:55<3:54:39, 64.00s/it]
 67%|██████▋   | 450/669 [4:28:39<3:31:59, 58.08s/it]
                                                     
{'loss': 0.4835, 'grad_norm': 0.5089879136002737, 'learning_rate': 1.6616084977238243e-06, 'epoch': 1.99}

 67%|██████▋   | 450/669 [4:28:39<3:31:59, 58.08s/it]
 67%|██████▋   | 451/669 [4:29:23<3:15:56, 53.93s/it]
                                                     
{'loss': 0.4685, 'grad_norm': 0.540987114653738, 'learning_rate': 1.6540212443095602e-06, 'epoch': 2.0}

 67%|██████▋   | 451/669 [4:29:23<3:15:56, 53.93s/it]
 68%|██████▊   | 452/669 [4:30:07<3:04:12, 50.93s/it]
                                                     
{'loss': 0.4729, 'grad_norm': 0.5330263791441335, 'learning_rate': 1.6464339908952961e-06, 'epoch': 2.0}

 68%|██████▊   | 452/669 [4:30:07<3:04:12, 50.93s/it]
 68%|██████▊   | 453/669 [4:30:51<2:55:52, 48.85s/it]
                                                     
{'loss': 0.4739, 'grad_norm': 0.5025591514510047, 'learning_rate': 1.638846737481032e-06, 'epoch': 2.01}

 68%|██████▊   | 453/669 [4:30:51<2:55:52, 48.85s/it]
 68%|██████▊   | 454/669 [4:31:35<2:49:45, 47.38s/it]
                                                     
{'loss': 0.4772, 'grad_norm': 0.4938210284492649, 'learning_rate': 1.6312594840667682e-06, 'epoch': 2.01}

 68%|██████▊   | 454/669 [4:31:35<2:49:45, 47.38s/it]
 68%|██████▊   | 455/669 [4:32:19<2:45:17, 46.34s/it]
                                                     
{'loss': 0.4651, 'grad_norm': 0.5590423643222178, 'learning_rate': 1.6236722306525039e-06, 'epoch': 2.01}

 68%|██████▊   | 455/669 [4:32:19<2:45:17, 46.34s/it]
 68%|██████▊   | 456/669 [4:33:03<2:41:56, 45.62s/it]
                                                     
{'loss': 0.4989, 'grad_norm': 0.4994221649154916, 'learning_rate': 1.6160849772382398e-06, 'epoch': 2.02}

 68%|██████▊   | 456/669 [4:33:03<2:41:56, 45.62s/it][2024-08-02 08:13:03,575] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 08:13:03,575] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 08:13:03,578] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 08:13:03,578] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 08:13:03,578] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 08:13:03,579] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 08:13:03,579] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 08:13:03,579] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 08:13:03,705] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 08:13:03,706] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 08:13:03,709] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 08:13:03,709] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 08:13:03,710] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 08:13:03,710] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 08:13:03,711] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488
[2024-08-02 08:13:03,711] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 14695488

 68%|██████▊   | 457/669 [4:33:47<2:39:50, 45.24s/it]
                                                     
{'loss': 0.4779, 'grad_norm': 0.5578750520607872, 'learning_rate': 1.6084977238239757e-06, 'epoch': 2.0}

 68%|██████▊   | 457/669 [4:33:47<2:39:50, 45.24s/it]
 68%|██████▊   | 458/669 [4:34:31<2:37:46, 44.86s/it]
                                                     
{'loss': 0.462, 'grad_norm': 0.5277075050354076, 'learning_rate': 1.6009104704097116e-06, 'epoch': 2.01}

 68%|██████▊   | 458/669 [4:34:31<2:37:46, 44.86s/it]
 69%|██████▊   | 459/669 [4:35:15<2:36:05, 44.60s/it]
                                                     
{'loss': 0.4892, 'grad_norm': 0.5125996961203592, 'learning_rate': 1.5933232169954477e-06, 'epoch': 2.01}

 69%|██████▊   | 459/669 [4:35:15<2:36:05, 44.60s/it]
 69%|██████▉   | 460/669 [4:35:59<2:34:41, 44.41s/it]
                                                     
{'loss': 0.4913, 'grad_norm': 0.5150882727539035, 'learning_rate': 1.5857359635811836e-06, 'epoch': 2.02}

 69%|██████▉   | 460/669 [4:35:59<2:34:41, 44.41s/it]
 69%|██████▉   | 461/669 [4:36:43<2:33:28, 44.27s/it]
                                                     
{'loss': 0.4789, 'grad_norm': 0.5395569256138755, 'learning_rate': 1.5781487101669196e-06, 'epoch': 2.02}

 69%|██████▉   | 461/669 [4:36:43<2:33:28, 44.27s/it]
 69%|██████▉   | 462/669 [4:38:43<3:50:29, 66.81s/it]
                                                     
{'loss': 0.4699, 'grad_norm': 0.4907655840755226, 'learning_rate': 1.5705614567526557e-06, 'epoch': 2.03}

 69%|██████▉   | 462/669 [4:38:43<3:50:29, 66.81s/it]
 69%|██████▉   | 463/669 [4:39:27<3:25:38, 59.90s/it]
                                                     
{'loss': 0.4671, 'grad_norm': 0.508487377683692, 'learning_rate': 1.5629742033383916e-06, 'epoch': 2.03}

 69%|██████▉   | 463/669 [4:39:27<3:25:38, 59.90s/it]
 69%|██████▉   | 464/669 [4:40:10<3:08:19, 55.12s/it]
                                                     
{'loss': 0.48, 'grad_norm': 0.5211842893319257, 'learning_rate': 1.5553869499241275e-06, 'epoch': 2.04}

 69%|██████▉   | 464/669 [4:40:10<3:08:19, 55.12s/it]
 70%|██████▉   | 465/669 [4:40:54<2:55:59, 51.76s/it]
                                                     
{'loss': 0.4839, 'grad_norm': 0.5540445804429339, 'learning_rate': 1.5477996965098634e-06, 'epoch': 2.04}

 70%|██████▉   | 465/669 [4:40:54<2:55:59, 51.76s/it]
 70%|██████▉   | 466/669 [4:41:38<2:47:11, 49.42s/it]
                                                     
{'loss': 0.4991, 'grad_norm': 0.502144462775385, 'learning_rate': 1.5402124430955995e-06, 'epoch': 2.04}

 70%|██████▉   | 466/669 [4:41:38<2:47:11, 49.42s/it]
 70%|██████▉   | 467/669 [4:42:22<2:40:56, 47.81s/it]
                                                     
{'loss': 0.5268, 'grad_norm': 0.4963826920124019, 'learning_rate': 1.5326251896813355e-06, 'epoch': 2.05}

 70%|██████▉   | 467/669 [4:42:22<2:40:56, 47.81s/it]
 70%|██████▉   | 468/669 [4:43:06<2:36:15, 46.65s/it]
                                                     
{'loss': 0.4572, 'grad_norm': 0.48862357759049696, 'learning_rate': 1.5250379362670714e-06, 'epoch': 2.05}

 70%|██████▉   | 468/669 [4:43:06<2:36:15, 46.65s/it]
 70%|███████   | 469/669 [4:43:50<2:32:44, 45.82s/it]
                                                     
{'loss': 0.4761, 'grad_norm': 0.48705814461293884, 'learning_rate': 1.5174506828528073e-06, 'epoch': 2.06}

 70%|███████   | 469/669 [4:43:50<2:32:44, 45.82s/it]
 70%|███████   | 470/669 [4:44:34<2:30:08, 45.27s/it]
                                                     
{'loss': 0.468, 'grad_norm': 0.50151248823518, 'learning_rate': 1.5098634294385434e-06, 'epoch': 2.06}

 70%|███████   | 470/669 [4:44:34<2:30:08, 45.27s/it]
 70%|███████   | 471/669 [4:45:18<2:28:04, 44.87s/it]
                                                     
{'loss': 0.4496, 'grad_norm': 0.47075309247993236, 'learning_rate': 1.5022761760242793e-06, 'epoch': 2.07}

 70%|███████   | 471/669 [4:45:18<2:28:04, 44.87s/it]
 71%|███████   | 472/669 [4:46:02<2:26:22, 44.58s/it]
                                                     
{'loss': 0.4626, 'grad_norm': 0.5055609634042088, 'learning_rate': 1.4946889226100152e-06, 'epoch': 2.07}

 71%|███████   | 472/669 [4:46:02<2:26:22, 44.58s/it]
 71%|███████   | 473/669 [4:46:46<2:24:57, 44.38s/it]
                                                     
{'loss': 0.4743, 'grad_norm': 0.4931560723293831, 'learning_rate': 1.4871016691957514e-06, 'epoch': 2.08}

 71%|███████   | 473/669 [4:46:46<2:24:57, 44.38s/it]
 71%|███████   | 474/669 [4:47:30<2:23:49, 44.25s/it]
                                                     
{'loss': 0.4849, 'grad_norm': 0.4970179492434698, 'learning_rate': 1.4795144157814873e-06, 'epoch': 2.08}

 71%|███████   | 474/669 [4:47:30<2:23:49, 44.25s/it]
 71%|███████   | 475/669 [4:48:14<2:22:48, 44.17s/it]
                                                     
{'loss': 0.4647, 'grad_norm': 0.5205249715468657, 'learning_rate': 1.4719271623672232e-06, 'epoch': 2.08}

 71%|███████   | 475/669 [4:48:14<2:22:48, 44.17s/it]
 71%|███████   | 476/669 [4:48:58<2:21:49, 44.09s/it]
                                                     
{'loss': 0.4698, 'grad_norm': 0.5106891771853166, 'learning_rate': 1.4643399089529591e-06, 'epoch': 2.09}

 71%|███████   | 476/669 [4:48:58<2:21:49, 44.09s/it]
 71%|███████▏  | 477/669 [4:49:42<2:20:56, 44.04s/it]
                                                     
{'loss': 0.4435, 'grad_norm': 0.5030533511191912, 'learning_rate': 1.4567526555386952e-06, 'epoch': 2.09}

 71%|███████▏  | 477/669 [4:49:42<2:20:56, 44.04s/it]
 71%|███████▏  | 478/669 [4:50:26<2:20:05, 44.01s/it]
                                                     
{'loss': 0.4551, 'grad_norm': 0.5267045945145785, 'learning_rate': 1.4491654021244311e-06, 'epoch': 2.1}

 71%|███████▏  | 478/669 [4:50:26<2:20:05, 44.01s/it]
 72%|███████▏  | 479/669 [4:51:10<2:19:17, 43.99s/it]
                                                     
{'loss': 0.4893, 'grad_norm': 0.4910735265010368, 'learning_rate': 1.441578148710167e-06, 'epoch': 2.1}

 72%|███████▏  | 479/669 [4:51:10<2:19:17, 43.99s/it]
 72%|███████▏  | 480/669 [4:51:53<2:18:28, 43.96s/it]
                                                     
{'loss': 0.4631, 'grad_norm': 0.5098674286346987, 'learning_rate': 1.433990895295903e-06, 'epoch': 2.11}

 72%|███████▏  | 480/669 [4:51:53<2:18:28, 43.96s/it]
 72%|███████▏  | 481/669 [4:52:38<2:17:50, 43.99s/it]
                                                     
{'loss': 0.4478, 'grad_norm': 0.49833407519162715, 'learning_rate': 1.426403641881639e-06, 'epoch': 2.11}

 72%|███████▏  | 481/669 [4:52:38<2:17:50, 43.99s/it]
 72%|███████▏  | 482/669 [4:53:21<2:17:02, 43.97s/it]
                                                     
{'loss': 0.4661, 'grad_norm': 0.4968638787102979, 'learning_rate': 1.418816388467375e-06, 'epoch': 2.12}

 72%|███████▏  | 482/669 [4:53:21<2:17:02, 43.97s/it]
 72%|███████▏  | 483/669 [4:54:05<2:16:15, 43.95s/it]
                                                     
{'loss': 0.487, 'grad_norm': 0.5287581344581138, 'learning_rate': 1.411229135053111e-06, 'epoch': 2.12}

 72%|███████▏  | 483/669 [4:54:05<2:16:15, 43.95s/it]
 72%|███████▏  | 484/669 [4:54:49<2:15:29, 43.94s/it]
                                                     
{'loss': 0.4641, 'grad_norm': 0.5169520017940434, 'learning_rate': 1.403641881638847e-06, 'epoch': 2.13}

 72%|███████▏  | 484/669 [4:54:49<2:15:29, 43.94s/it]
 72%|███████▏  | 485/669 [4:55:33<2:14:43, 43.93s/it]
                                                     
{'loss': 0.4657, 'grad_norm': 0.4889710176907046, 'learning_rate': 1.396054628224583e-06, 'epoch': 2.13}

 72%|███████▏  | 485/669 [4:55:33<2:14:43, 43.93s/it]
 73%|███████▎  | 486/669 [4:56:17<2:14:00, 43.94s/it]
                                                     
{'loss': 0.4655, 'grad_norm': 0.5068054201127302, 'learning_rate': 1.3884673748103189e-06, 'epoch': 2.13}

 73%|███████▎  | 486/669 [4:56:17<2:14:00, 43.94s/it]
 73%|███████▎  | 487/669 [4:58:17<3:22:02, 66.60s/it]
                                                     
{'loss': 0.4489, 'grad_norm': 0.5227391076748372, 'learning_rate': 1.3808801213960548e-06, 'epoch': 2.14}

 73%|███████▎  | 487/669 [4:58:17<3:22:02, 66.60s/it]
 73%|███████▎  | 488/669 [4:59:00<3:00:14, 59.75s/it]
                                                     
{'loss': 0.4518, 'grad_norm': 0.4685531353247096, 'learning_rate': 1.3732928679817905e-06, 'epoch': 2.14}

 73%|███████▎  | 488/669 [4:59:00<3:00:14, 59.75s/it]
 73%|███████▎  | 489/669 [4:59:44<2:44:59, 55.00s/it]
                                                     
{'loss': 0.4539, 'grad_norm': 0.48729624697805496, 'learning_rate': 1.3657056145675266e-06, 'epoch': 2.15}

 73%|███████▎  | 489/669 [4:59:44<2:44:59, 55.00s/it]
 73%|███████▎  | 490/669 [5:00:28<2:34:09, 51.67s/it]
                                                     
{'loss': 0.4698, 'grad_norm': 0.49686659002832273, 'learning_rate': 1.3581183611532625e-06, 'epoch': 2.15}

 73%|███████▎  | 490/669 [5:00:28<2:34:09, 51.67s/it]
 73%|███████▎  | 491/669 [5:01:12<2:26:31, 49.39s/it]
                                                     
{'loss': 0.5282, 'grad_norm': 0.5165370496352449, 'learning_rate': 1.3505311077389985e-06, 'epoch': 2.16}

 73%|███████▎  | 491/669 [5:01:12<2:26:31, 49.39s/it]
 74%|███████▎  | 492/669 [5:01:56<2:20:52, 47.75s/it]
                                                     
{'loss': 0.5011, 'grad_norm': 0.49390583090447004, 'learning_rate': 1.3429438543247344e-06, 'epoch': 2.16}

 74%|███████▎  | 492/669 [5:01:56<2:20:52, 47.75s/it]
 74%|███████▎  | 493/669 [5:02:40<2:16:42, 46.61s/it]
                                                     
{'loss': 0.4888, 'grad_norm': 0.4966083299512843, 'learning_rate': 1.3353566009104705e-06, 'epoch': 2.17}

 74%|███████▎  | 493/669 [5:02:40<2:16:42, 46.61s/it]
 74%|███████▍  | 494/669 [5:03:24<2:13:34, 45.79s/it]
                                                     
{'loss': 0.4629, 'grad_norm': 0.47651097727052893, 'learning_rate': 1.3277693474962064e-06, 'epoch': 2.17}

 74%|███████▍  | 494/669 [5:03:24<2:13:34, 45.79s/it]
 74%|███████▍  | 495/669 [5:04:08<2:11:11, 45.24s/it]
                                                     
{'loss': 0.4538, 'grad_norm': 0.5171229401967155, 'learning_rate': 1.3201820940819423e-06, 'epoch': 2.17}

 74%|███████▍  | 495/669 [5:04:08<2:11:11, 45.24s/it]
 74%|███████▍  | 496/669 [5:04:52<2:09:17, 44.84s/it]
                                                     
{'loss': 0.4822, 'grad_norm': 0.5177579051508318, 'learning_rate': 1.3125948406676784e-06, 'epoch': 2.18}

 74%|███████▍  | 496/669 [5:04:52<2:09:17, 44.84s/it]
 74%|███████▍  | 497/669 [5:05:36<2:07:50, 44.60s/it]
                                                     
{'loss': 0.4879, 'grad_norm': 0.5363730349952008, 'learning_rate': 1.3050075872534144e-06, 'epoch': 2.18}

 74%|███████▍  | 497/669 [5:05:36<2:07:50, 44.60s/it]
 74%|███████▍  | 498/669 [5:06:20<2:06:34, 44.41s/it]
                                                     
{'loss': 0.4732, 'grad_norm': 0.49547274684149156, 'learning_rate': 1.2974203338391503e-06, 'epoch': 2.19}

 74%|███████▍  | 498/669 [5:06:20<2:06:34, 44.41s/it]
 75%|███████▍  | 499/669 [5:07:04<2:05:24, 44.26s/it]
                                                     
{'loss': 0.4918, 'grad_norm': 0.48560343902497966, 'learning_rate': 1.2898330804248862e-06, 'epoch': 2.19}

 75%|███████▍  | 499/669 [5:07:04<2:05:24, 44.26s/it]
 75%|███████▍  | 500/669 [5:07:48<2:04:24, 44.17s/it]
                                                     
{'loss': 0.4618, 'grad_norm': 0.49966297308311575, 'learning_rate': 1.2822458270106223e-06, 'epoch': 2.2}

 75%|███████▍  | 500/669 [5:07:48<2:04:24, 44.17s/it]
 75%|███████▍  | 501/669 [5:08:32<2:03:28, 44.10s/it]
                                                     
{'loss': 0.4953, 'grad_norm': 0.472067295226221, 'learning_rate': 1.2746585735963582e-06, 'epoch': 2.2}

 75%|███████▍  | 501/669 [5:08:32<2:03:28, 44.10s/it]
 75%|███████▌  | 502/669 [5:09:16<2:02:37, 44.05s/it]
                                                     
{'loss': 0.4797, 'grad_norm': 0.5241464474412257, 'learning_rate': 1.2670713201820941e-06, 'epoch': 2.21}

 75%|███████▌  | 502/669 [5:09:16<2:02:37, 44.05s/it]
 75%|███████▌  | 503/669 [5:10:00<2:01:46, 44.01s/it]
                                                     
{'loss': 0.4493, 'grad_norm': 0.5063960714766558, 'learning_rate': 1.25948406676783e-06, 'epoch': 2.21}

 75%|███████▌  | 503/669 [5:10:00<2:01:46, 44.01s/it]
 75%|███████▌  | 504/669 [5:10:44<2:01:00, 44.00s/it]
                                                     
{'loss': 0.4736, 'grad_norm': 0.48407847297667117, 'learning_rate': 1.2518968133535662e-06, 'epoch': 2.21}

 75%|███████▌  | 504/669 [5:10:44<2:01:00, 44.00s/it]/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(

 75%|███████▌  | 505/669 [5:12:01<2:27:34, 53.99s/it]
                                                     
{'loss': 0.4476, 'grad_norm': 0.4859389089228068, 'learning_rate': 1.244309559939302e-06, 'epoch': 2.22}

 75%|███████▌  | 505/669 [5:12:01<2:27:34, 53.99s/it]
 76%|███████▌  | 506/669 [5:12:45<2:18:31, 50.99s/it]
                                                     
{'loss': 0.4447, 'grad_norm': 0.49468343574845597, 'learning_rate': 1.236722306525038e-06, 'epoch': 2.22}

 76%|███████▌  | 506/669 [5:12:45<2:18:31, 50.99s/it]
 76%|███████▌  | 507/669 [5:13:29<2:11:58, 48.88s/it]
                                                     
{'loss': 0.4599, 'grad_norm': 0.45980997966401754, 'learning_rate': 1.229135053110774e-06, 'epoch': 2.23}

 76%|███████▌  | 507/669 [5:13:29<2:11:58, 48.88s/it]
 76%|███████▌  | 508/669 [5:14:13<2:07:19, 47.45s/it]
                                                     
{'loss': 0.4852, 'grad_norm': 0.5135776498149942, 'learning_rate': 1.22154779969651e-06, 'epoch': 2.23}

 76%|███████▌  | 508/669 [5:14:13<2:07:19, 47.45s/it]
 76%|███████▌  | 509/669 [5:14:57<2:03:44, 46.40s/it]
                                                     
{'loss': 0.5157, 'grad_norm': 0.48749385873656575, 'learning_rate': 1.213960546282246e-06, 'epoch': 2.24}

 76%|███████▌  | 509/669 [5:14:57<2:03:44, 46.40s/it]
 76%|███████▌  | 510/669 [5:15:41<2:01:07, 45.71s/it]
                                                     
{'loss': 0.4437, 'grad_norm': 0.48899276420726745, 'learning_rate': 1.2063732928679819e-06, 'epoch': 2.24}

 76%|███████▌  | 510/669 [5:15:41<2:01:07, 45.71s/it]
 76%|███████▋  | 511/669 [5:16:25<1:59:06, 45.23s/it]
                                                     
{'loss': 0.4974, 'grad_norm': 0.4918342780724834, 'learning_rate': 1.198786039453718e-06, 'epoch': 2.25}

 76%|███████▋  | 511/669 [5:16:25<1:59:06, 45.23s/it]
 77%|███████▋  | 512/669 [5:18:23<2:55:09, 66.94s/it]
                                                     
{'loss': 0.4509, 'grad_norm': 0.4774969997416472, 'learning_rate': 1.191198786039454e-06, 'epoch': 2.25}

 77%|███████▋  | 512/669 [5:18:23<2:55:09, 66.94s/it]
 77%|███████▋  | 513/669 [5:19:06<2:35:58, 59.99s/it]
                                                     
{'loss': 0.4599, 'grad_norm': 0.4791749757272505, 'learning_rate': 1.1836115326251896e-06, 'epoch': 2.25}

 77%|███████▋  | 513/669 [5:19:06<2:35:58, 59.99s/it]
 77%|███████▋  | 514/669 [5:19:50<2:22:31, 55.17s/it]
                                                     
{'loss': 0.4643, 'grad_norm': 0.5072314853369309, 'learning_rate': 1.1760242792109257e-06, 'epoch': 2.26}

 77%|███████▋  | 514/669 [5:19:50<2:22:31, 55.17s/it]
 77%|███████▋  | 515/669 [5:20:34<2:12:57, 51.80s/it]
                                                     
{'loss': 0.4688, 'grad_norm': 0.5004472392134528, 'learning_rate': 1.1684370257966617e-06, 'epoch': 2.26}

 77%|███████▋  | 515/669 [5:20:34<2:12:57, 51.80s/it]
 77%|███████▋  | 516/669 [5:21:18<2:06:13, 49.50s/it]
                                                     
{'loss': 0.5069, 'grad_norm': 0.48150392498478517, 'learning_rate': 1.1608497723823976e-06, 'epoch': 2.27}

 77%|███████▋  | 516/669 [5:21:18<2:06:13, 49.50s/it]
 77%|███████▋  | 517/669 [5:22:02<2:01:08, 47.82s/it]
                                                     
{'loss': 0.4571, 'grad_norm': 0.4828855445414688, 'learning_rate': 1.1532625189681337e-06, 'epoch': 2.27}

 77%|███████▋  | 517/669 [5:22:02<2:01:08, 47.82s/it]
 77%|███████▋  | 518/669 [5:22:46<1:57:24, 46.65s/it]
                                                     
{'loss': 0.4651, 'grad_norm': 0.49789128855450915, 'learning_rate': 1.1456752655538696e-06, 'epoch': 2.28}

 77%|███████▋  | 518/669 [5:22:46<1:57:24, 46.65s/it]
 78%|███████▊  | 519/669 [5:23:30<1:54:35, 45.84s/it]
                                                     
{'loss': 0.4757, 'grad_norm': 0.49890243849633514, 'learning_rate': 1.1380880121396055e-06, 'epoch': 2.28}

 78%|███████▊  | 519/669 [5:23:30<1:54:35, 45.84s/it]
 78%|███████▊  | 520/669 [5:24:14<1:52:25, 45.27s/it]
                                                     
{'loss': 0.4668, 'grad_norm': 0.48825603773254755, 'learning_rate': 1.1305007587253414e-06, 'epoch': 2.29}

 78%|███████▊  | 520/669 [5:24:14<1:52:25, 45.27s/it]
 78%|███████▊  | 521/669 [5:24:58<1:50:39, 44.86s/it]
                                                     
{'loss': 0.4576, 'grad_norm': 0.48797823917514366, 'learning_rate': 1.1229135053110776e-06, 'epoch': 2.29}

 78%|███████▊  | 521/669 [5:24:58<1:50:39, 44.86s/it]
 78%|███████▊  | 522/669 [5:25:42<1:49:12, 44.57s/it]
                                                     
{'loss': 0.4998, 'grad_norm': 0.4713265522467315, 'learning_rate': 1.1153262518968135e-06, 'epoch': 2.29}

 78%|███████▊  | 522/669 [5:25:42<1:49:12, 44.57s/it]
 78%|███████▊  | 523/669 [5:26:26<1:48:00, 44.38s/it]
                                                     
{'loss': 0.4583, 'grad_norm': 0.48731093463299924, 'learning_rate': 1.1077389984825494e-06, 'epoch': 2.3}

 78%|███████▊  | 523/669 [5:26:26<1:48:00, 44.38s/it]
 78%|███████▊  | 524/669 [5:27:10<1:46:59, 44.27s/it]
                                                     
{'loss': 0.4964, 'grad_norm': 0.5356992050391173, 'learning_rate': 1.1001517450682853e-06, 'epoch': 2.3}

 78%|███████▊  | 524/669 [5:27:10<1:46:59, 44.27s/it]
 78%|███████▊  | 525/669 [5:27:54<1:45:59, 44.16s/it]
                                                     
{'loss': 0.4456, 'grad_norm': 0.5185855845215128, 'learning_rate': 1.0925644916540214e-06, 'epoch': 2.31}

 78%|███████▊  | 525/669 [5:27:54<1:45:59, 44.16s/it]
 79%|███████▊  | 526/669 [5:28:38<1:45:02, 44.07s/it]
                                                     
{'loss': 0.4667, 'grad_norm': 0.5340341729511069, 'learning_rate': 1.0849772382397573e-06, 'epoch': 2.31}

 79%|███████▊  | 526/669 [5:28:38<1:45:02, 44.07s/it]
 79%|███████▉  | 527/669 [5:29:22<1:44:11, 44.02s/it]
                                                     
{'loss': 0.4907, 'grad_norm': 0.46791114784776194, 'learning_rate': 1.0773899848254933e-06, 'epoch': 2.32}

 79%|███████▉  | 527/669 [5:29:22<1:44:11, 44.02s/it]
 79%|███████▉  | 528/669 [5:30:05<1:43:22, 43.99s/it]
                                                     
{'loss': 0.4411, 'grad_norm': 0.5224682012273069, 'learning_rate': 1.0698027314112294e-06, 'epoch': 2.32}

 79%|███████▉  | 528/669 [5:30:05<1:43:22, 43.99s/it]
 79%|███████▉  | 529/669 [5:30:50<1:42:40, 44.00s/it]
                                                     
{'loss': 0.4872, 'grad_norm': 0.5013175127206561, 'learning_rate': 1.062215477996965e-06, 'epoch': 2.33}

 79%|███████▉  | 529/669 [5:30:50<1:42:40, 44.00s/it]
 79%|███████▉  | 530/669 [5:31:33<1:41:53, 43.98s/it]
                                                     
{'loss': 0.4547, 'grad_norm': 0.49567401298066655, 'learning_rate': 1.054628224582701e-06, 'epoch': 2.33}

 79%|███████▉  | 530/669 [5:31:33<1:41:53, 43.98s/it]
 79%|███████▉  | 531/669 [5:32:17<1:41:07, 43.97s/it]
                                                     
{'loss': 0.4774, 'grad_norm': 0.5092075971225333, 'learning_rate': 1.0470409711684371e-06, 'epoch': 2.34}

 79%|███████▉  | 531/669 [5:32:17<1:41:07, 43.97s/it]
 80%|███████▉  | 532/669 [5:33:01<1:40:21, 43.95s/it]
                                                     
{'loss': 0.4593, 'grad_norm': 0.5206537811339336, 'learning_rate': 1.039453717754173e-06, 'epoch': 2.34}

 80%|███████▉  | 532/669 [5:33:01<1:40:21, 43.95s/it]
 80%|███████▉  | 533/669 [5:33:45<1:39:36, 43.94s/it]
                                                     
{'loss': 0.4947, 'grad_norm': 0.5247188237207532, 'learning_rate': 1.031866464339909e-06, 'epoch': 2.34}

 80%|███████▉  | 533/669 [5:33:45<1:39:36, 43.94s/it]
 80%|███████▉  | 534/669 [5:34:29<1:38:52, 43.94s/it]
                                                     
{'loss': 0.4675, 'grad_norm': 0.48511496092549006, 'learning_rate': 1.024279210925645e-06, 'epoch': 2.35}

 80%|███████▉  | 534/669 [5:34:29<1:38:52, 43.94s/it]
 80%|███████▉  | 535/669 [5:35:13<1:38:11, 43.96s/it]
                                                     
{'loss': 0.4654, 'grad_norm': 0.49031229609343957, 'learning_rate': 1.016691957511381e-06, 'epoch': 2.35}

 80%|███████▉  | 535/669 [5:35:13<1:38:11, 43.96s/it]
 80%|████████  | 536/669 [5:35:57<1:37:25, 43.95s/it]
                                                     
{'loss': 0.4881, 'grad_norm': 0.5022437562865831, 'learning_rate': 1.009104704097117e-06, 'epoch': 2.36}

 80%|████████  | 536/669 [5:35:57<1:37:25, 43.95s/it]
 80%|████████  | 537/669 [5:37:55<2:25:26, 66.11s/it]
                                                     
{'loss': 0.4722, 'grad_norm': 0.48427536854604036, 'learning_rate': 1.0015174506828528e-06, 'epoch': 2.36}

 80%|████████  | 537/669 [5:37:55<2:25:26, 66.11s/it]
 80%|████████  | 538/669 [5:38:39<2:09:41, 59.40s/it]
                                                     
{'loss': 0.4816, 'grad_norm': 0.4951077677256604, 'learning_rate': 9.93930197268589e-07, 'epoch': 2.37}

 80%|████████  | 538/669 [5:38:39<2:09:41, 59.40s/it]
 81%|████████  | 539/669 [5:39:23<1:58:40, 54.77s/it]
                                                     
{'loss': 0.448, 'grad_norm': 0.506971274674449, 'learning_rate': 9.863429438543249e-07, 'epoch': 2.37}

 81%|████████  | 539/669 [5:39:23<1:58:40, 54.77s/it]
 81%|████████  | 540/669 [5:40:07<1:50:49, 51.55s/it]
                                                     
{'loss': 0.4644, 'grad_norm': 0.4843767211745505, 'learning_rate': 9.787556904400608e-07, 'epoch': 2.38}

 81%|████████  | 540/669 [5:40:07<1:50:49, 51.55s/it]
 81%|████████  | 541/669 [5:40:51<1:45:10, 49.30s/it]
                                                     
{'loss': 0.4681, 'grad_norm': 0.4787312911085378, 'learning_rate': 9.711684370257967e-07, 'epoch': 2.38}

 81%|████████  | 541/669 [5:40:51<1:45:10, 49.30s/it]
 81%|████████  | 542/669 [5:41:35<1:40:54, 47.68s/it]
                                                     
{'loss': 0.4595, 'grad_norm': 0.5040950848521704, 'learning_rate': 9.635811836115328e-07, 'epoch': 2.38}

 81%|████████  | 542/669 [5:41:35<1:40:54, 47.68s/it]
 81%|████████  | 543/669 [5:42:19<1:37:45, 46.55s/it]
                                                     
{'loss': 0.4542, 'grad_norm': 0.4742009596031962, 'learning_rate': 9.559939301972687e-07, 'epoch': 2.39}

 81%|████████  | 543/669 [5:42:19<1:37:45, 46.55s/it]
 81%|████████▏ | 544/669 [5:43:02<1:35:19, 45.76s/it]
                                                     
{'loss': 0.4818, 'grad_norm': 0.4819914757091554, 'learning_rate': 9.484066767830046e-07, 'epoch': 2.39}

 81%|████████▏ | 544/669 [5:43:02<1:35:19, 45.76s/it]
 81%|████████▏ | 545/669 [5:43:46<1:33:26, 45.22s/it]
                                                     
{'loss': 0.4459, 'grad_norm': 0.4794763925814031, 'learning_rate': 9.408194233687407e-07, 'epoch': 2.4}

 81%|████████▏ | 545/669 [5:43:46<1:33:26, 45.22s/it]
 82%|████████▏ | 546/669 [5:44:30<1:31:53, 44.83s/it]
                                                     
{'loss': 0.4986, 'grad_norm': 0.5065706041345039, 'learning_rate': 9.332321699544765e-07, 'epoch': 2.4}

 82%|████████▏ | 546/669 [5:44:30<1:31:53, 44.83s/it]
 82%|████████▏ | 547/669 [5:45:14<1:30:36, 44.56s/it]
                                                     
{'loss': 0.4618, 'grad_norm': 0.4918492784353524, 'learning_rate': 9.256449165402125e-07, 'epoch': 2.41}

 82%|████████▏ | 547/669 [5:45:14<1:30:36, 44.56s/it]
 82%|████████▏ | 548/669 [5:45:58<1:29:27, 44.36s/it]
                                                     
{'loss': 0.4358, 'grad_norm': 0.48410824249150014, 'learning_rate': 9.180576631259484e-07, 'epoch': 2.41}

 82%|████████▏ | 548/669 [5:45:58<1:29:27, 44.36s/it]
 82%|████████▏ | 549/669 [5:46:42<1:28:31, 44.26s/it]
                                                     
{'loss': 0.4522, 'grad_norm': 0.5071386174484, 'learning_rate': 9.104704097116844e-07, 'epoch': 2.42}

 82%|████████▏ | 549/669 [5:46:42<1:28:31, 44.26s/it]
 82%|████████▏ | 550/669 [5:47:26<1:27:35, 44.16s/it]
                                                     
{'loss': 0.4718, 'grad_norm': 0.5179035085963222, 'learning_rate': 9.028831562974203e-07, 'epoch': 2.42}

 82%|████████▏ | 550/669 [5:47:26<1:27:35, 44.16s/it]
 82%|████████▏ | 551/669 [5:48:10<1:26:44, 44.10s/it]
                                                     
{'loss': 0.4946, 'grad_norm': 0.48695002255677194, 'learning_rate': 8.952959028831563e-07, 'epoch': 2.42}

 82%|████████▏ | 551/669 [5:48:10<1:26:44, 44.10s/it]
 83%|████████▎ | 552/669 [5:48:54<1:25:56, 44.08s/it]
                                                     
{'loss': 0.482, 'grad_norm': 0.4697467884861594, 'learning_rate': 8.877086494688924e-07, 'epoch': 2.43}

 83%|████████▎ | 552/669 [5:48:54<1:25:56, 44.08s/it]
 83%|████████▎ | 553/669 [5:49:38<1:25:08, 44.04s/it]
                                                     
{'loss': 0.4528, 'grad_norm': 0.5039869709064255, 'learning_rate': 8.801213960546283e-07, 'epoch': 2.43}

 83%|████████▎ | 553/669 [5:49:38<1:25:08, 44.04s/it]
 83%|████████▎ | 554/669 [5:50:22<1:24:21, 44.02s/it]
                                                     
{'loss': 0.4416, 'grad_norm': 0.4956541767737936, 'learning_rate': 8.725341426403643e-07, 'epoch': 2.44}

 83%|████████▎ | 554/669 [5:50:22<1:24:21, 44.02s/it]
 83%|████████▎ | 555/669 [5:51:06<1:23:33, 43.98s/it]
                                                     
{'loss': 0.4534, 'grad_norm': 0.5220748437476452, 'learning_rate': 8.649468892261002e-07, 'epoch': 2.44}

 83%|████████▎ | 555/669 [5:51:06<1:23:33, 43.98s/it]
 83%|████████▎ | 556/669 [5:51:50<1:22:45, 43.94s/it]
                                                     
{'loss': 0.4523, 'grad_norm': 0.5004114462806725, 'learning_rate': 8.573596358118362e-07, 'epoch': 2.45}

 83%|████████▎ | 556/669 [5:51:50<1:22:45, 43.94s/it]
 83%|████████▎ | 557/669 [5:52:34<1:22:00, 43.93s/it]
                                                     
{'loss': 0.4483, 'grad_norm': 0.5002883358165275, 'learning_rate': 8.497723823975721e-07, 'epoch': 2.45}

 83%|████████▎ | 557/669 [5:52:34<1:22:00, 43.93s/it]
 83%|████████▎ | 558/669 [5:53:18<1:21:16, 43.93s/it]
                                                     
{'loss': 0.4547, 'grad_norm': 0.5001315929982186, 'learning_rate': 8.421851289833082e-07, 'epoch': 2.46}

 83%|████████▎ | 558/669 [5:53:18<1:21:16, 43.93s/it]
 84%|████████▎ | 559/669 [5:54:02<1:20:33, 43.95s/it]
                                                     
{'loss': 0.5064, 'grad_norm': 0.5141518132214271, 'learning_rate': 8.345978755690441e-07, 'epoch': 2.46}

 84%|████████▎ | 559/669 [5:54:02<1:20:33, 43.95s/it]
 84%|████████▎ | 560/669 [5:54:46<1:19:49, 43.94s/it]
                                                     
{'loss': 0.4473, 'grad_norm': 0.4960566337702956, 'learning_rate': 8.270106221547801e-07, 'epoch': 2.46}

 84%|████████▎ | 560/669 [5:54:46<1:19:49, 43.94s/it][2024-08-02 09:34:45,853] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:45,853] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:45,853] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:45,853] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:45,853] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:45,853] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:45,853] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:45,853] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:45,854] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218803] [RANK:4] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:45,854] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218802] [RANK:3] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:45,854] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218800] [RANK:1] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:45,854] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218805] [RANK:6] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:45,854] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218806] [RANK:7] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:45,854] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218804] [RANK:5] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:45,854] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218801] [RANK:2] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:45,854] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:49,120] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:49,122] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:49,123] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:49,123] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


  0%|          | 0/8 [00:00<?, ?it/s][2024-08-02 09:34:52,664] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:52,665] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 25%|██▌       | 2/8 [00:03<00:10,  1.77s/it][2024-08-02 09:34:56,205] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:56,206] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 38%|███▊      | 3/8 [00:07<00:12,  2.51s/it][2024-08-02 09:34:59,744] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:34:59,745] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 50%|█████     | 4/8 [00:10<00:11,  2.89s/it][2024-08-02 09:35:03,463] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:35:03,464] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 62%|██████▎   | 5/8 [00:14<00:09,  3.18s/it][2024-08-02 09:35:07,002] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:35:07,003] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 75%|███████▌  | 6/8 [00:17<00:06,  3.30s/it][2024-08-02 09:35:10,544] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:35:10,545] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


 88%|████████▊ | 7/8 [00:21<00:03,  3.38s/it][2024-08-02 09:35:14,095] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:35:14,096] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


100%|██████████| 8/8 [00:24<00:00,  3.43s/it][2024-08-02 09:35:17,737] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160
[2024-08-02 09:35:17,738] [INFO] [axolotl.utils.samplers.multipack._len_est:185] [PID:218799] [RANK:0] packing_efficiency_estimate: 0.99 total_num_tokens per device: 149160


9it [00:28,  3.50s/it]                       
                                                     

                      
{'eval_loss': 0.4975128769874573, 'eval_runtime': 32.1892, 'eval_samples_per_second': 82.947, 'eval_steps_per_second': 5.188, 'epoch': 2.46}

 84%|████████▎ | 560/669 [5:55:18<1:19:49, 43.94s/it]

9it [00:28,  3.50s/it]

                      /root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(

 84%|████████▍ | 561/669 [5:56:35<1:54:36, 63.67s/it]
                                                     
{'loss': 0.4559, 'grad_norm': 0.4897198333575608, 'learning_rate': 8.19423368740516e-07, 'epoch': 2.47}

 84%|████████▍ | 561/669 [5:56:35<1:54:36, 63.67s/it]
 84%|████████▍ | 562/669 [5:58:31<2:21:27, 79.33s/it]
                                                     
{'loss': 0.4651, 'grad_norm': 0.49985877359817477, 'learning_rate': 8.118361153262519e-07, 'epoch': 2.47}

 84%|████████▍ | 562/669 [5:58:31<2:21:27, 79.33s/it]
 84%|████████▍ | 563/669 [5:59:15<2:01:27, 68.75s/it]
                                                     
{'loss': 0.4803, 'grad_norm': 0.49485306049230043, 'learning_rate': 8.042488619119878e-07, 'epoch': 2.48}

 84%|████████▍ | 563/669 [5:59:15<2:01:27, 68.75s/it]
 84%|████████▍ | 564/669 [5:59:59<1:47:15, 61.29s/it]
                                                     
{'loss': 0.4553, 'grad_norm': 0.5208900276481403, 'learning_rate': 7.966616084977239e-07, 'epoch': 2.48}

 84%|████████▍ | 564/669 [5:59:59<1:47:15, 61.29s/it]
 84%|████████▍ | 565/669 [6:00:43<1:37:13, 56.09s/it]
                                                     
{'loss': 0.4681, 'grad_norm': 0.48837650054879367, 'learning_rate': 7.890743550834598e-07, 'epoch': 2.49}

 84%|████████▍ | 565/669 [6:00:43<1:37:13, 56.09s/it]
 85%|████████▍ | 566/669 [6:01:27<1:30:01, 52.44s/it]
                                                     
{'loss': 0.4564, 'grad_norm': 0.47305516115923485, 'learning_rate': 7.814871016691958e-07, 'epoch': 2.49}

 85%|████████▍ | 566/669 [6:01:27<1:30:01, 52.44s/it]
 85%|████████▍ | 567/669 [6:02:11<1:24:54, 49.95s/it]
                                                     
{'loss': 0.4632, 'grad_norm': 0.5049602967209187, 'learning_rate': 7.738998482549317e-07, 'epoch': 2.5}

 85%|████████▍ | 567/669 [6:02:11<1:24:54, 49.95s/it]
 85%|████████▍ | 568/669 [6:02:55<1:21:02, 48.14s/it]
                                                     
{'loss': 0.448, 'grad_norm': 0.48547661489581506, 'learning_rate': 7.663125948406677e-07, 'epoch': 2.5}

 85%|████████▍ | 568/669 [6:02:55<1:21:02, 48.14s/it]
 85%|████████▌ | 569/669 [6:03:39<1:18:06, 46.86s/it]
                                                     
{'loss': 0.4432, 'grad_norm': 0.5747686136950829, 'learning_rate': 7.587253414264036e-07, 'epoch': 2.51}

 85%|████████▌ | 569/669 [6:03:39<1:18:06, 46.86s/it]
 85%|████████▌ | 570/669 [6:04:23<1:15:50, 45.97s/it]
                                                     
{'loss': 0.432, 'grad_norm': 0.5499322063415041, 'learning_rate': 7.511380880121397e-07, 'epoch': 2.51}

 85%|████████▌ | 570/669 [6:04:23<1:15:50, 45.97s/it]
 85%|████████▌ | 571/669 [6:05:07<1:14:04, 45.35s/it]
                                                     
{'loss': 0.4325, 'grad_norm': 0.49909881450641125, 'learning_rate': 7.435508345978757e-07, 'epoch': 2.51}

 85%|████████▌ | 571/669 [6:05:07<1:14:04, 45.35s/it]
 86%|████████▌ | 572/669 [6:05:51<1:12:37, 44.93s/it]
                                                     
{'loss': 0.4374, 'grad_norm': 0.5137001617086465, 'learning_rate': 7.359635811836116e-07, 'epoch': 2.52}

 86%|████████▌ | 572/669 [6:05:51<1:12:37, 44.93s/it]
 86%|████████▌ | 573/669 [6:06:34<1:11:23, 44.62s/it]
                                                     
{'loss': 0.4751, 'grad_norm': 0.5043757440405017, 'learning_rate': 7.283763277693476e-07, 'epoch': 2.52}

 86%|████████▌ | 573/669 [6:06:34<1:11:23, 44.62s/it]
 86%|████████▌ | 574/669 [6:07:18<1:10:20, 44.42s/it]
                                                     
{'loss': 0.4368, 'grad_norm': 0.4882894387336232, 'learning_rate': 7.207890743550835e-07, 'epoch': 2.53}

 86%|████████▌ | 574/669 [6:07:18<1:10:20, 44.42s/it]
 86%|████████▌ | 575/669 [6:08:02<1:09:21, 44.27s/it]
                                                     
{'loss': 0.4174, 'grad_norm': 0.5302357654250439, 'learning_rate': 7.132018209408196e-07, 'epoch': 2.53}

 86%|████████▌ | 575/669 [6:08:02<1:09:21, 44.27s/it]
 86%|████████▌ | 576/669 [6:08:46<1:08:27, 44.17s/it]
                                                     
{'loss': 0.4353, 'grad_norm': 0.5229623842607946, 'learning_rate': 7.056145675265555e-07, 'epoch': 2.54}

 86%|████████▌ | 576/669 [6:08:46<1:08:27, 44.17s/it]
 86%|████████▌ | 577/669 [6:09:30<1:07:35, 44.08s/it]
                                                     
{'loss': 0.4278, 'grad_norm': 0.5145290981136341, 'learning_rate': 6.980273141122915e-07, 'epoch': 2.54}

 86%|████████▌ | 577/669 [6:09:30<1:07:35, 44.08s/it]
 86%|████████▋ | 578/669 [6:10:14<1:06:50, 44.07s/it]
                                                     
{'loss': 0.434, 'grad_norm': 0.49830066459201744, 'learning_rate': 6.904400606980274e-07, 'epoch': 2.55}

 86%|████████▋ | 578/669 [6:10:14<1:06:50, 44.07s/it]
 87%|████████▋ | 579/669 [6:10:58<1:06:00, 44.01s/it]
                                                     
{'loss': 0.4375, 'grad_norm': 0.5220459047427843, 'learning_rate': 6.828528072837633e-07, 'epoch': 2.55}

 87%|████████▋ | 579/669 [6:10:58<1:06:00, 44.01s/it]
 87%|████████▋ | 580/669 [6:11:42<1:05:14, 43.99s/it]
                                                     
{'loss': 0.4295, 'grad_norm': 0.5111391623585094, 'learning_rate': 6.752655538694992e-07, 'epoch': 2.55}

 87%|████████▋ | 580/669 [6:11:42<1:05:14, 43.99s/it]
 87%|████████▋ | 581/669 [6:12:26<1:04:28, 43.95s/it]
                                                     
{'loss': 0.4205, 'grad_norm': 0.4944043554942434, 'learning_rate': 6.676783004552352e-07, 'epoch': 2.56}

 87%|████████▋ | 581/669 [6:12:26<1:04:28, 43.95s/it]
 87%|████████▋ | 582/669 [6:13:10<1:03:42, 43.93s/it]
                                                     
{'loss': 0.4502, 'grad_norm': 0.49154601956517674, 'learning_rate': 6.600910470409712e-07, 'epoch': 2.56}

 87%|████████▋ | 582/669 [6:13:10<1:03:42, 43.93s/it]
 87%|████████▋ | 583/669 [6:13:54<1:02:57, 43.93s/it]
                                                     
{'loss': 0.4453, 'grad_norm': 0.5008348019898189, 'learning_rate': 6.525037936267072e-07, 'epoch': 2.57}

 87%|████████▋ | 583/669 [6:13:54<1:02:57, 43.93s/it]
 87%|████████▋ | 584/669 [6:14:38<1:02:11, 43.90s/it]
                                                     
{'loss': 0.4353, 'grad_norm': 0.5005572885290496, 'learning_rate': 6.449165402124431e-07, 'epoch': 2.57}

 87%|████████▋ | 584/669 [6:14:38<1:02:11, 43.90s/it]
 87%|████████▋ | 585/669 [6:15:21<1:01:28, 43.91s/it]
                                                     
{'loss': 0.4269, 'grad_norm': 0.5302967367534124, 'learning_rate': 6.373292867981791e-07, 'epoch': 2.58}

 87%|████████▋ | 585/669 [6:15:21<1:01:28, 43.91s/it]
 88%|████████▊ | 586/669 [6:16:05<1:00:44, 43.91s/it]
                                                     
{'loss': 0.4372, 'grad_norm': 0.5050943269077487, 'learning_rate': 6.29742033383915e-07, 'epoch': 2.58}

 88%|████████▊ | 586/669 [6:16:05<1:00:44, 43.91s/it]
 88%|████████▊ | 587/669 [6:18:04<1:30:45, 66.41s/it]
                                                     
{'loss': 0.445, 'grad_norm': 0.5223556534474804, 'learning_rate': 6.22154779969651e-07, 'epoch': 2.59}

 88%|████████▊ | 587/669 [6:18:04<1:30:45, 66.41s/it]
 88%|████████▊ | 588/669 [6:18:48<1:20:28, 59.61s/it]
                                                     
{'loss': 0.4171, 'grad_norm': 0.4891137574673572, 'learning_rate': 6.14567526555387e-07, 'epoch': 2.59}

 88%|████████▊ | 588/669 [6:18:48<1:20:28, 59.61s/it]
 88%|████████▊ | 589/669 [6:19:32<1:13:13, 54.92s/it]
                                                     
{'loss': 0.4522, 'grad_norm': 0.5172276124686148, 'learning_rate': 6.06980273141123e-07, 'epoch': 2.59}

 88%|████████▊ | 589/669 [6:19:32<1:13:13, 54.92s/it]
 88%|████████▊ | 590/669 [6:20:16<1:07:59, 51.63s/it]
                                                     
{'loss': 0.4286, 'grad_norm': 0.48873823307103154, 'learning_rate': 5.99393019726859e-07, 'epoch': 2.6}

 88%|████████▊ | 590/669 [6:20:16<1:07:59, 51.63s/it]
 88%|████████▊ | 591/669 [6:21:00<1:04:06, 49.32s/it]
                                                     
{'loss': 0.4461, 'grad_norm': 0.502230133682332, 'learning_rate': 5.918057663125948e-07, 'epoch': 2.6}

 88%|████████▊ | 591/669 [6:21:00<1:04:06, 49.32s/it]
 88%|████████▊ | 592/669 [6:21:44<1:01:13, 47.71s/it]
                                                     
{'loss': 0.4289, 'grad_norm': 0.4898396410998909, 'learning_rate': 5.842185128983308e-07, 'epoch': 2.61}

 88%|████████▊ | 592/669 [6:21:44<1:01:13, 47.71s/it]
 89%|████████▊ | 593/669 [6:22:28<58:58, 46.57s/it]  
                                                   
{'loss': 0.4423, 'grad_norm': 0.48422568768513635, 'learning_rate': 5.766312594840668e-07, 'epoch': 2.61}

 89%|████████▊ | 593/669 [6:22:28<58:58, 46.57s/it]
 89%|████████▉ | 594/669 [6:23:12<57:13, 45.77s/it]
                                                   
{'loss': 0.437, 'grad_norm': 0.5003994120834477, 'learning_rate': 5.690440060698028e-07, 'epoch': 2.62}

 89%|████████▉ | 594/669 [6:23:12<57:13, 45.77s/it]
 89%|████████▉ | 595/669 [6:23:56<55:48, 45.24s/it]
                                                   
{'loss': 0.4516, 'grad_norm': 0.5079705228889307, 'learning_rate': 5.614567526555388e-07, 'epoch': 2.62}

 89%|████████▉ | 595/669 [6:23:56<55:48, 45.24s/it]
 89%|████████▉ | 596/669 [6:24:40<54:32, 44.83s/it]
                                                   
{'loss': 0.4368, 'grad_norm': 0.49246631419316356, 'learning_rate': 5.538694992412747e-07, 'epoch': 2.63}

 89%|████████▉ | 596/669 [6:24:40<54:32, 44.83s/it]
 89%|████████▉ | 597/669 [6:25:23<53:28, 44.56s/it]
                                                   
{'loss': 0.4227, 'grad_norm': 0.5086922889782935, 'learning_rate': 5.462822458270107e-07, 'epoch': 2.63}

 89%|████████▉ | 597/669 [6:25:23<53:28, 44.56s/it]
 89%|████████▉ | 598/669 [6:26:07<52:30, 44.37s/it]
                                                   
{'loss': 0.4462, 'grad_norm': 0.4780759168718152, 'learning_rate': 5.386949924127466e-07, 'epoch': 2.63}

 89%|████████▉ | 598/669 [6:26:07<52:30, 44.37s/it]
 90%|████████▉ | 599/669 [6:26:51<51:35, 44.22s/it]
                                                   
{'loss': 0.4334, 'grad_norm': 0.464930849547806, 'learning_rate': 5.311077389984825e-07, 'epoch': 2.64}

 90%|████████▉ | 599/669 [6:26:51<51:35, 44.22s/it]
 90%|████████▉ | 600/669 [6:27:35<50:44, 44.12s/it]
                                                   
{'loss': 0.4641, 'grad_norm': 0.5002300455424397, 'learning_rate': 5.235204855842186e-07, 'epoch': 2.64}

 90%|████████▉ | 600/669 [6:27:35<50:44, 44.12s/it]
 90%|████████▉ | 601/669 [6:28:19<49:56, 44.07s/it]
                                                   
{'loss': 0.4534, 'grad_norm': 0.4901819784256759, 'learning_rate': 5.159332321699545e-07, 'epoch': 2.65}

 90%|████████▉ | 601/669 [6:28:19<49:56, 44.07s/it]
 90%|████████▉ | 602/669 [6:29:03<49:10, 44.04s/it]
                                                   
{'loss': 0.4246, 'grad_norm': 0.48520187595124403, 'learning_rate': 5.083459787556905e-07, 'epoch': 2.65}

 90%|████████▉ | 602/669 [6:29:03<49:10, 44.04s/it]
 90%|█████████ | 603/669 [6:29:47<48:25, 44.03s/it]
                                                   
{'loss': 0.458, 'grad_norm': 0.5093634792454178, 'learning_rate': 5.007587253414264e-07, 'epoch': 2.66}

 90%|█████████ | 603/669 [6:29:47<48:25, 44.03s/it]
 90%|█████████ | 604/669 [6:30:31<47:42, 44.05s/it]
                                                   
{'loss': 0.4535, 'grad_norm': 0.47795678253705354, 'learning_rate': 4.931714719271624e-07, 'epoch': 2.66}

 90%|█████████ | 604/669 [6:30:31<47:42, 44.05s/it]
 90%|█████████ | 605/669 [6:31:15<46:57, 44.02s/it]
                                                   
{'loss': 0.4721, 'grad_norm': 0.48340944166993405, 'learning_rate': 4.855842185128983e-07, 'epoch': 2.67}

 90%|█████████ | 605/669 [6:31:15<46:57, 44.02s/it]
 91%|█████████ | 606/669 [6:31:59<46:12, 44.01s/it]
                                                   
{'loss': 0.4718, 'grad_norm': 0.47996848455446794, 'learning_rate': 4.779969650986344e-07, 'epoch': 2.67}

 91%|█████████ | 606/669 [6:31:59<46:12, 44.01s/it]
 91%|█████████ | 607/669 [6:32:43<45:27, 43.99s/it]
                                                   
{'loss': 0.4232, 'grad_norm': 0.5064110255390989, 'learning_rate': 4.7040971168437033e-07, 'epoch': 2.67}

 91%|█████████ | 607/669 [6:32:43<45:27, 43.99s/it]
 91%|█████████ | 608/669 [6:33:27<44:42, 43.98s/it]
                                                   
{'loss': 0.4307, 'grad_norm': 0.4824240302682651, 'learning_rate': 4.6282245827010624e-07, 'epoch': 2.68}

 91%|█████████ | 608/669 [6:33:27<44:42, 43.98s/it]
 91%|█████████ | 609/669 [6:34:11<43:57, 43.96s/it]
                                                   
{'loss': 0.4435, 'grad_norm': 0.4920800315955025, 'learning_rate': 4.552352048558422e-07, 'epoch': 2.68}

 91%|█████████ | 609/669 [6:34:11<43:57, 43.96s/it]
 91%|█████████ | 610/669 [6:34:55<43:12, 43.94s/it]
                                                   
{'loss': 0.4409, 'grad_norm': 0.48921504704563673, 'learning_rate': 4.476479514415782e-07, 'epoch': 2.69}

 91%|█████████ | 610/669 [6:34:55<43:12, 43.94s/it]
 91%|█████████▏| 611/669 [6:35:39<42:28, 43.94s/it]
                                                   
{'loss': 0.4512, 'grad_norm': 0.500357475284234, 'learning_rate': 4.4006069802731414e-07, 'epoch': 2.69}

 91%|█████████▏| 611/669 [6:35:39<42:28, 43.94s/it]
 91%|█████████▏| 612/669 [6:37:38<1:03:17, 66.62s/it]
                                                     
{'loss': 0.4305, 'grad_norm': 0.4772853981005586, 'learning_rate': 4.324734446130501e-07, 'epoch': 2.7}

 91%|█████████▏| 612/669 [6:37:38<1:03:17, 66.62s/it]
 92%|█████████▏| 613/669 [6:38:22<55:45, 59.74s/it]  
                                                   
{'loss': 0.4285, 'grad_norm': 0.5075964338612743, 'learning_rate': 4.248861911987861e-07, 'epoch': 2.7}

 92%|█████████▏| 613/669 [6:38:22<55:45, 59.74s/it]
 92%|█████████▏| 614/669 [6:39:06<50:24, 54.99s/it]
                                                   
{'loss': 0.4586, 'grad_norm': 0.4916721748305679, 'learning_rate': 4.1729893778452204e-07, 'epoch': 2.71}

 92%|█████████▏| 614/669 [6:39:06<50:24, 54.99s/it]
 92%|█████████▏| 615/669 [6:39:50<46:31, 51.69s/it]
                                                   
{'loss': 0.4385, 'grad_norm': 0.4746003660659448, 'learning_rate': 4.09711684370258e-07, 'epoch': 2.71}

 92%|█████████▏| 615/669 [6:39:50<46:31, 51.69s/it]
 92%|█████████▏| 616/669 [6:40:34<43:36, 49.36s/it]
                                                   
{'loss': 0.4349, 'grad_norm': 0.49004649736827927, 'learning_rate': 4.021244309559939e-07, 'epoch': 2.72}

 92%|█████████▏| 616/669 [6:40:34<43:36, 49.36s/it]/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(

 92%|█████████▏| 617/669 [6:41:51<50:04, 57.79s/it]
                                                   
{'loss': 0.4741, 'grad_norm': 0.5055169203153188, 'learning_rate': 3.945371775417299e-07, 'epoch': 2.72}

 92%|█████████▏| 617/669 [6:41:51<50:04, 57.79s/it]
 92%|█████████▏| 618/669 [6:42:35<45:35, 53.65s/it]
                                                   
{'loss': 0.4285, 'grad_norm': 0.5491416139361283, 'learning_rate': 3.8694992412746586e-07, 'epoch': 2.72}

 92%|█████████▏| 618/669 [6:42:35<45:35, 53.65s/it]
 93%|█████████▎| 619/669 [6:43:19<42:16, 50.73s/it]
                                                   
{'loss': 0.4147, 'grad_norm': 0.4860669923591883, 'learning_rate': 3.793626707132018e-07, 'epoch': 2.73}

 93%|█████████▎| 619/669 [6:43:19<42:16, 50.73s/it]
 93%|█████████▎| 620/669 [6:44:03<39:46, 48.70s/it]
                                                   
{'loss': 0.4538, 'grad_norm': 0.48537127261627433, 'learning_rate': 3.7177541729893784e-07, 'epoch': 2.73}

 93%|█████████▎| 620/669 [6:44:03<39:46, 48.70s/it]
 93%|█████████▎| 621/669 [6:44:47<37:48, 47.26s/it]
                                                   
{'loss': 0.454, 'grad_norm': 0.47557197343237195, 'learning_rate': 3.641881638846738e-07, 'epoch': 2.74}

 93%|█████████▎| 621/669 [6:44:47<37:48, 47.26s/it]
 93%|█████████▎| 622/669 [6:45:31<36:13, 46.24s/it]
                                                   
{'loss': 0.4273, 'grad_norm': 0.4844044051975442, 'learning_rate': 3.566009104704098e-07, 'epoch': 2.74}

 93%|█████████▎| 622/669 [6:45:31<36:13, 46.24s/it]
 93%|█████████▎| 623/669 [6:46:15<34:55, 45.55s/it]
                                                   
{'loss': 0.4315, 'grad_norm': 0.5000172198487197, 'learning_rate': 3.4901365705614574e-07, 'epoch': 2.75}

 93%|█████████▎| 623/669 [6:46:15<34:55, 45.55s/it]
 93%|█████████▎| 624/669 [6:46:59<33:47, 45.05s/it]
                                                   
{'loss': 0.4495, 'grad_norm': 0.4848986778566373, 'learning_rate': 3.4142640364188166e-07, 'epoch': 2.75}

 93%|█████████▎| 624/669 [6:46:59<33:47, 45.05s/it]
 93%|█████████▎| 625/669 [6:47:43<32:46, 44.70s/it]
                                                   
{'loss': 0.4185, 'grad_norm': 0.48082262468066483, 'learning_rate': 3.338391502276176e-07, 'epoch': 2.76}

 93%|█████████▎| 625/669 [6:47:43<32:46, 44.70s/it]
 94%|█████████▎| 626/669 [6:48:26<31:51, 44.44s/it]
                                                   
{'loss': 0.4538, 'grad_norm': 0.4865964770369407, 'learning_rate': 3.262518968133536e-07, 'epoch': 2.76}

 94%|█████████▎| 626/669 [6:48:26<31:51, 44.44s/it]
 94%|█████████▎| 627/669 [6:49:10<30:59, 44.28s/it]
                                                   
{'loss': 0.459, 'grad_norm': 0.4815694373638968, 'learning_rate': 3.1866464339908956e-07, 'epoch': 2.76}

 94%|█████████▎| 627/669 [6:49:10<30:59, 44.28s/it]
 94%|█████████▍| 628/669 [6:49:54<30:10, 44.17s/it]
                                                   
{'loss': 0.4527, 'grad_norm': 0.49485301744734556, 'learning_rate': 3.110773899848255e-07, 'epoch': 2.77}

 94%|█████████▍| 628/669 [6:49:54<30:10, 44.17s/it]
 94%|█████████▍| 629/669 [6:50:38<29:23, 44.10s/it]
                                                   
{'loss': 0.4393, 'grad_norm': 0.5214886405175438, 'learning_rate': 3.034901365705615e-07, 'epoch': 2.77}

 94%|█████████▍| 629/669 [6:50:38<29:23, 44.10s/it]
 94%|█████████▍| 630/669 [6:51:22<28:37, 44.04s/it]
                                                   
{'loss': 0.4723, 'grad_norm': 0.4645817613806632, 'learning_rate': 2.959028831562974e-07, 'epoch': 2.78}

 94%|█████████▍| 630/669 [6:51:22<28:37, 44.04s/it]
 94%|█████████▍| 631/669 [6:52:06<27:52, 44.01s/it]
                                                   
{'loss': 0.418, 'grad_norm': 0.4887881586065441, 'learning_rate': 2.883156297420334e-07, 'epoch': 2.78}

 94%|█████████▍| 631/669 [6:52:06<27:52, 44.01s/it]
 94%|█████████▍| 632/669 [6:52:50<27:08, 44.00s/it]
                                                   
{'loss': 0.4347, 'grad_norm': 0.47562569136752975, 'learning_rate': 2.807283763277694e-07, 'epoch': 2.79}

 94%|█████████▍| 632/669 [6:52:50<27:08, 44.00s/it]
 95%|█████████▍| 633/669 [6:53:34<26:25, 44.03s/it]
                                                   
{'loss': 0.4266, 'grad_norm': 0.4848933584055101, 'learning_rate': 2.7314112291350536e-07, 'epoch': 2.79}

 95%|█████████▍| 633/669 [6:53:34<26:25, 44.03s/it]
 95%|█████████▍| 634/669 [6:54:18<25:39, 43.99s/it]
                                                   
{'loss': 0.4522, 'grad_norm': 0.48829866920710585, 'learning_rate': 2.6555386949924127e-07, 'epoch': 2.8}

 95%|█████████▍| 634/669 [6:54:18<25:39, 43.99s/it]
 95%|█████████▍| 635/669 [6:55:02<24:54, 43.96s/it]
                                                   
{'loss': 0.4293, 'grad_norm': 0.4776396492507318, 'learning_rate': 2.5796661608497724e-07, 'epoch': 2.8}

 95%|█████████▍| 635/669 [6:55:02<24:54, 43.96s/it]
 95%|█████████▌| 636/669 [6:55:46<24:10, 43.96s/it]
                                                   
{'loss': 0.437, 'grad_norm': 0.48342626581685083, 'learning_rate': 2.503793626707132e-07, 'epoch': 2.8}

 95%|█████████▌| 636/669 [6:55:46<24:10, 43.96s/it]
 95%|█████████▌| 637/669 [6:57:44<35:17, 66.18s/it]
                                                   
{'loss': 0.4333, 'grad_norm': 0.47481545268800873, 'learning_rate': 2.4279210925644917e-07, 'epoch': 2.81}

 95%|█████████▌| 637/669 [6:57:44<35:17, 66.18s/it]
 95%|█████████▌| 638/669 [6:58:28<30:42, 59.44s/it]
                                                   
{'loss': 0.4367, 'grad_norm': 0.5105857690012979, 'learning_rate': 2.3520485584218516e-07, 'epoch': 2.81}

 95%|█████████▌| 638/669 [6:58:28<30:42, 59.44s/it]
 96%|█████████▌| 639/669 [6:59:11<27:23, 54.78s/it]
                                                   
{'loss': 0.4355, 'grad_norm': 0.49397311695562557, 'learning_rate': 2.276176024279211e-07, 'epoch': 2.82}

 96%|█████████▌| 639/669 [6:59:11<27:23, 54.78s/it]
 96%|█████████▌| 640/669 [6:59:55<24:54, 51.53s/it]
                                                   
{'loss': 0.4478, 'grad_norm': 0.4766966322472669, 'learning_rate': 2.2003034901365707e-07, 'epoch': 2.82}

 96%|█████████▌| 640/669 [6:59:55<24:54, 51.53s/it]
 96%|█████████▌| 641/669 [7:00:39<22:58, 49.25s/it]
                                                   
{'loss': 0.4366, 'grad_norm': 0.47484165703817144, 'learning_rate': 2.1244309559939304e-07, 'epoch': 2.83}

 96%|█████████▌| 641/669 [7:00:39<22:58, 49.25s/it]
 96%|█████████▌| 642/669 [7:01:23<21:26, 47.65s/it]
                                                   
{'loss': 0.4473, 'grad_norm': 0.47271019258961905, 'learning_rate': 2.04855842185129e-07, 'epoch': 2.83}

 96%|█████████▌| 642/669 [7:01:23<21:26, 47.65s/it]
 96%|█████████▌| 643/669 [7:02:07<20:09, 46.54s/it]
                                                   
{'loss': 0.4341, 'grad_norm': 0.4668544421054751, 'learning_rate': 1.9726858877086494e-07, 'epoch': 2.84}

 96%|█████████▌| 643/669 [7:02:07<20:09, 46.54s/it]
 96%|█████████▋| 644/669 [7:02:51<19:03, 45.75s/it]
                                                   
{'loss': 0.4223, 'grad_norm': 0.47716986376667936, 'learning_rate': 1.896813353566009e-07, 'epoch': 2.84}

 96%|█████████▋| 644/669 [7:02:51<19:03, 45.75s/it]
 96%|█████████▋| 645/669 [7:03:35<18:05, 45.21s/it]
                                                   
{'loss': 0.4441, 'grad_norm': 0.4939991882199546, 'learning_rate': 1.820940819423369e-07, 'epoch': 2.84}

 96%|█████████▋| 645/669 [7:03:35<18:05, 45.21s/it]
 97%|█████████▋| 646/669 [7:04:19<17:10, 44.81s/it]
                                                   
{'loss': 0.485, 'grad_norm': 0.47705877463455093, 'learning_rate': 1.7450682852807287e-07, 'epoch': 2.85}

 97%|█████████▋| 646/669 [7:04:19<17:10, 44.81s/it]
 97%|█████████▋| 647/669 [7:05:03<16:19, 44.54s/it]
                                                   
{'loss': 0.4701, 'grad_norm': 0.49941990019846055, 'learning_rate': 1.669195751138088e-07, 'epoch': 2.85}

 97%|█████████▋| 647/669 [7:05:03<16:19, 44.54s/it]
 97%|█████████▋| 648/669 [7:05:47<15:31, 44.36s/it]
                                                   
{'loss': 0.454, 'grad_norm': 0.47625945090740057, 'learning_rate': 1.5933232169954478e-07, 'epoch': 2.86}

 97%|█████████▋| 648/669 [7:05:47<15:31, 44.36s/it]
 97%|█████████▋| 649/669 [7:06:31<14:44, 44.22s/it]
                                                   
{'loss': 0.4296, 'grad_norm': 0.5028271948555065, 'learning_rate': 1.5174506828528074e-07, 'epoch': 2.86}

 97%|█████████▋| 649/669 [7:06:31<14:44, 44.22s/it]
 97%|█████████▋| 650/669 [7:07:15<13:58, 44.14s/it]
                                                   
{'loss': 0.4324, 'grad_norm': 0.49358352504671604, 'learning_rate': 1.441578148710167e-07, 'epoch': 2.87}

 97%|█████████▋| 650/669 [7:07:15<13:58, 44.14s/it]
 97%|█████████▋| 651/669 [7:07:59<13:13, 44.08s/it]
                                                   
{'loss': 0.4424, 'grad_norm': 0.48576223310259714, 'learning_rate': 1.3657056145675268e-07, 'epoch': 2.87}

 97%|█████████▋| 651/669 [7:07:59<13:13, 44.08s/it]
 97%|█████████▋| 652/669 [7:08:42<12:28, 44.01s/it]
                                                   
{'loss': 0.4555, 'grad_norm': 0.47971171549572994, 'learning_rate': 1.2898330804248862e-07, 'epoch': 2.88}

 97%|█████████▋| 652/669 [7:08:42<12:28, 44.01s/it]
 98%|█████████▊| 653/669 [7:09:26<11:43, 43.97s/it]
                                                   
{'loss': 0.4338, 'grad_norm': 0.49955392754214234, 'learning_rate': 1.2139605462822459e-07, 'epoch': 2.88}

 98%|█████████▊| 653/669 [7:09:26<11:43, 43.97s/it]
 98%|█████████▊| 654/669 [7:10:10<10:59, 43.97s/it]
                                                   
{'loss': 0.4349, 'grad_norm': 0.47232821100099637, 'learning_rate': 1.1380880121396055e-07, 'epoch': 2.88}

 98%|█████████▊| 654/669 [7:10:10<10:59, 43.97s/it]
 98%|█████████▊| 655/669 [7:10:54<10:15, 43.96s/it]
                                                   
{'loss': 0.4564, 'grad_norm': 0.4834170945858496, 'learning_rate': 1.0622154779969652e-07, 'epoch': 2.89}

 98%|█████████▊| 655/669 [7:10:54<10:15, 43.96s/it]
 98%|█████████▊| 656/669 [7:11:38<09:31, 43.94s/it]
                                                   
{'loss': 0.4384, 'grad_norm': 0.48493984226629705, 'learning_rate': 9.863429438543247e-08, 'epoch': 2.89}

 98%|█████████▊| 656/669 [7:11:38<09:31, 43.94s/it]
 98%|█████████▊| 657/669 [7:12:22<08:47, 43.93s/it]
                                                   
{'loss': 0.4512, 'grad_norm': 0.4601302405573049, 'learning_rate': 9.104704097116845e-08, 'epoch': 2.9}

 98%|█████████▊| 657/669 [7:12:22<08:47, 43.93s/it]
 98%|█████████▊| 658/669 [7:13:06<08:03, 43.94s/it]
                                                   
{'loss': 0.4354, 'grad_norm': 0.4768143158802154, 'learning_rate': 8.34597875569044e-08, 'epoch': 2.9}

 98%|█████████▊| 658/669 [7:13:06<08:03, 43.94s/it]
 99%|█████████▊| 659/669 [7:13:50<07:19, 43.94s/it]
                                                   
{'loss': 0.4259, 'grad_norm': 0.49427262358096746, 'learning_rate': 7.587253414264037e-08, 'epoch': 2.91}

 99%|█████████▊| 659/669 [7:13:50<07:19, 43.94s/it]
 99%|█████████▊| 660/669 [7:14:34<06:35, 43.92s/it]
                                                   
{'loss': 0.4445, 'grad_norm': 0.4749814243858473, 'learning_rate': 6.828528072837634e-08, 'epoch': 2.91}

 99%|█████████▊| 660/669 [7:14:34<06:35, 43.92s/it]
 99%|█████████▉| 661/669 [7:15:18<05:51, 43.93s/it]
                                                   
{'loss': 0.4799, 'grad_norm': 0.48533516583361297, 'learning_rate': 6.069802731411229e-08, 'epoch': 2.92}

 99%|█████████▉| 661/669 [7:15:18<05:51, 43.93s/it]
 99%|█████████▉| 662/669 [7:17:15<07:42, 66.05s/it]
                                                   
{'loss': 0.4574, 'grad_norm': 0.4839142899304728, 'learning_rate': 5.311077389984826e-08, 'epoch': 2.92}

 99%|█████████▉| 662/669 [7:17:15<07:42, 66.05s/it]
 99%|█████████▉| 663/669 [7:17:59<05:56, 59.36s/it]
                                                   
{'loss': 0.4446, 'grad_norm': 0.486543073762813, 'learning_rate': 4.5523520485584226e-08, 'epoch': 2.93}

 99%|█████████▉| 663/669 [7:17:59<05:56, 59.36s/it]
 99%|█████████▉| 664/669 [7:18:43<04:33, 54.73s/it]
                                                   
{'loss': 0.4315, 'grad_norm': 0.4980109939724879, 'learning_rate': 3.7936267071320186e-08, 'epoch': 2.93}

 99%|█████████▉| 664/669 [7:18:43<04:33, 54.73s/it]
 99%|█████████▉| 665/669 [7:19:27<03:25, 51.46s/it]
                                                   
{'loss': 0.4318, 'grad_norm': 0.48840296818117557, 'learning_rate': 3.0349013657056146e-08, 'epoch': 2.93}

 99%|█████████▉| 665/669 [7:19:27<03:25, 51.46s/it]
100%|█████████▉| 666/669 [7:20:11<02:27, 49.24s/it]
                                                   
{'loss': 0.4357, 'grad_norm': 0.4903615270099699, 'learning_rate': 2.2761760242792113e-08, 'epoch': 2.94}

100%|█████████▉| 666/669 [7:20:11<02:27, 49.24s/it]
100%|█████████▉| 667/669 [7:20:55<01:35, 47.64s/it]
                                                   
{'loss': 0.4334, 'grad_norm': 0.48850430880690693, 'learning_rate': 1.5174506828528073e-08, 'epoch': 2.94}

100%|█████████▉| 667/669 [7:20:55<01:35, 47.64s/it]
100%|█████████▉| 668/669 [7:21:39<00:46, 46.53s/it]
                                                   
{'loss': 0.4281, 'grad_norm': 0.4773488572878224, 'learning_rate': 7.587253414264037e-09, 'epoch': 2.95}

100%|█████████▉| 668/669 [7:21:39<00:46, 46.53s/it]
100%|██████████| 669/669 [7:22:23<00:00, 45.82s/it]
                                                   
{'loss': 0.4504, 'grad_norm': 0.4760606483764159, 'learning_rate': 0.0, 'epoch': 2.95}

100%|██████████| 669/669 [7:22:23<00:00, 45.82s/it]
                                                   
{'train_runtime': 26579.6548, 'train_samples_per_second': 29.834, 'train_steps_per_second': 0.025, 'train_loss': 0.3977131161989118, 'epoch': 2.95}

100%|██████████| 669/669 [7:22:57<00:00, 45.82s/it]
100%|██████████| 669/669 [7:22:57<00:00, 39.73s/it]
[2024-08-02 11:02:57,594] [INFO] [axolotl.train.train:190] [PID:218799] [RANK:0] Training Completed!!! Saving pre-trained model to llama_3.1_8b_function_calling
(LlamaForCausalLM(   (model): LlamaModel(     (embed_tokens): Embedding(128258, 4096)     (layers): ModuleList(       (0-31): 32 x LlamaDecoderLayer(         (self_attn): LlamaFlashAttention2(           (q_proj): Linear(in_features=4096, out_features=4096, bias=False)           (k_proj): Linear(in_features=4096, out_features=1024, bias=False)           (v_proj): Linear(in_features=4096, out_features=1024, bias=False)           (o_proj): Linear(in_features=4096, out_features=4096, bias=False)           (rotary_emb): LlamaRotaryEmbedding()         )         (mlp): LlamaMLP(           (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)           (up_proj): Linear(in_features=4096, out_features=14336, bias=False)           (down_proj): Linear(in_features=14336, out_features=4096, bias=False)           (act_fn): SiLU()         )         (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)         (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)       )     )     (norm): LlamaRMSNorm((4096,), eps=1e-05)     (rotary_emb): LlamaRotaryEmbedding()   )   (lm_head): Linear(in_features=4096, out_features=128258, bias=False) ), PreTrainedTokenizerFast(name_or_path='meta-llama/Meta-Llama-3.1-8B', vocab_size=128000, model_max_length=131072, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<|begin_of_text|>', 'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>'}, clean_up_tokenization_spaces=True),  added_tokens_decoder={ 	128000: AddedToken("<|begin_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128001: AddedToken("<|end_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128002: AddedToken("<|reserved_special_token_0|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128003: AddedToken("<|reserved_special_token_1|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128004: AddedToken("<|finetune_right_pad_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128005: AddedToken("<|reserved_special_token_2|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128006: AddedToken("<|start_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128007: AddedToken("<|end_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128008: AddedToken("<|eom_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128009: AddedToken("<|eot_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128010: AddedToken("<|python_tag|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128011: AddedToken("<|reserved_special_token_3|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128012: AddedToken("<|reserved_special_token_4|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128013: AddedToken("<|reserved_special_token_5|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128014: AddedToken("<|reserved_special_token_6|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128015: AddedToken("<|reserved_special_token_7|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128016: AddedToken("<|reserved_special_token_8|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128017: AddedToken("<|reserved_special_token_9|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128018: AddedToken("<|reserved_special_token_10|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128019: AddedToken("<|reserved_special_token_11|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128020: AddedToken("<|reserved_special_token_12|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128021: AddedToken("<|reserved_special_token_13|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128022: AddedToken("<|reserved_special_token_14|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128023: AddedToken("<|reserved_special_token_15|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128024: AddedToken("<|reserved_special_token_16|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128025: AddedToken("<|reserved_special_token_17|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128026: AddedToken("<|reserved_special_token_18|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128027: AddedToken("<|reserved_special_token_19|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128028: AddedToken("<|reserved_special_token_20|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128029: AddedToken("<|reserved_special_token_21|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128030: AddedToken("<|reserved_special_token_22|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128031: AddedToken("<|reserved_special_token_23|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128032: AddedToken("<|reserved_special_token_24|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128033: AddedToken("<|reserved_special_token_25|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128034: AddedToken("<|reserved_special_token_26|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128035: AddedToken("<|reserved_special_token_27|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128036: AddedToken("<|reserved_special_token_28|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128037: AddedToken("<|reserved_special_token_29|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128038: AddedToken("<|reserved_special_token_30|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128039: AddedToken("<|reserved_special_token_31|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128040: AddedToken("<|reserved_special_token_32|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128041: AddedToken("<|reserved_special_token_33|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128042: AddedToken("<|reserved_special_token_34|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128043: AddedToken("<|reserved_special_token_35|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128044: AddedToken("<|reserved_special_token_36|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128045: AddedToken("<|reserved_special_token_37|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128046: AddedToken("<|reserved_special_token_38|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128047: AddedToken("<|reserved_special_token_39|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128048: AddedToken("<|reserved_special_token_40|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128049: AddedToken("<|reserved_special_token_41|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128050: AddedToken("<|reserved_special_token_42|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128051: AddedToken("<|reserved_special_token_43|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128052: AddedToken("<|reserved_special_token_44|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128053: AddedToken("<|reserved_special_token_45|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128054: AddedToken("<|reserved_special_token_46|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128055: AddedToken("<|reserved_special_token_47|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128056: AddedToken("<|reserved_special_token_48|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128057: AddedToken("<|reserved_special_token_49|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128058: AddedToken("<|reserved_special_token_50|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128059: AddedToken("<|reserved_special_token_51|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128060: AddedToken("<|reserved_special_token_52|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128061: AddedToken("<|reserved_special_token_53|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128062: AddedToken("<|reserved_special_token_54|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128063: AddedToken("<|reserved_special_token_55|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128064: AddedToken("<|reserved_special_token_56|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128065: AddedToken("<|reserved_special_token_57|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128066: AddedToken("<|reserved_special_token_58|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128067: AddedToken("<|reserved_special_token_59|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128068: AddedToken("<|reserved_special_token_60|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128069: AddedToken("<|reserved_special_token_61|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128070: AddedToken("<|reserved_special_token_62|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128071: AddedToken("<|reserved_special_token_63|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128072: AddedToken("<|reserved_special_token_64|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128073: AddedToken("<|reserved_special_token_65|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128074: AddedToken("<|reserved_special_token_66|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128075: AddedToken("<|reserved_special_token_67|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128076: AddedToken("<|reserved_special_token_68|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128077: AddedToken("<|reserved_special_token_69|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128078: AddedToken("<|reserved_special_token_70|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128079: AddedToken("<|reserved_special_token_71|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128080: AddedToken("<|reserved_special_token_72|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128081: AddedToken("<|reserved_special_token_73|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128082: AddedToken("<|reserved_special_token_74|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128083: AddedToken("<|reserved_special_token_75|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128084: AddedToken("<|reserved_special_token_76|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128085: AddedToken("<|reserved_special_token_77|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128086: AddedToken("<|reserved_special_token_78|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128087: AddedToken("<|reserved_special_token_79|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128088: AddedToken("<|reserved_special_token_80|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128089: AddedToken("<|reserved_special_token_81|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128090: AddedToken("<|reserved_special_token_82|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128091: AddedToken("<|reserved_special_token_83|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128092: AddedToken("<|reserved_special_token_84|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128093: AddedToken("<|reserved_special_token_85|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128094: AddedToken("<|reserved_special_token_86|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128095: AddedToken("<|reserved_special_token_87|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128096: AddedToken("<|reserved_special_token_88|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128097: AddedToken("<|reserved_special_token_89|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128098: AddedToken("<|reserved_special_token_90|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128099: AddedToken("<|reserved_special_token_91|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128100: AddedToken("<|reserved_special_token_92|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128101: AddedToken("<|reserved_special_token_93|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128102: AddedToken("<|reserved_special_token_94|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128103: AddedToken("<|reserved_special_token_95|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128104: AddedToken("<|reserved_special_token_96|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128105: AddedToken("<|reserved_special_token_97|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128106: AddedToken("<|reserved_special_token_98|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128107: AddedToken("<|reserved_special_token_99|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128108: AddedToken("<|reserved_special_token_100|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128109: AddedToken("<|reserved_special_token_101|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128110: AddedToken("<|reserved_special_token_102|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128111: AddedToken("<|reserved_special_token_103|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128112: AddedToken("<|reserved_special_token_104|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128113: AddedToken("<|reserved_special_token_105|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128114: AddedToken("<|reserved_special_token_106|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128115: AddedToken("<|reserved_special_token_107|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128116: AddedToken("<|reserved_special_token_108|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128117: AddedToken("<|reserved_special_token_109|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128118: AddedToken("<|reserved_special_token_110|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128119: AddedToken("<|reserved_special_token_111|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128120: AddedToken("<|reserved_special_token_112|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128121: AddedToken("<|reserved_special_token_113|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128122: AddedToken("<|reserved_special_token_114|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128123: AddedToken("<|reserved_special_token_115|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128124: AddedToken("<|reserved_special_token_116|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128125: AddedToken("<|reserved_special_token_117|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128126: AddedToken("<|reserved_special_token_118|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128127: AddedToken("<|reserved_special_token_119|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128128: AddedToken("<|reserved_special_token_120|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128129: AddedToken("<|reserved_special_token_121|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128130: AddedToken("<|reserved_special_token_122|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128131: AddedToken("<|reserved_special_token_123|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128132: AddedToken("<|reserved_special_token_124|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128133: AddedToken("<|reserved_special_token_125|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128134: AddedToken("<|reserved_special_token_126|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128135: AddedToken("<|reserved_special_token_127|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128136: AddedToken("<|reserved_special_token_128|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128137: AddedToken("<|reserved_special_token_129|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128138: AddedToken("<|reserved_special_token_130|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128139: AddedToken("<|reserved_special_token_131|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128140: AddedToken("<|reserved_special_token_132|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128141: AddedToken("<|reserved_special_token_133|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128142: AddedToken("<|reserved_special_token_134|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128143: AddedToken("<|reserved_special_token_135|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128144: AddedToken("<|reserved_special_token_136|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128145: AddedToken("<|reserved_special_token_137|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128146: AddedToken("<|reserved_special_token_138|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128147: AddedToken("<|reserved_special_token_139|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128148: AddedToken("<|reserved_special_token_140|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128149: AddedToken("<|reserved_special_token_141|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128150: AddedToken("<|reserved_special_token_142|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128151: AddedToken("<|reserved_special_token_143|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128152: AddedToken("<|reserved_special_token_144|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128153: AddedToken("<|reserved_special_token_145|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128154: AddedToken("<|reserved_special_token_146|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128155: AddedToken("<|reserved_special_token_147|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128156: AddedToken("<|reserved_special_token_148|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128157: AddedToken("<|reserved_special_token_149|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128158: AddedToken("<|reserved_special_token_150|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128159: AddedToken("<|reserved_special_token_151|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128160: AddedToken("<|reserved_special_token_152|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128161: AddedToken("<|reserved_special_token_153|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128162: AddedToken("<|reserved_special_token_154|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128163: AddedToken("<|reserved_special_token_155|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128164: AddedToken("<|reserved_special_token_156|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128165: AddedToken("<|reserved_special_token_157|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128166: AddedToken("<|reserved_special_token_158|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128167: AddedToken("<|reserved_special_token_159|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128168: AddedToken("<|reserved_special_token_160|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128169: AddedToken("<|reserved_special_token_161|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128170: AddedToken("<|reserved_special_token_162|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128171: AddedToken("<|reserved_special_token_163|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128172: AddedToken("<|reserved_special_token_164|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128173: AddedToken("<|reserved_special_token_165|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128174: AddedToken("<|reserved_special_token_166|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128175: AddedToken("<|reserved_special_token_167|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128176: AddedToken("<|reserved_special_token_168|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128177: AddedToken("<|reserved_special_token_169|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128178: AddedToken("<|reserved_special_token_170|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128179: AddedToken("<|reserved_special_token_171|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128180: AddedToken("<|reserved_special_token_172|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128181: AddedToken("<|reserved_special_token_173|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128182: AddedToken("<|reserved_special_token_174|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128183: AddedToken("<|reserved_special_token_175|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128184: AddedToken("<|reserved_special_token_176|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128185: AddedToken("<|reserved_special_token_177|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128186: AddedToken("<|reserved_special_token_178|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128187: AddedToken("<|reserved_special_token_179|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128188: AddedToken("<|reserved_special_token_180|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128189: AddedToken("<|reserved_special_token_181|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128190: AddedToken("<|reserved_special_token_182|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128191: AddedToken("<|reserved_special_token_183|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128192: AddedToken("<|reserved_special_token_184|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128193: AddedToken("<|reserved_special_token_185|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128194: AddedToken("<|reserved_special_token_186|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128195: AddedToken("<|reserved_special_token_187|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128196: AddedToken("<|reserved_special_token_188|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128197: AddedToken("<|reserved_special_token_189|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128198: AddedToken("<|reserved_special_token_190|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128199: AddedToken("<|reserved_special_token_191|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128200: AddedToken("<|reserved_special_token_192|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128201: AddedToken("<|reserved_special_token_193|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128202: AddedToken("<|reserved_special_token_194|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128203: AddedToken("<|reserved_special_token_195|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128204: AddedToken("<|reserved_special_token_196|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128205: AddedToken("<|reserved_special_token_197|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128206: AddedToken("<|reserved_special_token_198|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128207: AddedToken("<|reserved_special_token_199|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128208: AddedToken("<|reserved_special_token_200|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128209: AddedToken("<|reserved_special_token_201|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128210: AddedToken("<|reserved_special_token_202|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128211: AddedToken("<|reserved_special_token_203|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128212: AddedToken("<|reserved_special_token_204|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128213: AddedToken("<|reserved_special_token_205|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128214: AddedToken("<|reserved_special_token_206|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128215: AddedToken("<|reserved_special_token_207|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128216: AddedToken("<|reserved_special_token_208|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128217: AddedToken("<|reserved_special_token_209|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128218: AddedToken("<|reserved_special_token_210|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128219: AddedToken("<|reserved_special_token_211|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128220: AddedToken("<|reserved_special_token_212|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128221: AddedToken("<|reserved_special_token_213|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128222: AddedToken("<|reserved_special_token_214|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128223: AddedToken("<|reserved_special_token_215|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128224: AddedToken("<|reserved_special_token_216|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128225: AddedToken("<|reserved_special_token_217|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128226: AddedToken("<|reserved_special_token_218|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128227: AddedToken("<|reserved_special_token_219|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128228: AddedToken("<|reserved_special_token_220|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128229: AddedToken("<|reserved_special_token_221|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128230: AddedToken("<|reserved_special_token_222|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128231: AddedToken("<|reserved_special_token_223|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128232: AddedToken("<|reserved_special_token_224|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128233: AddedToken("<|reserved_special_token_225|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128234: AddedToken("<|reserved_special_token_226|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128235: AddedToken("<|reserved_special_token_227|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128236: AddedToken("<|reserved_special_token_228|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128237: AddedToken("<|reserved_special_token_229|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128238: AddedToken("<|reserved_special_token_230|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128239: AddedToken("<|reserved_special_token_231|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128240: AddedToken("<|reserved_special_token_232|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128241: AddedToken("<|reserved_special_token_233|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128242: AddedToken("<|reserved_special_token_234|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128243: AddedToken("<|reserved_special_token_235|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128244: AddedToken("<|reserved_special_token_236|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128245: AddedToken("<|reserved_special_token_237|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128246: AddedToken("<|reserved_special_token_238|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128247: AddedToken("<|reserved_special_token_239|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128248: AddedToken("<|reserved_special_token_240|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128249: AddedToken("<|reserved_special_token_241|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128250: AddedToken("<|reserved_special_token_242|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128251: AddedToken("<|reserved_special_token_243|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128252: AddedToken("<|reserved_special_token_244|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128253: AddedToken("<|reserved_special_token_245|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128254: AddedToken("<|reserved_special_token_246|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128255: AddedToken("<|reserved_special_token_247|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128256: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128257: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), })
(LlamaForCausalLM(   (model): LlamaModel(     (embed_tokens): Embedding(128258, 4096)     (layers): ModuleList(       (0-31): 32 x LlamaDecoderLayer(         (self_attn): LlamaFlashAttention2(           (q_proj): Linear(in_features=4096, out_features=4096, bias=False)           (k_proj): Linear(in_features=4096, out_features=1024, bias=False)           (v_proj): Linear(in_features=4096, out_features=1024, bias=False)           (o_proj): Linear(in_features=4096, out_features=4096, bias=False)           (rotary_emb): LlamaRotaryEmbedding()         )         (mlp): LlamaMLP(           (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)           (up_proj): Linear(in_features=4096, out_features=14336, bias=False)           (down_proj): Linear(in_features=14336, out_features=4096, bias=False)           (act_fn): SiLU()         )         (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)         (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)       )     )     (norm): LlamaRMSNorm((4096,), eps=1e-05)     (rotary_emb): LlamaRotaryEmbedding()   )   (lm_head): Linear(in_features=4096, out_features=128258, bias=False) ), PreTrainedTokenizerFast(name_or_path='meta-llama/Meta-Llama-3.1-8B', vocab_size=128000, model_max_length=131072, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<|begin_of_text|>', 'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>'}, clean_up_tokenization_spaces=True),  added_tokens_decoder={ 	128000: AddedToken("<|begin_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128001: AddedToken("<|end_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128002: AddedToken("<|reserved_special_token_0|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128003: AddedToken("<|reserved_special_token_1|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128004: AddedToken("<|finetune_right_pad_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128005: AddedToken("<|reserved_special_token_2|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128006: AddedToken("<|start_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128007: AddedToken("<|end_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128008: AddedToken("<|eom_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128009: AddedToken("<|eot_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128010: AddedToken("<|python_tag|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128011: AddedToken("<|reserved_special_token_3|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128012: AddedToken("<|reserved_special_token_4|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128013: AddedToken("<|reserved_special_token_5|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128014: AddedToken("<|reserved_special_token_6|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128015: AddedToken("<|reserved_special_token_7|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128016: AddedToken("<|reserved_special_token_8|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128017: AddedToken("<|reserved_special_token_9|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128018: AddedToken("<|reserved_special_token_10|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128019: AddedToken("<|reserved_special_token_11|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128020: AddedToken("<|reserved_special_token_12|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128021: AddedToken("<|reserved_special_token_13|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128022: AddedToken("<|reserved_special_token_14|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128023: AddedToken("<|reserved_special_token_15|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128024: AddedToken("<|reserved_special_token_16|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128025: AddedToken("<|reserved_special_token_17|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128026: AddedToken("<|reserved_special_token_18|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128027: AddedToken("<|reserved_special_token_19|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128028: AddedToken("<|reserved_special_token_20|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128029: AddedToken("<|reserved_special_token_21|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128030: AddedToken("<|reserved_special_token_22|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128031: AddedToken("<|reserved_special_token_23|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128032: AddedToken("<|reserved_special_token_24|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128033: AddedToken("<|reserved_special_token_25|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128034: AddedToken("<|reserved_special_token_26|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128035: AddedToken("<|reserved_special_token_27|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128036: AddedToken("<|reserved_special_token_28|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128037: AddedToken("<|reserved_special_token_29|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128038: AddedToken("<|reserved_special_token_30|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128039: AddedToken("<|reserved_special_token_31|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128040: AddedToken("<|reserved_special_token_32|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128041: AddedToken("<|reserved_special_token_33|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128042: AddedToken("<|reserved_special_token_34|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128043: AddedToken("<|reserved_special_token_35|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128044: AddedToken("<|reserved_special_token_36|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128045: AddedToken("<|reserved_special_token_37|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128046: AddedToken("<|reserved_special_token_38|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128047: AddedToken("<|reserved_special_token_39|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128048: AddedToken("<|reserved_special_token_40|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128049: AddedToken("<|reserved_special_token_41|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128050: AddedToken("<|reserved_special_token_42|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128051: AddedToken("<|reserved_special_token_43|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128052: AddedToken("<|reserved_special_token_44|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128053: AddedToken("<|reserved_special_token_45|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128054: AddedToken("<|reserved_special_token_46|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128055: AddedToken("<|reserved_special_token_47|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128056: AddedToken("<|reserved_special_token_48|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128057: AddedToken("<|reserved_special_token_49|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128058: AddedToken("<|reserved_special_token_50|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128059: AddedToken("<|reserved_special_token_51|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128060: AddedToken("<|reserved_special_token_52|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128061: AddedToken("<|reserved_special_token_53|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128062: AddedToken("<|reserved_special_token_54|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128063: AddedToken("<|reserved_special_token_55|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128064: AddedToken("<|reserved_special_token_56|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128065: AddedToken("<|reserved_special_token_57|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128066: AddedToken("<|reserved_special_token_58|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128067: AddedToken("<|reserved_special_token_59|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128068: AddedToken("<|reserved_special_token_60|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128069: AddedToken("<|reserved_special_token_61|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128070: AddedToken("<|reserved_special_token_62|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128071: AddedToken("<|reserved_special_token_63|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128072: AddedToken("<|reserved_special_token_64|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128073: AddedToken("<|reserved_special_token_65|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128074: AddedToken("<|reserved_special_token_66|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128075: AddedToken("<|reserved_special_token_67|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128076: AddedToken("<|reserved_special_token_68|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128077: AddedToken("<|reserved_special_token_69|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128078: AddedToken("<|reserved_special_token_70|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128079: AddedToken("<|reserved_special_token_71|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128080: AddedToken("<|reserved_special_token_72|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128081: AddedToken("<|reserved_special_token_73|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128082: AddedToken("<|reserved_special_token_74|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128083: AddedToken("<|reserved_special_token_75|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128084: AddedToken("<|reserved_special_token_76|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128085: AddedToken("<|reserved_special_token_77|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128086: AddedToken("<|reserved_special_token_78|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128087: AddedToken("<|reserved_special_token_79|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128088: AddedToken("<|reserved_special_token_80|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128089: AddedToken("<|reserved_special_token_81|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128090: AddedToken("<|reserved_special_token_82|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128091: AddedToken("<|reserved_special_token_83|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128092: AddedToken("<|reserved_special_token_84|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128093: AddedToken("<|reserved_special_token_85|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128094: AddedToken("<|reserved_special_token_86|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128095: AddedToken("<|reserved_special_token_87|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128096: AddedToken("<|reserved_special_token_88|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128097: AddedToken("<|reserved_special_token_89|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128098: AddedToken("<|reserved_special_token_90|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128099: AddedToken("<|reserved_special_token_91|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128100: AddedToken("<|reserved_special_token_92|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128101: AddedToken("<|reserved_special_token_93|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128102: AddedToken("<|reserved_special_token_94|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128103: AddedToken("<|reserved_special_token_95|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128104: AddedToken("<|reserved_special_token_96|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128105: AddedToken("<|reserved_special_token_97|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128106: AddedToken("<|reserved_special_token_98|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128107: AddedToken("<|reserved_special_token_99|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128108: AddedToken("<|reserved_special_token_100|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128109: AddedToken("<|reserved_special_token_101|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128110: AddedToken("<|reserved_special_token_102|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128111: AddedToken("<|reserved_special_token_103|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128112: AddedToken("<|reserved_special_token_104|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128113: AddedToken("<|reserved_special_token_105|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128114: AddedToken("<|reserved_special_token_106|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128115: AddedToken("<|reserved_special_token_107|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128116: AddedToken("<|reserved_special_token_108|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128117: AddedToken("<|reserved_special_token_109|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128118: AddedToken("<|reserved_special_token_110|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128119: AddedToken("<|reserved_special_token_111|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128120: AddedToken("<|reserved_special_token_112|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128121: AddedToken("<|reserved_special_token_113|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128122: AddedToken("<|reserved_special_token_114|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128123: AddedToken("<|reserved_special_token_115|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128124: AddedToken("<|reserved_special_token_116|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128125: AddedToken("<|reserved_special_token_117|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128126: AddedToken("<|reserved_special_token_118|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128127: AddedToken("<|reserved_special_token_119|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128128: AddedToken("<|reserved_special_token_120|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128129: AddedToken("<|reserved_special_token_121|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128130: AddedToken("<|reserved_special_token_122|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128131: AddedToken("<|reserved_special_token_123|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128132: AddedToken("<|reserved_special_token_124|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128133: AddedToken("<|reserved_special_token_125|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128134: AddedToken("<|reserved_special_token_126|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128135: AddedToken("<|reserved_special_token_127|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128136: AddedToken("<|reserved_special_token_128|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128137: AddedToken("<|reserved_special_token_129|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128138: AddedToken("<|reserved_special_token_130|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128139: AddedToken("<|reserved_special_token_131|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128140: AddedToken("<|reserved_special_token_132|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128141: AddedToken("<|reserved_special_token_133|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128142: AddedToken("<|reserved_special_token_134|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128143: AddedToken("<|reserved_special_token_135|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128144: AddedToken("<|reserved_special_token_136|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128145: AddedToken("<|reserved_special_token_137|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128146: AddedToken("<|reserved_special_token_138|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128147: AddedToken("<|reserved_special_token_139|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128148: AddedToken("<|reserved_special_token_140|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128149: AddedToken("<|reserved_special_token_141|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128150: AddedToken("<|reserved_special_token_142|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128151: AddedToken("<|reserved_special_token_143|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128152: AddedToken("<|reserved_special_token_144|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128153: AddedToken("<|reserved_special_token_145|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128154: AddedToken("<|reserved_special_token_146|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128155: AddedToken("<|reserved_special_token_147|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128156: AddedToken("<|reserved_special_token_148|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128157: AddedToken("<|reserved_special_token_149|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128158: AddedToken("<|reserved_special_token_150|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128159: AddedToken("<|reserved_special_token_151|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128160: AddedToken("<|reserved_special_token_152|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128161: AddedToken("<|reserved_special_token_153|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128162: AddedToken("<|reserved_special_token_154|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128163: AddedToken("<|reserved_special_token_155|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128164: AddedToken("<|reserved_special_token_156|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128165: AddedToken("<|reserved_special_token_157|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128166: AddedToken("<|reserved_special_token_158|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128167: AddedToken("<|reserved_special_token_159|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128168: AddedToken("<|reserved_special_token_160|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128169: AddedToken("<|reserved_special_token_161|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128170: AddedToken("<|reserved_special_token_162|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128171: AddedToken("<|reserved_special_token_163|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128172: AddedToken("<|reserved_special_token_164|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128173: AddedToken("<|reserved_special_token_165|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128174: AddedToken("<|reserved_special_token_166|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128175: AddedToken("<|reserved_special_token_167|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128176: AddedToken("<|reserved_special_token_168|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128177: AddedToken("<|reserved_special_token_169|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128178: AddedToken("<|reserved_special_token_170|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128179: AddedToken("<|reserved_special_token_171|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128180: AddedToken("<|reserved_special_token_172|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128181: AddedToken("<|reserved_special_token_173|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128182: AddedToken("<|reserved_special_token_174|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128183: AddedToken("<|reserved_special_token_175|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128184: AddedToken("<|reserved_special_token_176|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128185: AddedToken("<|reserved_special_token_177|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128186: AddedToken("<|reserved_special_token_178|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128187: AddedToken("<|reserved_special_token_179|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128188: AddedToken("<|reserved_special_token_180|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128189: AddedToken("<|reserved_special_token_181|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128190: AddedToken("<|reserved_special_token_182|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128191: AddedToken("<|reserved_special_token_183|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128192: AddedToken("<|reserved_special_token_184|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128193: AddedToken("<|reserved_special_token_185|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128194: AddedToken("<|reserved_special_token_186|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128195: AddedToken("<|reserved_special_token_187|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128196: AddedToken("<|reserved_special_token_188|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128197: AddedToken("<|reserved_special_token_189|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128198: AddedToken("<|reserved_special_token_190|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128199: AddedToken("<|reserved_special_token_191|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128200: AddedToken("<|reserved_special_token_192|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128201: AddedToken("<|reserved_special_token_193|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128202: AddedToken("<|reserved_special_token_194|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128203: AddedToken("<|reserved_special_token_195|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128204: AddedToken("<|reserved_special_token_196|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128205: AddedToken("<|reserved_special_token_197|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128206: AddedToken("<|reserved_special_token_198|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128207: AddedToken("<|reserved_special_token_199|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128208: AddedToken("<|reserved_special_token_200|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128209: AddedToken("<|reserved_special_token_201|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128210: AddedToken("<|reserved_special_token_202|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128211: AddedToken("<|reserved_special_token_203|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128212: AddedToken("<|reserved_special_token_204|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128213: AddedToken("<|reserved_special_token_205|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128214: AddedToken("<|reserved_special_token_206|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128215: AddedToken("<|reserved_special_token_207|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128216: AddedToken("<|reserved_special_token_208|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128217: AddedToken("<|reserved_special_token_209|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128218: AddedToken("<|reserved_special_token_210|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128219: AddedToken("<|reserved_special_token_211|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128220: AddedToken("<|reserved_special_token_212|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128221: AddedToken("<|reserved_special_token_213|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128222: AddedToken("<|reserved_special_token_214|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128223: AddedToken("<|reserved_special_token_215|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128224: AddedToken("<|reserved_special_token_216|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128225: AddedToken("<|reserved_special_token_217|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128226: AddedToken("<|reserved_special_token_218|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128227: AddedToken("<|reserved_special_token_219|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128228: AddedToken("<|reserved_special_token_220|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128229: AddedToken("<|reserved_special_token_221|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128230: AddedToken("<|reserved_special_token_222|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128231: AddedToken("<|reserved_special_token_223|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128232: AddedToken("<|reserved_special_token_224|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128233: AddedToken("<|reserved_special_token_225|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128234: AddedToken("<|reserved_special_token_226|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128235: AddedToken("<|reserved_special_token_227|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128236: AddedToken("<|reserved_special_token_228|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128237: AddedToken("<|reserved_special_token_229|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128238: AddedToken("<|reserved_special_token_230|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128239: AddedToken("<|reserved_special_token_231|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128240: AddedToken("<|reserved_special_token_232|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128241: AddedToken("<|reserved_special_token_233|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128242: AddedToken("<|reserved_special_token_234|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128243: AddedToken("<|reserved_special_token_235|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128244: AddedToken("<|reserved_special_token_236|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128245: AddedToken("<|reserved_special_token_237|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128246: AddedToken("<|reserved_special_token_238|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128247: AddedToken("<|reserved_special_token_239|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128248: AddedToken("<|reserved_special_token_240|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128249: AddedToken("<|reserved_special_token_241|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128250: AddedToken("<|reserved_special_token_242|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128251: AddedToken("<|reserved_special_token_243|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128252: AddedToken("<|reserved_special_token_244|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128253: AddedToken("<|reserved_special_token_245|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128254: AddedToken("<|reserved_special_token_246|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128255: AddedToken("<|reserved_special_token_247|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128256: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128257: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), })
(LlamaForCausalLM(   (model): LlamaModel(     (embed_tokens): Embedding(128258, 4096)     (layers): ModuleList(       (0-31): 32 x LlamaDecoderLayer(         (self_attn): LlamaFlashAttention2(           (q_proj): Linear(in_features=4096, out_features=4096, bias=False)           (k_proj): Linear(in_features=4096, out_features=1024, bias=False)           (v_proj): Linear(in_features=4096, out_features=1024, bias=False)           (o_proj): Linear(in_features=4096, out_features=4096, bias=False)           (rotary_emb): LlamaRotaryEmbedding()         )         (mlp): LlamaMLP(           (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)           (up_proj): Linear(in_features=4096, out_features=14336, bias=False)           (down_proj): Linear(in_features=14336, out_features=4096, bias=False)           (act_fn): SiLU()         )         (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)         (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)       )     )     (norm): LlamaRMSNorm((4096,), eps=1e-05)     (rotary_emb): LlamaRotaryEmbedding()   )   (lm_head): Linear(in_features=4096, out_features=128258, bias=False) ), PreTrainedTokenizerFast(name_or_path='meta-llama/Meta-Llama-3.1-8B', vocab_size=128000, model_max_length=131072, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<|begin_of_text|>', 'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>'}, clean_up_tokenization_spaces=True),  added_tokens_decoder={ 	128000: AddedToken("<|begin_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128001: AddedToken("<|end_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128002: AddedToken("<|reserved_special_token_0|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128003: AddedToken("<|reserved_special_token_1|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128004: AddedToken("<|finetune_right_pad_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128005: AddedToken("<|reserved_special_token_2|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128006: AddedToken("<|start_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128007: AddedToken("<|end_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128008: AddedToken("<|eom_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128009: AddedToken("<|eot_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128010: AddedToken("<|python_tag|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128011: AddedToken("<|reserved_special_token_3|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128012: AddedToken("<|reserved_special_token_4|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128013: AddedToken("<|reserved_special_token_5|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128014: AddedToken("<|reserved_special_token_6|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128015: AddedToken("<|reserved_special_token_7|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128016: AddedToken("<|reserved_special_token_8|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128017: AddedToken("<|reserved_special_token_9|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128018: AddedToken("<|reserved_special_token_10|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128019: AddedToken("<|reserved_special_token_11|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128020: AddedToken("<|reserved_special_token_12|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128021: AddedToken("<|reserved_special_token_13|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128022: AddedToken("<|reserved_special_token_14|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128023: AddedToken("<|reserved_special_token_15|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128024: AddedToken("<|reserved_special_token_16|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128025: AddedToken("<|reserved_special_token_17|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128026: AddedToken("<|reserved_special_token_18|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128027: AddedToken("<|reserved_special_token_19|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128028: AddedToken("<|reserved_special_token_20|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128029: AddedToken("<|reserved_special_token_21|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128030: AddedToken("<|reserved_special_token_22|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128031: AddedToken("<|reserved_special_token_23|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128032: AddedToken("<|reserved_special_token_24|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128033: AddedToken("<|reserved_special_token_25|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128034: AddedToken("<|reserved_special_token_26|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128035: AddedToken("<|reserved_special_token_27|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128036: AddedToken("<|reserved_special_token_28|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128037: AddedToken("<|reserved_special_token_29|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128038: AddedToken("<|reserved_special_token_30|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128039: AddedToken("<|reserved_special_token_31|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128040: AddedToken("<|reserved_special_token_32|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128041: AddedToken("<|reserved_special_token_33|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128042: AddedToken("<|reserved_special_token_34|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128043: AddedToken("<|reserved_special_token_35|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128044: AddedToken("<|reserved_special_token_36|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128045: AddedToken("<|reserved_special_token_37|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128046: AddedToken("<|reserved_special_token_38|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128047: AddedToken("<|reserved_special_token_39|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128048: AddedToken("<|reserved_special_token_40|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128049: AddedToken("<|reserved_special_token_41|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128050: AddedToken("<|reserved_special_token_42|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128051: AddedToken("(LlamaForCausalLM(   (model): LlamaModel(     (embed_tokens): Embedding(128258, 4096)     (layers): ModuleList(       (0-31): 32 x LlamaDecoderLayer(         (self_attn): LlamaFlashAttention2(           (q_proj): Linear(in_features=4096, out_features=4096, bias=False)           (k_proj): Linear(in_features=4096, out_features=1024, bias=False)           (v_proj): Linear(in_features=4096, out_features=1024, bias=False)           (o_proj): Linear(in_features=4096, out_features=4096, bias=False)           (rotary_emb): LlamaRotaryEmbedding()         )         (mlp): LlamaMLP(           (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)           (up_proj): Linear(in_features=4096, out_features=14336, bias=False)           (down_proj): Linear(in_features=14336, out_features=4096, bias=False)           (act_fn): SiLU()         )         (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)         (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)       )     )     (norm): LlamaRMSNorm((4096,), eps=1e-05)     (rotary_emb): LlamaRotaryEmbedding()   )   (lm_head): Linear(in_features=4096, out_features=128258, bias=False) ), PreTrainedTokenizerFast(name_or_path='meta-llama/Meta-Llama-3.1-8B', vocab_size=128000, model_max_length=131072, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<|begin_of_text|>', 'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>'}, clean_up_tokenization_spaces=True),  added_tokens_decoder={ 	128000: AddedToken("<|begin_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128001: AddedToken("<|end_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128002: AddedToken("<|reserved_special_token_0|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128003: AddedToken("<|reserved_special_token_1|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128004: AddedToken("<|finetune_right_pad_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128005: AddedToken("<|reserved_special_token_2|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128006: AddedToken("<|start_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128007: AddedToken("<|end_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128008: AddedToken("<|eom_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128009: AddedToken("<|eot_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128010: AddedToken("<|python_tag|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128011: AddedToken("<|reserved_special_token_3|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128012: AddedToken("<|reserved_special_token_4|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128013: AddedToken("<|reserved_special_token_5|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128014: AddedToken("<|reserved_special_token_6|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128015: AddedToken("<|reserved_special_token_7|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128016: AddedToken("<|reserved_special_token_8|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128017: AddedToken("<|reserved_special_token_9|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128018: AddedToken("<|reserved_special_token_10|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128019: AddedToken("<|reserved_special_token_11|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128020: AddedToken("<|reserved_special_token_12|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128021: AddedToken("<|reserved_special_token_13|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128022: AddedToken("<|reserved_special_token_14|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128023: AddedToken("<|reserved_special_token_15|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128024: AddedToken("<|reserved_special_token_16|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128025: AddedToken("<|reserved_special_token_17|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128026: AddedToken("<|reserved_special_token_18|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128027: AddedToken("<|reserved_special_token_19|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128028: AddedToken("<|reserved_special_token_20|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128029: AddedToken("<|reserved_special_token_21|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128030: AddedToken("<|reserved_special_token_22|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128031: AddedToken("<|reserved_special_token_23|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128032: AddedToken("<|reserved_special_token_24|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128033: AddedToken("<|reserved_special_token_25|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128034: AddedToken("<|reserved_special_token_26|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128035: AddedToken("<|reserved_special_token_27|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128036: AddedToken("<|reserved_special_token_28|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128037: AddedToken("<|reserved_special_token_29|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128038: AddedToken("<|reserved_special_token_30|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128039: AddedToken("<|reserved_special_token_31|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128040: AddedToken("<|reserved_special_token_32|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128041: AddedToken("<|reserved_special_token_33|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128042: AddedToken("<|reserved_special_token_34|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128043: AddedToken("<|reserved_special_token_35|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128044: AddedToken("<|reserved_special_token_36|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128045: AddedToken("<|reserved_special_token_37|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128046: AddedToken("<|reserved_special_token_38|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128047: AddedToken("<|reserved_special_token_39|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128048: AddedToken("<|reserved_special_token_40|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128049: AddedToken("<|reserved_special_token_41|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128050: AddedToken("<|reserved_special_token_42|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128051: AddedToken("(LlamaForCausalLM(   (model): LlamaModel(     (embed_tokens): Embedding(128258, 4096)     (layers): ModuleList(       (0-31): 32 x LlamaDecoderLayer(         (self_attn): LlamaFlashAttention2(           (q_proj): Linear(in_features=4096, out_features=4096, bias=False)           (k_proj): Linear(in_features=4096, out_features=1024, bias=False)           (v_proj): Linear(in_features=4096, out_features=1024, bias=False)           (o_proj): Linear(in_features=4096, out_features=4096, bias=False)           (rotary_emb): LlamaRotaryEmbedding()         )         (mlp): LlamaMLP(           (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)           (up_proj): Linear(in_features=4096, out_features=14336, bias=False)           (down_proj): Linear(in_features=14336, out_features=4096, bias=False)           (act_fn): SiLU()         )         (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)         (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)       )     )     (norm): LlamaRMSNorm((4096,), eps=1e-05)     (rotary_emb): LlamaRotaryEmbedding()   )   (lm_head): Linear(in_features=4096, out_features=128258, bias=False) ), PreTrainedTokenizerFast(name_or_path='meta-llama/Meta-Llama-3.1-8B', vocab_size=128000, model_max_length=131072, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<|begin_of_text|>', 'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>'}, clean_up_tokenization_spaces=True),  added_tokens_decoder={ 	128000: AddedToken("<|begin_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128001: AddedToken("<|end_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128002: AddedToken("<|reserved_special_token_0|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128003: AddedToken("<|reserved_special_token_1|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128004: AddedToken("<|finetune_right_pad_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128005: AddedToken("<|reserved_special_token_2|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128006: AddedToken("<|start_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128007: AddedToken("<|end_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128008: AddedToken("<|eom_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128009: AddedToken("<|eot_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128010: AddedToken("<|python_tag|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128011: AddedToken("<|reserved_special_token_3|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128012: AddedToken("<|reserved_special_token_4|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128013: AddedToken("<|reserved_special_token_5|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128014: AddedToken("<|reserved_special_token_6|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128015: AddedToken("<|reserved_special_token_7|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128016: AddedToken("<|reserved_special_token_8|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128017: AddedToken("<|reserved_special_token_9|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128018: AddedToken("<|reserved_special_token_10|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128019: AddedToken("<|reserved_special_token_11|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128020: AddedToken("<|reserved_special_token_12|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128021: AddedToken("<|reserved_special_token_13|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128022: AddedToken("<|reserved_special_token_14|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128023: AddedToken("<|reserved_special_token_15|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128024: AddedToken("<|reserved_special_token_16|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128025: AddedToken("<|reserved_special_token_17|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128026: AddedToken("<|reserved_special_token_18|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128027: AddedToken("<|reserved_special_token_19|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128028: AddedToken("<|reserved_special_token_20|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128029: AddedToken("<|reserved_special_token_21|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128030: AddedToken("<|reserved_special_token_22|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128031: AddedToken("<|reserved_special_token_23|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128032: AddedToken("<|reserved_special_token_24|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128033: AddedToken("<|reserved_special_token_25|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128034: AddedToken("<|reserved_special_token_26|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128035: AddedToken("<|reserved_special_token_27|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128036: AddedToken("<|reserved_special_token_28|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128037: AddedToken("<|reserved_special_token_29|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128038: AddedToken("<|reserved_special_token_30|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128039: AddedToken("<|reserved_special_token_31|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128040: AddedToken("<|reserved_special_token_32|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128041: AddedToken("<|reserved_special_token_33|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128042: AddedToken("<|reserved_special_token_34|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128043: AddedToken("<|reserved_special_token_35|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128044: AddedToken("<|reserved_special_token_36|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128045: AddedToken("<|reserved_special_token_37|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128046: AddedToken("<|reserved_special_token_38|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128047: AddedToken("<|reserved_special_token_39|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128048: AddedToken("<|reserved_special_token_40|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128049: AddedToken("<|reserved_special_token_41|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128050: AddedToken("<|reserved_special_token_42|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128051: AddedToken("<|reserved_special_token_43|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128052: AddedToken("<|reserved_special_token_44|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128053: AddedToken("<|reserved_special_token_45|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128054: AddedToken("<|reserved_special_token_46|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128055: AddedToken("<|reserved_special_token_47|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128056: AddedToken("<|reserved_special_token_48|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128057: AddedToken("<|reserved_special_token_49|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128058: AddedToken("<|reserved_special_token_50|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128059: AddedToken("<|reserved_special_token_51|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128060: AddedToken("<|reserved_special_token_52|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128061: AddedToken("<|reserved_special_token_53|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128062: AddedToken("<|reserved_special_token_54|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128063: AddedToken("<|reserved_special_token_55|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128064: AddedToken("<|reserved_special_token_56|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128065: AddedToken("<|reserved_special_token_57|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128066: AddedToken("<|reserved_special_token_58|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128067: AddedToken("<|reserved_special_token_59|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128068: AddedToken("<|reserved_special_token_60|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128069: AddedToken("<|reserved_special_token_61|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128070: AddedToken("<|reserved_special_token_62|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128071: AddedToken("<|reserved_special_token_63|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128072: AddedToken("<|reserved_special_token_64|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128073: AddedToken("<|reserved_special_token_65|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128074: AddedToken("<|reserved_special_token_66|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128075: AddedToken("<|reserved_special_token_67|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128076: AddedToken("<|reserved_special_token_68|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128077: AddedToken("<|reserved_special_token_69|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128078: AddedToken("<|reserved_special_token_70|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128079: AddedToken("<|reserved_special_token_71|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128080: AddedToken("<|reserved_special_token_72|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128081: AddedToken("<|reserved_special_token_73|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128082: AddedToken("<|reserved_special_token_74|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128083: AddedToken("<|reserved_special_token_75|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128084: AddedToken("<|reserved_special_token_76|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128085: AddedToken("<|reserved_special_token_77|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128086: AddedToken("<|reserved_special_token_78|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128087: AddedToken("<|reserved_special_token_79|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128088: AddedToken("<|reserved_special_token_80|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128089: AddedToken("<|reserved_special_token_81|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128090: AddedToken("<|reserved_special_token_82|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128091: AddedToken("<|reserved_special_token_83|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128092: AddedToken("<|reserved_special_token_84|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128093: AddedToken("<|reserved_special_token_85|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128094: AddedToken("<|reserved_special_token_86|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128095: AddedToken("<|reserved_special_token_87|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128096: AddedToken("<|reserved_special_token_88|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128097: AddedToken("<|reserved_special_token_89|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128098: AddedToken("<|reserved_special_token_90|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128099: AddedToken("<|reserved_special_token_91|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128100: AddedToken("<|reserved_special_token_92|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128101: AddedToken("<|reserved_special_token_93|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128102: AddedToken("<|reserved_special_token_94|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128103: AddedToken("<|reserved_special_token_95|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128104: AddedToken("<|reserved_special_token_96|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128105: AddedToken("<|reserved_special_token_97|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128106: AddedToken("<|reserved_special_token_98|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128107: AddedToken("<|reserved_special_token_99|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128108: AddedToken("<|reserved_special_token_100|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128109: AddedToken("<|reserved_special_token_101|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128110: AddedToken("<|reserved_special_token_102|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128111: AddedToken("<|reserved_special_token_103|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128112: AddedToken("<|reserved_special_token_104|>", rstrip=False, lstrip=False, single_word=Fa(LlamaForCausalLM(   (model): LlamaModel(     (embed_tokens): Embedding(128258, 4096)     (layers): ModuleList(       (0-31): 32 x LlamaDecoderLayer(         (self_attn): LlamaFlashAttention2(           (q_proj): Linear(in_features=4096, out_features=4096, bias=False)           (k_proj): Linear(in_features=4096, out_features=1024, bias=False)           (v_proj): Linear(in_features=4096, out_features=1024, bias=False)           (o_proj): Linear(in_features=4096, out_features=4096, bias=False)           (rotary_emb): LlamaRotaryEmbedding()         )         (mlp): LlamaMLP(           (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)           (up_proj): Linear(in_features=4096, out_features=14336, bias=False)           (down_proj): Linear(in_features=14336, out_features=4096, bias=False)           (act_fn): SiLU()         )         (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)         (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)       )     )     (norm): LlamaRMSNorm((4096,), eps=1e-05)     (rotary_emb): LlamaRotaryEmbedding()   )   (lm_head): Linear(in_features=4096, out_features=128258, bias=False) ), PreTrainedTokenizerFast(name_or_path='meta-llama/Meta-Llama-3.1-8B', vocab_size=128000, model_max_length=131072, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<|begin_of_text|>', 'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>'}, clean_up_tokenization_spaces=True),  added_tokens_decoder={ 	128000: AddedToken("<|begin_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128001: AddedToken("<|end_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128002: AddedToken("<|reserved_special_token_0|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128003: AddedToken("<|reserved_special_token_1|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128004: AddedToken("<|finetune_right_pad_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128005: AddedToken("<|reserved_special_token_2|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128006: AddedToken("<|start_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128007: AddedToken("<|end_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128008: AddedToken("<|eom_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128009: AddedToken("<|eot_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128010: AddedToken("<|python_tag|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128011: AddedToken("<|reserved_special_token_3|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128012: AddedToken("<|reserved_special_token_4|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128013: AddedToken("<|reserved_special_token_5|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128014: AddedToken("<|reserved_special_token_6|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128015: AddedToken("<|reserved_special_token_7|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128016: AddedToken("<|reserved_special_token_8|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128017: AddedToken("<|reserved_special_token_9|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128018: AddedToken("<|reserved_special_token_10|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128019: AddedToken("<|reserved_special_token_11|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128020: AddedToken("<|reserved_special_token_12|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128021: AddedToken("<|reserved_special_token_13|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128022: AddedToken("<|reserved_special_token_14|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128023: AddedToken("<|reserved_special_token_15|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128024: AddedToken("<|reserved_special_token_16|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128025: AddedToken("<|reserved_special_token_17|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128026: AddedToken("<|reserved_special_token_18|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128027: AddedToken("<|reserved_special_token_19|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128028: AddedToken("<|reserved_special_token_20|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128029: AddedToken("<|reserved_special_token_21|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128030: AddedToken("<|reserved_special_token_22|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128031: AddedToken("<|reserved_special_token_23|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128032: AddedToken("<|reserved_special_token_24|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128033: AddedToken("<|reserved_special_token_25|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128034: AddedToken("<|reserved_special_token_26|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128035: AddedToken("<|reserved_special_token_27|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128036: AddedToken("<|reserved_special_token_28|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128037: AddedToken("<|reserved_special_token_29|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128038: AddedToken("<|reserved_special_token_30|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128039: AddedToken("<|reserved_special_token_31|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128040: AddedToken("<|reserved_special_token_32|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128041: AddedToken("<|reserved_special_token_33|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128042: AddedToken("<|reserved_special_token_34|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128043: AddedToken("<|reserved_special_token_35|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128044: AddedToken("<|reserved_special_token_36|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128045: AddedToken("<|reserved_special_token_37|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128046: AddedToken("<|reserved_special_token_38|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128047: AddedToken("<|reserved_special_token_39|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128048: AddedToken("<|reserved_special_token_40|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128049: AddedToken("<|reserved_special_token_41|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128050: AddedToken("<|reserved_special_token_42|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128051: AddedToken("<|reserved_special_token_43|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128052: AddedToken("<|reserved_special_token_44|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128053: AddedToken("<|reserved_special_token_45|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128054: AddedToken("<|reserved_special_token_46|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128055: AddedToken("<|reserved_special_token_47|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128056: AddedToken("<|reserved_special_token_48|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128057: AddedToken("<|reserved_special_token_49|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128058: AddedToken("<|reserved_special_token_50|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128059: AddedToken("<|reserved_special_token_51|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128060: AddedToken("<|reserved_special_token_52|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128061: AddedToken("<|reserved_special_token_53|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128062: AddedToken("<|reserved_special_token_54|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128063: AddedToken("<|reserved_special_token_55|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128064: AddedToken("<|reserved_special_token_56|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128065: AddedToken("<|reserved_special_token_57|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128066: AddedToken("<|reserved_special_token_58|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128067: AddedToken("<|reserved_special_token_59|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128068: AddedToken("<|reserved_special_token_60|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128069: AddedToken("<|reserved_special_token_61|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128070: AddedToken("<|reserved_special_token_62|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128071: AddedToken("<|reserved_special_token_63|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128072: AddedToken("<|reserved_special_token_64|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128073: AddedToken("<|reserved_special_token_65|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128074: AddedToken("<|reserved_special_token_66|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128075: AddedToken("<|reserved_special_token_67|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128076: AddedToken("<|reserved_special_token_68|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128077: AddedToken("<|reserved_special_token_69|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128078: AddedToken("<|reserved_special_token_70|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128079: AddedToken("<|reserved_special_token_71|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128080: AddedToken("<|reserved_special_token_72|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128081: AddedToken("<|reserved_special_token_73|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128082: AddedToken("<|reserved_special_token_74|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128083: AddedToken("<|reserved_special_token_75|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128084: AddedToken("<|reserved_special_token_76|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128085: AddedToken("<|reserved_special_token_77|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128086: AddedToken("<|reserved_special_token_78|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128087: AddedToken("<|reserved_special_token_79|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128088: AddedToken("<|reserved_special_token_80|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128089: AddedToken("<|reserved_special_token_81|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128090: AddedToken("<|reserved_special_token_82|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128091: AddedToken("<|reserved_special_token_83|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128092: AddedToken("<|reserved_special_token_84|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128093: AddedToken("<|reserved_special_token_85|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128094: AddedToken("<|reserved_special_token_86|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128095: AddedToken("<|reserved_special_token_87|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128096: AddedToken("<|reserved_special_token_88|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128097: AddedToken("<|reserved_special_token_89|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128098: AddedToken("<|reserved_special_token_90|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128099: AddedToken("<|reserved_special_token_91|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128100: AddedToken("<|reserved_special_token_92|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128101: AddedToken("<|reserved_special_token_93|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128102: AddedToken("<|reserved_special_token_94|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128103: AddedToken("<|reserved_special_token_95|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128104: AddedToken("<|reserved_special_token_96|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128105: AddedToken("<|reserved_special_token_97|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128106: AddedToken("<|reserved_special_token_98|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128107: AddedToken("<|reserved_special_token_99|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128108: AddedToken("<|reserved_special_token_100|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128109: AddedToken("<|reserved_special_token_101|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128110: AddedToken("<|reserved_special_token_102|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128111: AddedToken("<|reserved_special_token_103|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128112: AddedToken("<|reserved_special_token_104|>", rstrip=False, lstrip=False, single_word=Fa(LlamaForCausalLM(   (model): LlamaModel(     (embed_tokens): Embedding(128258, 4096)     (layers): ModuleList(       (0-31): 32 x LlamaDecoderLayer(         (self_attn): LlamaFlashAttention2(           (q_proj): Linear(in_features=4096, out_features=4096, bias=False)           (k_proj): Linear(in_features=4096, out_features=1024, bias=False)           (v_proj): Linear(in_features=4096, out_features=1024, bias=False)           (o_proj): Linear(in_features=4096, out_features=4096, bias=False)           (rotary_emb): LlamaRotaryEmbedding()         )         (mlp): LlamaMLP(           (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)           (up_proj): Linear(in_features=4096, out_features=14336, bias=False)           (down_proj): Linear(in_features=14336, out_features=4096, bias=False)           (act_fn): SiLU()         )         (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)         (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)       )     )     (norm): LlamaRMSNorm((4096,), eps=1e-05)     (rotary_emb): LlamaRotaryEmbedding()   )   (lm_head): Linear(in_features=4096, out_features=128258, bias=False) ), PreTrainedTokenizerFast(name_or_path='meta-llama/Meta-Llama-3.1-8B', vocab_size=128000, model_max_length=131072, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<|begin_of_text|>', 'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>'}, clean_up_tokenization_spaces=True),  added_tokens_decoder={ 	128000: AddedToken("<|begin_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128001: AddedToken("<|end_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128002: AddedToken("<|reserved_special_token_0|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128003: AddedToken("<|reserved_special_token_1|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128004: AddedToken("<|finetune_right_pad_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128005: AddedToken("<|reserved_special_token_2|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128006: AddedToken("<|start_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128007: AddedToken("<|end_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128008: AddedToken("<|eom_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128009: AddedToken("<|eot_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128010: AddedToken("<|python_tag|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128011: AddedToken("<|reserved_special_token_3|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128012: AddedToken("<|reserved_special_token_4|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128013: AddedToken("<|reserved_special_token_5|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128014: AddedToken("<|reserved_special_token_6|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128015: AddedToken("<|reserved_special_token_7|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128016: AddedToken("<|reserved_special_token_8|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128017: AddedToken("<|reserved_special_token_9|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128018: AddedToken("<|reserved_special_token_10|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128019: AddedToken("<|reserved_special_token_11|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128020: AddedToken("<|reserved_special_token_12|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128021: AddedToken("<|reserved_special_token_13|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128022: AddedToken("<|reserved_special_token_14|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128023: AddedToken("<|reserved_special_token_15|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128024: AddedToken("<|reserved_special_token_16|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128025: AddedToken("<|reserved_special_token_17|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128026: AddedToken("<|reserved_special_token_18|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128027: AddedToken("<|reserved_special_token_19|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128028: AddedToken("<|reserved_special_token_20|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128029: AddedToken("<|reserved_special_token_21|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128030: AddedToken("<|reserved_special_token_22|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128031: AddedToken("<|reserved_special_token_23|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128032: AddedToken("<|reserved_special_token_24|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128033: AddedToken("<|reserved_special_token_25|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128034: AddedToken("<|reserved_special_token_26|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128035: AddedToken("<|reserved_special_token_27|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128036: AddedToken("<|reserved_special_token_28|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128037: AddedToken("<|reserved_special_token_29|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128038: AddedToken("<|reserved_special_token_30|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128039: AddedToken("<|reserved_special_token_31|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128040: AddedToken("<|reserved_special_token_32|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128041: AddedToken("<|reserved_special_token_33|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128042: AddedToken("<|reserved_special_token_34|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128043: AddedToken("<|reserved_special_token_35|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128044: AddedToken("<|reserved_special_token_36|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128045: AddedToken("<|reserved_special_token_37|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128046: AddedToken("<|reserved_special_token_38|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128047: AddedToken("<|reserved_special_token_39|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128048: AddedToken("<|reserved_special_token_40|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128049: AddedToken("<|reserved_special_token_41|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128050: AddedToken("<|reserved_special_token_42|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128051: AddedToken("<|reserved_special_token_43|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128052: AddedToken("<|reserved_special_token_44|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128053: AddedToken("<|reserved_special_token_45|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128054: AddedToken("<|reserved_special_token_46|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128055: AddedToken("<|reserved_special_token_47|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128056: AddedToken("<|reserved_special_token_48|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128057: AddedToken("<|reserved_special_token_49|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128058: AddedToken("<|reserved_special_token_50|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128059: AddedToken("<|reserved_special_token_51|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128060: AddedToken("<|reserved_special_token_52|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128061: AddedToken("<|reserved_special_token_53|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128062: AddedToken("<|reserved_special_token_54|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128063: AddedToken("<|reserved_special_token_55|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128064: AddedToken("<|reserved_special_token_56|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128065: AddedToken("<|reserved_special_token_57|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128066: AddedToken("<|reserved_special_token_58|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128067: AddedToken("<|reserved_special_token_59|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128068: AddedToken("<|reserved_special_token_60|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128069: AddedToken("<|reserved_special_token_61|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128070: AddedToken("<|reserved_special_token_62|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128071: AddedToken("<|reserved_special_token_63|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128072: AddedToken("<|reserved_special_token_64|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128073: AddedToken("<|reserved_special_token_65|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128074: AddedToken("<|reserved_special_token_66|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128075: AddedToken("<|reserved_special_token_67|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128076: AddedToken("<|reserved_special_token_68|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128077: AddedToken("<|reserved_special_token_69|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128078: AddedToken("<|reserved_special_token_70|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128079: AddedToken("<|reserved_special_token_71|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128080: AddedToken("<|reserved_special_token_72|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128081: AddedToken("<|reserved_special_token_73|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128082: AddedToken("<|reserved_special_token_74|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128083: AddedToken("<|reserved_special_token_75|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128084: AddedToken("<|reserved_special_token_76|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128085: AddedToken("<|reserved_special_token_77|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128086: AddedToken("<|reserved_special_token_78|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128087: AddedToken("<|reserved_special_token_79|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128088: AddedToken("<|reserved_special_token_80|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128089: AddedToken("<|reserved_special_token_81|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128090: AddedToken("<|reserved_special_token_82|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128091: AddedToken("<|reserved_special_token_83|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128092: AddedToken("<|reserved_special_token_84|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128093: AddedToken("<|reserved_special_token_85|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128094: AddedToken("<|reserved_special_token_86|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128095: AddedToken("<|reserved_special_token_87|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128096: AddedToken("<|reserved_special_token_88|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128097: AddedToken("<|reserved_special_token_89|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128098: AddedToken("<|reserved_special_token_90|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128099: AddedToken("<|reserved_special_token_91|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128100: AddedToken("<|reserved_special_token_92|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128101: AddedToken("<|reserved_special_token_93|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128102: AddedToken("<|reserved_special_token_94|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128103: AddedToken("<|reserved_special_token_95|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128104: AddedToken("<|reserved_special_token_96|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128105: AddedToken("<|reserved_special_token_97|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128106: AddedToken("<|reserved_special_token_98|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128107: AddedToken("<|reserved_special_token_99|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128108: AddedToken("<|reserved_special_token_100|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128109: AddedToken("<|reserved_special_token_101|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128110: AddedToken("<|reserved_special_token_102|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128111: AddedToken("<|reserved_special_token_103|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128112: AddedToken("<|reserved_special_token_104|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128113: AddedToken("<|reserved_special_token_105|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128114: AddedToken("<|reserved_special_token_106|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128115: AddedToken("<|reserved_special_token_107|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128116: AddedToken("<|reserved_special_token_108|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128117: AddedToken("<|reserved_special_token_109|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128118: AddedToken("<|reserved_special_token_110|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128119: AddedToken("<|reserved_special_token_111|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128120: AddedToken("<|reserved_special_token_112|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128121: AddedToken("<|reserved_special_token_113|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128122: AddedToken("<|reserved_special_token_114|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128123: AddedToken("<|reserved_special_token_115|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128124: AddedToken("<|reserved_special_token_116|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128125: AddedToken("<|reserved_special_token_117|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128126: AddedToken("<|reserved_special_token_118|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128127: AddedToken("<|reserved_special_token_119|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128128: AddedToken("<|reserved_special_token_120|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128129: AddedToken("<|reserved_special_token_121|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128130: AddedToken("<|reserved_special_token_122|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128131: AddedToken("<|reserved_special_token_123|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128132: AddedToken("<|reserved_special_token_124|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128133: AddedToken("<|reserved_special_token_125|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128134: AddedToken("<|reserved_special_token_126|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128135: AddedToken("<|reserved_special_token_127|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128136: AddedToken("<|reserved_special_token_128|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128137: AddedToken("<|reserved_special_token_129|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128138: AddedToken("<|reserved_special_token_130|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128139: AddedToken("<|reserved_special_token_131|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128140: AddedToken("<|reserved_special_token_132|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128141: AddedToken("<|reserved_special_token_133|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128142: AddedToken("<|reserved_special_token_134|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128143: AddedToken("<|reserved_special_token_135|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128144: AddedToken("<|reserved_special_token_136|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128145: AddedToken("<|reserved_special_token_137|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128146: AddedToken("<|reserved_special_token_138|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128147: AddedToken("<|reserved_special_token_139|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128148: AddedToken("<|reserved_special_token_140|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128149: AddedToken("<|reserved_special_token_141|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128150: AddedToken("<|reserved_special_token_142|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128151: AddedToken("<|reserved_special_token_143|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128152: AddedToken("<|reserved_special_token_144|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128153: AddedToken("<|reserved_special_token_145|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128154: AddedToken("<|reserved_special_token_146|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128155: AddedToken("<|reserved_special_token_147|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128156: AddedToken("<|reserved_special_token_148|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128157: AddedToken("<|reserved_special_token_149|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128158: AddedToken("<|reserved_special_token_150|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128159: AddedToken("<|reserved_special_token_151|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128160: AddedToken("<|reserved_special_token_152|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128161: AddedToken("<|reserved_special_token_153|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128162: AddedToken("<|reserved_special_token_154|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128163: AddedToken("<|reserved_special_token_155|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128164: AddedToken("<|reserved_special_token_156|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128165: AddedToken("<|reserved_special_token_157|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128166: AddedToken("<|reserved_special_token_158|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128167: AddedToken("<|reserved_special_token_159|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128168: AddedToken("<|reserved_special_token_160|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128169: AddedToken("<|reserved_special_token_161|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128170: AddedToken("<|reserved_special_token_162|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128171: AddedToken("<|reserved_special_token_163|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128172: AddedToken("<|reserved_special_token_164|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128173: AddedToken("<|reserved_special_token_165|>", rstrip=False, lstrip=False, single_word=False, normalized=Fa<|reserved_special_token_43|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128052: AddedToken("<|reserved_special_token_44|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128053: AddedToken("<|reserved_special_token_45|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128054: AddedToken("<|reserved_special_token_46|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128055: AddedToken("<|reserved_special_token_47|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128056: AddedToken("<|reserved_special_token_48|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128057: AddedToken("<|reserved_special_token_49|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128058: AddedToken("<|reserved_special_token_50|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128059: AddedToken("<|reserved_special_token_51|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128060: AddedToken("<|reserved_special_token_52|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128061: AddedToken("<|reserved_special_token_53|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128062: AddedToken("<|reserved_special_token_54|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128063: AddedToken("<|reserved_special_token_55|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128064: AddedToken("<|reserved_special_token_56|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128065: AddedToken("<|reserved_special_token_57|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128066: AddedToken("<|reserved_special_token_58|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128067: AddedToken("<|reserved_special_token_59|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128068: AddedToken("<|reserved_special_token_60|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128069: AddedToken("<|reserved_special_token_61|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128070: AddedToken("<|reserved_special_token_62|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128071: AddedToken("<|reserved_special_token_63|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128072: AddedToken("<|reserved_special_token_64|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128073: AddedToken("<|reserved_special_token_65|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128074: AddedToken("<|reserved_special_token_66|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128075: AddedToken("<|reserved_special_token_67|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128076: AddedToken("<|reserved_special_token_68|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128077: AddedToken("<|reserved_special_token_69|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128078: AddedToken("<|reserved_special_token_70|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128079: AddedToken("<|reserved_special_token_71|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128080: AddedToken("<|reserved_special_token_72|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128081: AddedToken("<|reserved_special_token_73|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128082: AddedToken("<|reserved_special_token_74|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128083: AddedToken("<|reserved_special_token_75|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128084: AddedToken("<|reserved_special_token_76|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128085: AddedToken("<|reserved_special_token_77|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128086: AddedToken("<|reserved_special_token_78|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128087: AddedToken("<|reserved_special_token_79|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128088: AddedToken("<|reserved_special_token_80|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128089: AddedToken("<|reserved_special_token_81|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128090: AddedToken("<|reserved_special_token_82|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128091: AddedToken("<|reserved_special_token_83|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128092: AddedToken("<|reserved_special_token_84|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128093: AddedToken("<|reserved_special_token_85|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128094: AddedToken("<|reserved_special_token_86|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128095: AddedToken("<|reserved_special_token_87|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128096: AddedToken("<|reserved_special_token_88|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128097: AddedToken("<|reserved_special_token_89|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128098: AddedToken("<|reserved_special_token_90|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128099: AddedToken("<|reserved_special_token_91|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128100: AddedToken("<|reserved_special_token_92|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128101: AddedToken("<|reserved_special_token_93|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128102: AddedToken("<|reserved_special_token_94|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128103: AddedToken("<|reserved_special_token_95|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128104: AddedToken("<|reserved_special_token_96|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128105: AddedToken("<|reserved_special_token_97|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128106: AddedToken("<|reserved_special_token_98|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128107: AddedToken("<|reserved_special_token_99|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128108: AddedToken("<|reserved_special_token_100|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128109: AddedToken("<|reserved_special_token_101|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128110: AddedToken("<|reserved_special_token_102|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128111: AddedToken("<|reserved_special_token_103|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128112: AddedToken("<|reserved_special_token_104|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128113: AddedToken("<|reserved_special_token_105|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128114: AddedToken("<|reserved_special_token_106|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128115: AddedToken("<|reserved_special_token_107|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128116: AddedToken("<|reserved_special_token_108|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128117: AddedToken("<|reserved_special_token_109|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128118: AddedToken("<|reserved_special_token_110|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128119: AddedToken("<|reserved_special_token_111|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128120: AddedToken("<|reserved_special_token_112|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128121: AddedToken("<|reserved_special_token_113|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128122: AddedToken("<|reserved_special_token_114|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128123: AddedToken("<|reserved_special_token_115|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128124: AddedToken("<|reserved_special_token_116|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128125: AddedToken("<|reserved_special_token_117|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128126: AddedToken("<|reserved_special_token_118|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128127: AddedToken("<|reserved_special_token_119|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128128: AddedToken("<|reserved_special_token_120|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128129: AddedToken("<|reserved_special_token_121|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128130: AddedToken("<|reserved_special_token_122|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128131: AddedToken("<|reserved_special_token_123|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128132: AddedToken("<|reserved_special_token_124|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128133: AddedToken("<|reserved_special_token_125|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128134: AddedToken("<|reserved_special_token_126|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128135: AddedToken("<|reserved_special_token_127|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128136: AddedToken("<|reserved_special_token_128|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128137: AddedToken("<|reserved_special_token_129|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128138: AddedToken("<|reserved_special_token_130|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128139: AddedToken("<|reserved_special_token_131|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128140: AddedToken("<|reserved_special_token_132|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128141: AddedToken("<|reserved_special_token_133|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128142: AddedToken("<|reserved_special_token_134|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128143: AddedToken("<|reserved_special_token_135|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128144: AddedToken("<|reserved_special_token_136|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128145: AddedToken("<|reserved_special_token_137|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128146: AddedToken("<|reserved_special_token_138|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128147: AddedToken("<|reserved_special_token_139|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128148: AddedToken("<|reserved_special_token_140|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128149: AddedToken("<|reserved_special_token_141|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128150: AddedToken("<|reserved_special_token_142|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128151: AddedToken("<|reserved_special_token_143|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128152: AddedToken("<|reserved_special_token_144|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128153: AddedToken("<|reserved_special_token_145|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128154: AddedToken("<|reserved_special_token_146|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128155: AddedToken("<|reserved_special_token_147|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128156: AddedToken("<|reserved_special_token_148|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128157: AddedToken("<|reserved_special_token_149|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128158: AddedToken("<|reserved_special_token_150|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128159: AddedToken("<|reserved_special_token_151|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128160: AddedToken("<|reserved_special_token_152|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128161: AddedToken("<|reserved_special_token_153|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128162: AddedToken("<|reserved_special_token_154|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128163: AddedToken("<|reserved_special_token_155|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128164: AddedToken("<|reserved_special_token_156|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128165: AddedToken("<|reserved_special_token_157|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128166: AddedToken("<|reserved_special_token_158|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128167: AddedToken("<|reserved_special_token_159|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128168: AddedToken("<|reserved_special_token_160|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128169: AddedToken("<|reserved_special_token_161|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128170: AddedToken("<|reserved_special_token_162|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128171: AddedToken("<|reserved_special_token_163|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128172: AddedToken("<|reserved_special_token_164|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128173: AddedToken("<|reserved_special_token_165|>", rstrip=False, lstrip=False, single_word=False, normalized=Fa<|reserved_special_token_43|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128052: AddedToken("<|reserved_special_token_44|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128053: AddedToken("<|reserved_special_token_45|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128054: AddedToken("<|reserved_special_token_46|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128055: AddedToken("<|reserved_special_token_47|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128056: AddedToken("<|reserved_special_token_48|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128057: AddedToken("<|reserved_special_token_49|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128058: AddedToken("<|reserved_special_token_50|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128059: AddedToken("<|reserved_special_token_51|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128060: AddedToken("<|reserved_special_token_52|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128061: AddedToken("<|reserved_special_token_53|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128062: AddedToken("<|reserved_special_token_54|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128063: AddedToken("<|reserved_special_token_55|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128064: AddedToken("<|reserved_special_token_56|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128065: AddedToken("<|reserved_special_token_57|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128066: AddedToken("<|reserved_special_token_58|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128067: AddedToken("<|reserved_special_token_59|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128068: AddedToken("<|reserved_special_token_60|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128069: AddedToken("<|reserved_special_token_61|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128070: AddedToken("<|reserved_special_token_62|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128071: AddedToken("<|reserved_special_token_63|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128072: AddedToken("<|reserved_special_token_64|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128073: AddedToken("<|reserved_special_token_65|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128074: AddedToken("<|reserved_special_token_66|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128075: AddedToken("<|reserved_special_token_67|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128076: AddedToken("<|reserved_special_token_68|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128077: AddedToken("<|reserved_special_token_69|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128078: AddedToken("<|reserved_special_token_70|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128079: AddedToken("<|reserved_special_token_71|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128080: AddedToken("<|reserved_special_token_72|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128081: AddedToken("<|reserved_special_token_73|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128082: AddedToken("<|reserved_special_token_74|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128083: AddedToken("<|reserved_special_token_75|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128084: AddedToken("<|reserved_special_token_76|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128085: AddedToken("<|reserved_special_token_77|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128086: AddedToken("<|reserved_special_token_78|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128087: AddedToken("<|reserved_special_token_79|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128088: AddedToken("<|reserved_special_token_80|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128089: AddedToken("<|reserved_special_token_81|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128090: AddedToken("<|reserved_special_token_82|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128091: AddedToken("<|reserved_special_token_83|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128092: AddedToken("<|reserved_special_token_84|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128093: AddedToken("<|reserved_special_token_85|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128094: AddedToken("<|reserved_special_token_86|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128095: AddedToken("<|reserved_special_token_87|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128096: AddedToken("<|reserved_special_token_88|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128097: AddedToken("<|reserved_special_token_89|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128098: AddedToken("<|reserved_special_token_90|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128099: AddedToken("<|reserved_special_token_91|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128100: AddedToken("<|reserved_special_token_92|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128101: AddedToken("<|reserved_special_token_93|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128102: AddedToken("<|reserved_special_token_94|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128103: AddedToken("<|reserved_special_token_95|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128104: AddedToken("<|reserved_special_token_96|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128105: AddedToken("<|reserved_special_token_97|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128106: AddedToken("<|reserved_special_token_98|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128107: AddedToken("<|reserved_special_token_99|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128108: AddedToken("<|reserved_special_token_100|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128109: AddedToken("<|reserved_special_token_101|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128110: AddedToken("<|reserved_special_token_102|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128111: AddedToken("<|reserved_special_token_103|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128112: AddedToken("<|reserved_special_token_104|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128113: AddedToken("<|reserved_special_token_105|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128114: AddedToken("<|reserved_special_token_106|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128115: AddedToken("<|reserved_special_token_107|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128116: AddedToken("<|reserved_special_token_108|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128117: AddedToken("<|reserved_special_token_109|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128118: AddedToken("<|reserved_special_token_110|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128119: AddedToken("<|reserved_special_token_111|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128120: AddedToken("<|reserved_special_token_112|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128121: AddedToken("<|reserved_special_token_113|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128122: AddedToken("<|reserved_special_token_114|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128123: AddedToken("<|reserved_special_token_115|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128124: AddedToken("<|reserved_special_token_116|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128125: AddedToken("<|reserved_special_token_117|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128126: AddedToken("<|reserved_special_token_118|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128127: AddedToken("<|reserved_special_token_119|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128128: AddedToken("<|reserved_special_token_120|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128129: AddedToken("<|reserved_special_token_121|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128130: AddedToken("<|reserved_special_token_122|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128131: AddedToken("<|reserved_special_token_123|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128132: AddedToken("<|reserved_special_token_124|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128133: AddedToken("<|reserved_special_token_125|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128134: AddedToken("<|reserved_special_token_126|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128135: AddedToken("<|reserved_special_token_127|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128136: AddedToken("<|reserved_special_token_128|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128137: AddedToken("<|reserved_special_token_129|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128138: AddedToken("<|reserved_special_token_130|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128139: AddedToken("<|reserved_special_token_131|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128140: AddedToken("<|reserved_special_token_132|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128141: AddedToken("<|reserved_special_token_133|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128142: AddedToken("<|reserved_special_token_134|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128143: AddedToken("<|reserved_special_token_135|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128144: AddedToken("<|reserved_special_token_136|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128145: AddedToken("<|reserved_special_token_137|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128146: AddedToken("<|reserved_special_token_138|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128147: AddedToken("<|reserved_special_token_139|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128148: AddedToken("<|reserved_special_token_140|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128149: AddedToken("<|reserved_special_token_141|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128150: AddedToken("<|reserved_special_token_142|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128151: AddedToken("<|reserved_special_token_143|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128152: AddedToken("<|reserved_special_token_144|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128153: AddedToken("<|reserved_special_token_145|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128154: AddedToken("<|reserved_special_token_146|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128155: AddedToken("<|reserved_special_token_147|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128156: AddedToken("<|reserved_special_token_148|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128157: AddedToken("<|reserved_special_token_149|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128158: AddedToken("<|reserved_special_token_150|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128159: AddedToken("<|reserved_special_token_151|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128160: AddedToken("<|reserved_special_token_152|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128161: AddedToken("<|reserved_special_token_153|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128162: AddedToken("<|reserved_special_token_154|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128163: AddedToken("<|reserved_special_token_155|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128164: AddedToken("<|reserved_special_token_156|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128165: AddedToken("<|reserved_special_token_157|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128166: AddedToken("<|reserved_special_token_158|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128167: AddedToken("<|reserved_special_token_159|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128168: AddedToken("<|reserved_special_token_160|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128169: AddedToken("<|reserved_special_token_161|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128170: AddedToken("<|reserved_special_token_162|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128171: AddedToken("<|reserved_special_token_163|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128172: AddedToken("<|reserved_special_token_164|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128173: AddedToken("<|reserved_special_token_165|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128174: AddedToken("<|reserved_special_token_166|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128175: AddedToken("<|reserved_special_token_167|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128176: AddedToken("<|reserved_special_token_168|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128177: AddedToken("<|reserved_special_token_169|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128178: AddedToken("<|reserved_special_token_170|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128179: AddedToken("<|reserved_special_token_171|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128180: AddedToken("<|reserved_special_token_172|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128181: AddedToken("<|reserved_special_token_173|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128182: AddedToken("<|reserved_special_token_174|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128183: AddedToken("<|reserved_special_token_175|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128184: AddedToken("<|reserved_special_token_176|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128185: AddedToken("<|reserved_special_token_177|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128186: AddedToken("<|reserved_special_token_178|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128187: AddedToken("<|reserved_special_token_179|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128188: AddedToken("<|reserved_special_token_180|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128189: AddedToken("<|reserved_special_token_181|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128190: AddedToken("<|reserved_special_token_182|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128191: AddedToken("<|reserved_special_token_183|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128192: AddedToken("<|reserved_special_token_184|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128193: AddedToken("<|reserved_special_token_185|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128194: AddedToken("<|reserved_special_token_186|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128195: AddedToken("<|reserved_special_token_187|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128196: AddedToken("<|reserved_special_token_188|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128197: AddedToken("<|reserved_special_token_189|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128198: AddedToken("<|reserved_special_token_190|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128199: AddedToken("<|reserved_special_token_191|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128200: AddedToken("<|reserved_special_token_192|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128201: AddedToken("<|reserved_special_token_193|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128202: AddedToken("<|reserved_special_token_194|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128203: AddedToken("<|reserved_special_token_195|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128204: AddedToken("<|reserved_special_token_196|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128205: AddedToken("<|reserved_special_token_197|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128206: AddedToken("<|reserved_special_token_198|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128207: AddedToken("<|reserved_special_token_199|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128208: AddedToken("<|reserved_special_token_200|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128209: AddedToken("<|reserved_special_token_201|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128210: AddedToken("<|reserved_special_token_202|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128211: AddedToken("<|reserved_special_token_203|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128212: AddedToken("<|reserved_special_token_204|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128213: AddedToken("<|reserved_special_token_205|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128214: AddedToken("<|reserved_special_token_206|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128215: AddedToken("<|reserved_special_token_207|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128216: AddedToken("<|reserved_special_token_208|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128217: AddedToken("<|reserved_special_token_209|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128218: AddedToken("<|reserved_special_token_210|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128219: AddedToken("<|reserved_special_token_211|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128220: AddedToken("<|reserved_special_token_212|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128221: AddedToken("<|reserved_special_token_213|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128222: AddedToken("<|reserved_special_token_214|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128223: AddedToken("<|reserved_special_token_215|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128224: AddedToken("<|reserved_special_token_216|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128225: AddedToken("<|reserved_special_token_217|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128226: AddedToken("<|reserved_special_token_218|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128227: AddedToken("<|reserved_special_token_219|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128228: AddedToken("<|reserved_special_token_220|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128229: AddedToken("<|reserved_special_token_221|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128230: AddedToken("<|reserved_special_token_222|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128231: AddedToken("<|reserved_special_token_223|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128232: AddedToken("<|reserved_special_token_224|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128233: AddedToken("<|reserved_special_token_225|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128234: AddedToken("<|reserved_special_token_226|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True)lse, normalized=False, special=True), 	128113: AddedToken("<|reserved_special_token_105|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128114: AddedToken("<|reserved_special_token_106|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128115: AddedToken("<|reserved_special_token_107|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128116: AddedToken("<|reserved_special_token_108|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128117: AddedToken("<|reserved_special_token_109|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128118: AddedToken("<|reserved_special_token_110|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128119: AddedToken("<|reserved_special_token_111|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128120: AddedToken("<|reserved_special_token_112|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128121: AddedToken("<|reserved_special_token_113|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128122: AddedToken("<|reserved_special_token_114|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128123: AddedToken("<|reserved_special_token_115|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128124: AddedToken("<|reserved_special_token_116|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128125: AddedToken("<|reserved_special_token_117|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128126: AddedToken("<|reserved_special_token_118|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128127: AddedToken("<|reserved_special_token_119|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128128: AddedToken("<|reserved_special_token_120|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128129: AddedToken("<|reserved_special_token_121|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128130: AddedToken("<|reserved_special_token_122|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128131: AddedToken("<|reserved_special_token_123|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128132: AddedToken("<|reserved_special_token_124|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128133: AddedToken("<|reserved_special_token_125|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128134: AddedToken("<|reserved_special_token_126|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128135: AddedToken("<|reserved_special_token_127|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128136: AddedToken("<|reserved_special_token_128|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128137: AddedToken("<|reserved_special_token_129|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128138: AddedToken("<|reserved_special_token_130|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128139: AddedToken("<|reserved_special_token_131|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128140: AddedToken("<|reserved_special_token_132|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128141: AddedToken("<|reserved_special_token_133|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128142: AddedToken("<|reserved_special_token_134|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128143: AddedToken("<|reserved_special_token_135|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128144: AddedToken("<|reserved_special_token_136|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128145: AddedToken("<|reserved_special_token_137|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128146: AddedToken("<|reserved_special_token_138|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128147: AddedToken("<|reserved_special_token_139|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128148: AddedToken("<|reserved_special_token_140|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128149: AddedToken("<|reserved_special_token_141|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128150: AddedToken("<|reserved_special_token_142|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128151: AddedToken("<|reserved_special_token_143|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128152: AddedToken("<|reserved_special_token_144|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128153: AddedToken("<|reserved_special_token_145|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128154: AddedToken("<|reserved_special_token_146|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128155: AddedToken("<|reserved_special_token_147|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128156: AddedToken("<|reserved_special_token_148|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128157: AddedToken("<|reserved_special_token_149|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128158: AddedToken("<|reserved_special_token_150|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128159: AddedToken("<|reserved_special_token_151|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128160: AddedToken("<|reserved_special_token_152|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128161: AddedToken("<|reserved_special_token_153|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128162: AddedToken("<|reserved_special_token_154|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128163: AddedToken("<|reserved_special_token_155|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128164: AddedToken("<|reserved_special_token_156|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128165: AddedToken("<|reserved_special_token_157|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128166: AddedToken("<|reserved_special_token_158|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128167: AddedToken("<|reserved_special_token_159|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128168: AddedToken("<|reserved_special_token_160|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128169: AddedToken("<|reserved_special_token_161|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128170: AddedToken("<|reserved_special_token_162|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128171: AddedToken("<|reserved_special_token_163|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128172: AddedToken("<|reserved_special_token_164|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128173: AddedToken("<|reserved_special_token_165|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128174: AddedToken("<|reserved_special_token_166|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128175: AddedToken("<|reserved_special_token_167|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128176: AddedToken("<|reserved_special_token_168|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128177: AddedToken("<|reserved_special_token_169|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128178: AddedToken("<|reserved_special_token_170|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128179: AddedToken("<|reserved_special_token_171|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128180: AddedToken("<|reserved_special_token_172|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128181: AddedToken("<|reserved_special_token_173|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128182: AddedToken("<|reserved_special_token_174|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128183: AddedToken("<|reserved_special_token_175|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128184: AddedToken("<|reserved_special_token_176|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128185: AddedToken("<|reserved_special_token_177|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128186: AddedToken("<|reserved_special_token_178|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128187: AddedToken("<|reserved_special_token_179|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128188: AddedToken("<|reserved_special_token_180|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128189: AddedToken("<|reserved_special_token_181|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128190: AddedToken("<|reserved_special_token_182|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128191: AddedToken("<|reserved_special_token_183|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128192: AddedToken("<|reserved_special_token_184|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128193: AddedToken("<|reserved_special_token_185|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128194: AddedToken("<|reserved_special_token_186|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128195: AddedToken("<|reserved_special_token_187|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128196: AddedToken("<|reserved_special_token_188|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128197: AddedToken("<|reserved_special_token_189|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128198: AddedToken("<|reserved_special_token_190|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128199: AddedToken("<|reserved_special_token_191|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128200: AddedToken("<|reserved_special_token_192|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128201: AddedToken("<|reserved_special_token_193|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128202: AddedToken("<|reserved_special_token_194|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128203: AddedToken("<|reserved_special_token_195|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128204: AddedToken("<|reserved_special_token_196|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128205: AddedToken("<|reserved_special_token_197|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128206: AddedToken("<|reserved_special_token_198|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128207: AddedToken("<|reserved_special_token_199|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128208: AddedToken("<|reserved_special_token_200|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128209: AddedToken("<|reserved_special_token_201|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128210: AddedToken("<|reserved_special_token_202|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128211: AddedToken("<|reserved_special_token_203|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128212: AddedToken("<|reserved_special_token_204|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128213: AddedToken("<|reserved_special_token_205|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128214: AddedToken("<|reserved_special_token_206|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128215: AddedToken("<|reserved_special_token_207|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128216: AddedToken("<|reserved_special_token_208|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128217: AddedToken("<|reserved_special_token_209|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128218: AddedToken("<|reserved_special_token_210|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128219: AddedToken("<|reserved_special_token_211|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128220: AddedToken("<|reserved_special_token_212|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128221: AddedToken("<|reserved_special_token_213|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128222: AddedToken("<|reserved_special_token_214|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128223: AddedToken("<|reserved_special_token_215|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128224: AddedToken("<|reserved_special_token_216|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128225: AddedToken("<|reserved_special_token_217|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128226: AddedToken("<|reserved_special_token_218|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128227: AddedToken("<|reserved_special_token_219|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128228: AddedToken("<|reserved_special_token_220|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128229: AddedToken("<|reserved_special_token_221|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128230: AddedToken("<|reserved_special_token_222|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128231: AddedToken("<|reserved_special_token_223|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128232: AddedToken("<|reserved_special_token_224|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128233: AddedToken("<|reserved_special_token_225|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128234: AddedToken("<|reserved_special_token_226|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True)lse, normalized=False, special=True), 	128113: AddedToken("<|reserved_special_token_105|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128114: AddedToken("<|reserved_special_token_106|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128115: AddedToken("<|reserved_special_token_107|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128116: AddedToken("<|reserved_special_token_108|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128117: AddedToken("<|reserved_special_token_109|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128118: AddedToken("<|reserved_special_token_110|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128119: AddedToken("<|reserved_special_token_111|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128120: AddedToken("<|reserved_special_token_112|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128121: AddedToken("<|reserved_special_token_113|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128122: AddedToken("<|reserved_special_token_114|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128123: AddedToken("<|reserved_special_token_115|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128124: AddedToken("<|reserved_special_token_116|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128125: AddedToken("<|reserved_special_token_117|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128126: AddedToken("<|reserved_special_token_118|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128127: AddedToken("<|reserved_special_token_119|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128128: AddedToken("<|reserved_special_token_120|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128129: AddedToken("<|reserved_special_token_121|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128130: AddedToken("<|reserved_special_token_122|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128131: AddedToken("<|reserved_special_token_123|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128132: AddedToken("<|reserved_special_token_124|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128133: AddedToken("<|reserved_special_token_125|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128134: AddedToken("<|reserved_special_token_126|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128135: AddedToken("<|reserved_special_token_127|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128136: AddedToken("<|reserved_special_token_128|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128137: AddedToken("<|reserved_special_token_129|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128138: AddedToken("<|reserved_special_token_130|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128139: AddedToken("<|reserved_special_token_131|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128140: AddedToken("<|reserved_special_token_132|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128141: AddedToken("<|reserved_special_token_133|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128142: AddedToken("<|reserved_special_token_134|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128143: AddedToken("<|reserved_special_token_135|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128144: AddedToken("<|reserved_special_token_136|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128145: AddedToken("<|reserved_special_token_137|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128146: AddedToken("<|reserved_special_token_138|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128147: AddedToken("<|reserved_special_token_139|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128148: AddedToken("<|reserved_special_token_140|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128149: AddedToken("<|reserved_special_token_141|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128150: AddedToken("<|reserved_special_token_142|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128151: AddedToken("<|reserved_special_token_143|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128152: AddedToken("<|reserved_special_token_144|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128153: AddedToken("<|reserved_special_token_145|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128154: AddedToken("<|reserved_special_token_146|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128155: AddedToken("<|reserved_special_token_147|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128156: AddedToken("<|reserved_special_token_148|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128157: AddedToken("<|reserved_special_token_149|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128158: AddedToken("<|reserved_special_token_150|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128159: AddedToken("<|reserved_special_token_151|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128160: AddedToken("<|reserved_special_token_152|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128161: AddedToken("<|reserved_special_token_153|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128162: AddedToken("<|reserved_special_token_154|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128163: AddedToken("<|reserved_special_token_155|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128164: AddedToken("<|reserved_special_token_156|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128165: AddedToken("<|reserved_special_token_157|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128166: AddedToken("<|reserved_special_token_158|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128167: AddedToken("<|reserved_special_token_159|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128168: AddedToken("<|reserved_special_token_160|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128169: AddedToken("<|reserved_special_token_161|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128170: AddedToken("<|reserved_special_token_162|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128171: AddedToken("<|reserved_special_token_163|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128172: AddedToken("<|reserved_special_token_164|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128173: AddedToken("<|reserved_special_token_165|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128174: AddedToken("<|reserved_special_token_166|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128175: AddedToken("<|reserved_special_token_167|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128176: AddedToken("<|reserved_special_token_168|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128177: AddedToken("<|reserved_special_token_169|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128178: AddedToken("<|reserved_special_token_170|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128179: AddedToken("<|reserved_special_token_171|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128180: AddedToken("<|reserved_special_token_172|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128181: AddedToken("<|reserved_special_token_173|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128182: AddedToken("<|reserved_special_token_174|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128183: AddedToken("<|reserved_special_token_175|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128184: AddedToken("<|reserved_special_token_176|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128185: AddedToken("<|reserved_special_token_177|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128186: AddedToken("<|reserved_special_token_178|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128187: AddedToken("<|reserved_special_token_179|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128188: AddedToken("<|reserved_special_token_180|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128189: AddedToken("<|reserved_special_token_181|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128190: AddedToken("<|reserved_special_token_182|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128191: AddedToken("<|reserved_special_token_183|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128192: AddedToken("<|reserved_special_token_184|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128193: AddedToken("<|reserved_special_token_185|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128194: AddedToken("<|reserved_special_token_186|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128195: AddedToken("<|reserved_special_token_187|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128196: AddedToken("<|reserved_special_token_188|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128197: AddedToken("<|reserved_special_token_189|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128198: AddedToken("<|reserved_special_token_190|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128199: AddedToken("<|reserved_special_token_191|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128200: AddedToken("<|reserved_special_token_192|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128201: AddedToken("<|reserved_special_token_193|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128202: AddedToken("<|reserved_special_token_194|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128203: AddedToken("<|reserved_special_token_195|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128204: AddedToken("<|reserved_special_token_196|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128205: AddedToken("<|reserved_special_token_197|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128206: AddedToken("<|reserved_special_token_198|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128207: AddedToken("<|reserved_special_token_199|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128208: AddedToken("<|reserved_special_token_200|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128209: AddedToken("<|reserved_special_token_201|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128210: AddedToken("<|reserved_special_token_202|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128211: AddedToken("<|reserved_special_token_203|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128212: AddedToken("<|reserved_special_token_204|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128213: AddedToken("<|reserved_special_token_205|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128214: AddedToken("<|reserved_special_token_206|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128215: AddedToken("<|reserved_special_token_207|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128216: AddedToken("<|reserved_special_token_208|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128217: AddedToken("<|reserved_special_token_209|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128218: AddedToken("<|reserved_special_token_210|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128219: AddedToken("<|reserved_special_token_211|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128220: AddedToken("<|reserved_special_token_212|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128221: AddedToken("<|reserved_special_token_213|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128222: AddedToken("<|reserved_special_token_214|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128223: AddedToken("<|reserved_special_token_215|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128224: AddedToken("<|reserved_special_token_216|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128225: AddedToken("<|reserved_special_token_217|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128226: AddedToken("<|reserved_special_token_218|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128227: AddedToken("<|reserved_special_token_219|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128228: AddedToken("<|reserved_special_token_220|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128229: AddedToken("<|reserved_special_token_221|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128230: AddedToken("<|reserved_special_token_222|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128231: AddedToken("<|reserved_special_token_223|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128232: AddedToken("<|reserved_special_token_224|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128233: AddedToken("<|reserved_special_token_225|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128234: AddedToken("<|reserved_special_token_226|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128235: AddedToken("<|reserved_special_token_227|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128236: AddedToken("<|reserved_special_token_228|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128237: AddedToken("<|reserved_special_token_229|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128238: AddedToken("<|reserved_special_token_230|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128239: AddedToken("<|reserved_special_token_231|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128240: AddedToken("<|reserved_special_token_232|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128241: AddedToken("<|reserved_special_token_233|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128242: AddedToken("<|reserved_special_token_234|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128243: AddedToken("<|reserved_special_token_235|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128244: AddedToken("<|reserved_special_token_236|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128245: AddedToken("<|reserved_special_token_237|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128246: AddedToken("<|reserved_special_token_238|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128247: AddedToken("<|reserved_special_token_239|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128248: AddedToken("<|reserved_special_token_240|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128249: AddedToken("<|reserved_special_token_241|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128250: AddedToken("<|reserved_special_token_242|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128251: AddedToken("<|reserved_special_token_243|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128252: AddedToken("<|reserved_special_token_244|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128253: AddedToken("<|reserved_special_token_245|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128254: AddedToken("<|reserved_special_token_246|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128255: AddedToken("<|reserved_special_token_247|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128256: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128257: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), })lse, special=True), 	128174: AddedToken("<|reserved_special_token_166|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128175: AddedToken("<|reserved_special_token_167|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128176: AddedToken("<|reserved_special_token_168|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128177: AddedToken("<|reserved_special_token_169|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128178: AddedToken("<|reserved_special_token_170|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128179: AddedToken("<|reserved_special_token_171|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128180: AddedToken("<|reserved_special_token_172|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128181: AddedToken("<|reserved_special_token_173|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128182: AddedToken("<|reserved_special_token_174|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128183: AddedToken("<|reserved_special_token_175|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128184: AddedToken("<|reserved_special_token_176|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128185: AddedToken("<|reserved_special_token_177|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128186: AddedToken("<|reserved_special_token_178|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128187: AddedToken("<|reserved_special_token_179|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128188: AddedToken("<|reserved_special_token_180|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128189: AddedToken("<|reserved_special_token_181|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128190: AddedToken("<|reserved_special_token_182|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128191: AddedToken("<|reserved_special_token_183|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128192: AddedToken("<|reserved_special_token_184|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128193: AddedToken("<|reserved_special_token_185|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128194: AddedToken("<|reserved_special_token_186|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128195: AddedToken("<|reserved_special_token_187|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128196: AddedToken("<|reserved_special_token_188|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128197: AddedToken("<|reserved_special_token_189|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128198: AddedToken("<|reserved_special_token_190|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128199: AddedToken("<|reserved_special_token_191|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128200: AddedToken("<|reserved_special_token_192|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128201: AddedToken("<|reserved_special_token_193|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128202: AddedToken("<|reserved_special_token_194|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128203: AddedToken("<|reserved_special_token_195|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128204: AddedToken("<|reserved_special_token_196|>", rs, 	128235: AddedToken("<|reserved_special_token_227|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128236: AddedToken("<|reserved_special_token_228|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128237: AddedToken("<|reserved_special_token_229|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128238: AddedToken("<|reserved_special_token_230|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128239: AddedToken("<|reserved_special_token_231|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128240: AddedToken("<|reserved_special_token_232|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128241: AddedToken("<|reserved_special_token_233|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128242: AddedToken("<|reserved_special_token_234|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128243: AddedToken("<|reserved_special_token_235|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128244: AddedToken("<|reserved_special_token_236|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128245: AddedToken("<|reserved_special_token_237|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128246: AddedToken("<|reserved_special_token_238|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128247: AddedToken("<|reserved_special_token_239|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128248: AddedToken("<|reserved_special_token_240|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128249: AddedToken("<|reserved_special_token_241|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128250: AddedToken("<|reserved_special_token_242|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128251: AddedToken("<|reserved_special_token_243|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128252: AddedToken("<|reserved_special_token_244|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128253: AddedToken("<|reserved_special_token_245|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128254: AddedToken("<|reserved_special_token_246|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128255: AddedToken("<|reserved_special_token_247|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128256: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128257: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), })lse, special=True), 	128174: AddedToken("<|reserved_special_token_166|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128175: AddedToken("<|reserved_special_token_167|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128176: AddedToken("<|reserved_special_token_168|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128177: AddedToken("<|reserved_special_token_169|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128178: AddedToken("<|reserved_special_token_170|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128179: AddedToken("<|reserved_special_token_171|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128180: AddedToken("<|reserved_special_token_172|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128181: AddedToken("<|reserved_special_token_173|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128182: AddedToken("<|reserved_special_token_174|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128183: AddedToken("<|reserved_special_token_175|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128184: AddedToken("<|reserved_special_token_176|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128185: AddedToken("<|reserved_special_token_177|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128186: AddedToken("<|reserved_special_token_178|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128187: AddedToken("<|reserved_special_token_179|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128188: AddedToken("<|reserved_special_token_180|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128189: AddedToken("<|reserved_special_token_181|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128190: AddedToken("<|reserved_special_token_182|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128191: AddedToken("<|reserved_special_token_183|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128192: AddedToken("<|reserved_special_token_184|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128193: AddedToken("<|reserved_special_token_185|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128194: AddedToken("<|reserved_special_token_186|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128195: AddedToken("<|reserved_special_token_187|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128196: AddedToken("<|reserved_special_token_188|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128197: AddedToken("<|reserved_special_token_189|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128198: AddedToken("<|reserved_special_token_190|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128199: AddedToken("<|reserved_special_token_191|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128200: AddedToken("<|reserved_special_token_192|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128201: AddedToken("<|reserved_special_token_193|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128202: AddedToken("<|reserved_special_token_194|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128203: AddedToken("<|reserved_special_token_195|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128204: AddedToken("<|reserved_special_token_196|>", rs, 	128235: AddedToken("<|reserved_special_token_227|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128236: AddedToken("<|reserved_special_token_228|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128237: AddedToken("<|reserved_special_token_229|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128238: AddedToken("<|reserved_special_token_230|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128239: AddedToken("<|reserved_special_token_231|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128240: AddedToken("<|reserved_special_token_232|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128241: AddedToken("<|reserved_special_token_233|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128242: AddedToken("<|reserved_special_token_234|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128243: AddedToken("<|reserved_special_token_235|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128244: AddedToken("<|reserved_special_token_236|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128245: AddedToken("<|reserved_special_token_237|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128246: AddedToken("<|reserved_special_token_238|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128247: AddedToken("<|reserved_special_token_239|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128248: AddedToken("<|reserved_special_token_240|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128249: AddedToken("<|reserved_special_token_241|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128250: AddedToken("<|reserved_special_token_242|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128251: AddedToken("<|reserved_special_token_243|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128252: AddedToken("<|reserved_special_token_244|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128253: AddedToken("<|reserved_special_token_245|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128254: AddedToken("<|reserved_special_token_246|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128255: AddedToken("<|reserved_special_token_247|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128256: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128257: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), })trip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128205: AddedToken("<|reserved_special_token_197|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128206: AddedToken("<|reserved_special_token_198|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128207: AddedToken("<|reserved_special_token_199|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128208: AddedToken("<|reserved_special_token_200|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128209: AddedToken("<|reserved_special_token_201|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128210: AddedToken("<|reserved_special_token_202|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128211: AddedToken("<|reserved_special_token_203|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128212: AddedToken("<|reserved_special_token_204|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128213: AddedToken("<|reserved_special_token_205|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128214: AddedToken("<|reserved_special_token_206|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128215: AddedToken("<|reserved_special_token_207|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128216: AddedToken("<|reserved_special_token_208|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128217: AddedToken("<|reserved_special_token_209|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128218: AddedToken("<|reserved_special_token_210|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128219: AddedToken("<|reserved_special_token_211|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128220: AddedToken("<|reserved_special_token_212|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128221: AddedToken("<|reserved_special_token_213|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128222: AddedToken("<|reserved_special_token_214|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128223: AddedToken("<|reserved_special_token_215|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128224: AddedToken("<|reserved_special_token_216|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128225: AddedToken("<|reserved_special_token_217|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128226: AddedToken("<|reserved_special_token_218|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128227: AddedToken("<|reserved_special_token_219|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128228: AddedToken("<|reserved_special_token_220|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128229: AddedToken("<|reserved_special_token_221|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128230: AddedToken("<|reserved_special_token_222|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128231: AddedToken("<|reserved_special_token_223|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128232: AddedToken("<|reserved_special_token_224|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128233: AddedToken("<|reserved_special_token_225|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128234: AddedToken("<|reserved_special_token_226|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True)
trip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128205: AddedToken("<|reserved_special_token_197|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128206: AddedToken("<|reserved_special_token_198|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128207: AddedToken("<|reserved_special_token_199|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128208: AddedToken("<|reserved_special_token_200|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128209: AddedToken("<|reserved_special_token_201|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128210: AddedToken("<|reserved_special_token_202|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128211: AddedToken("<|reserved_special_token_203|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128212: AddedToken("<|reserved_special_token_204|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128213: AddedToken("<|reserved_special_token_205|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128214: AddedToken("<|reserved_special_token_206|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128215: AddedToken("<|reserved_special_token_207|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128216: AddedToken("<|reserved_special_token_208|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128217: AddedToken("<|reserved_special_token_209|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128218: AddedToken("<|reserved_special_token_210|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128219: AddedToken("<|reserved_special_token_211|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128220: AddedToken("<|reserved_special_token_212|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128221: AddedToken("<|reserved_special_token_213|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128222: AddedToken("<|reserved_special_token_214|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128223: AddedToken("<|reserved_special_token_215|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128224: AddedToken("<|reserved_special_token_216|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128225: AddedToken("<|reserved_special_token_217|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128226: AddedToken("<|reserved_special_token_218|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128227: AddedToken("<|reserved_special_token_219|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128228: AddedToken("<|reserved_special_token_220|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128229: AddedToken("<|reserved_special_token_221|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128230: AddedToken("<|reserved_special_token_222|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128231: AddedToken("<|reserved_special_token_223|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128232: AddedToken("<|reserved_special_token_224|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128233: AddedToken("<|reserved_special_token_225|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128234: AddedToken("<|reserved_special_token_226|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True)
, 	128235: AddedToken("<|reserved_special_token_227|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128236: AddedToken("<|reserved_special_token_228|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128237: AddedToken("<|reserved_special_token_229|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128238: AddedToken("<|reserved_special_token_230|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128239: AddedToken("<|reserved_special_token_231|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128240: AddedToken("<|reserved_special_token_232|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128241: AddedToken("<|reserved_special_token_233|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128242: AddedToken("<|reserved_special_token_234|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128243: AddedToken("<|reserved_special_token_235|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128244: AddedToken("<|reserved_special_token_236|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128245: AddedToken("<|reserved_special_token_237|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128246: AddedToken("<|reserved_special_token_238|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128247: AddedToken("<|reserved_special_token_239|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128248: AddedToken("<|reserved_special_token_240|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128249: AddedToken("<|reserved_special_token_241|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128250: AddedToken("<|reserved_special_token_242|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128251: AddedToken("<|reserved_special_token_243|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128252: AddedToken("<|reserved_special_token_244|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128253: AddedToken("<|reserved_special_token_245|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128254: AddedToken("<|reserved_special_token_246|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128255: AddedToken("<|reserved_special_token_247|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128256: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128257: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), })

, 	128235: AddedToken("<|reserved_special_token_227|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128236: AddedToken("<|reserved_special_token_228|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128237: AddedToken("<|reserved_special_token_229|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128238: AddedToken("<|reserved_special_token_230|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128239: AddedToken("<|reserved_special_token_231|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128240: AddedToken("<|reserved_special_token_232|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128241: AddedToken("<|reserved_special_token_233|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128242: AddedToken("<|reserved_special_token_234|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128243: AddedToken("<|reserved_special_token_235|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128244: AddedToken("<|reserved_special_token_236|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128245: AddedToken("<|reserved_special_token_237|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128246: AddedToken("<|reserved_special_token_238|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128247: AddedToken("<|reserved_special_token_239|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128248: AddedToken("<|reserved_special_token_240|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128249: AddedToken("<|reserved_special_token_241|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128250: AddedToken("<|reserved_special_token_242|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128251: AddedToken("<|reserved_special_token_243|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128252: AddedToken("<|reserved_special_token_244|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128253: AddedToken("<|reserved_special_token_245|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128254: AddedToken("<|reserved_special_token_246|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128255: AddedToken("<|reserved_special_token_247|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128256: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128257: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), })
(LlamaForCausalLM(   (model): LlamaModel(     (embed_tokens): Embedding(128258, 4096)     (layers): ModuleList(       (0-31): 32 x LlamaDecoderLayer(         (self_attn): LlamaFlashAttention2(           (q_proj): Linear(in_features=4096, out_features=4096, bias=False)           (k_proj): Linear(in_features=4096, out_features=1024, bias=False)           (v_proj): Linear(in_features=4096, out_features=1024, bias=False)           (o_proj): Linear(in_features=4096, out_features=4096, bias=False)           (rotary_emb): LlamaRotaryEmbedding()         )         (mlp): LlamaMLP(           (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)           (up_proj): Linear(in_features=4096, out_features=14336, bias=False)           (down_proj): Linear(in_features=14336, out_features=4096, bias=False)           (act_fn): SiLU()         )         (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)         (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)       )     )     (norm): LlamaRMSNorm((4096,), eps=1e-05)     (rotary_emb): LlamaRotaryEmbedding()   )   (lm_head): Linear(in_features=4096, out_features=128258, bias=False) ), PreTrainedTokenizerFast(name_or_path='meta-llama/Meta-Llama-3.1-8B', vocab_size=128000, model_max_length=131072, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<|begin_of_text|>', 'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>'}, clean_up_tokenization_spaces=True),  added_tokens_decoder={ 	128000: AddedToken("<|begin_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128001: AddedToken("<|end_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128002: AddedToken("<|reserved_special_token_0|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128003: AddedToken("<|reserved_special_token_1|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128004: AddedToken("<|finetune_right_pad_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128005: AddedToken("<|reserved_special_token_2|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128006: AddedToken("<|start_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128007: AddedToken("<|end_header_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128008: AddedToken("<|eom_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128009: AddedToken("<|eot_id|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128010: AddedToken("<|python_tag|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128011: AddedToken("<|reserved_special_token_3|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128012: AddedToken("<|reserved_special_token_4|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128013: AddedToken("<|reserved_special_token_5|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128014: AddedToken("<|reserved_special_token_6|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128015: AddedToken("<|reserved_special_token_7|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128016: AddedToken("<|reserved_special_token_8|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128017: AddedToken("<|reserved_special_token_9|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128018: AddedToken("<|reserved_special_token_10|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128019: AddedToken("<|reserved_special_token_11|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128020: AddedToken("<|reserved_special_token_12|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128021: AddedToken("<|reserved_special_token_13|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128022: AddedToken("<|reserved_special_token_14|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128023: AddedToken("<|reserved_special_token_15|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128024: AddedToken("<|reserved_special_token_16|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128025: AddedToken("<|reserved_special_token_17|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128026: AddedToken("<|reserved_special_token_18|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128027: AddedToken("<|reserved_special_token_19|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128028: AddedToken("<|reserved_special_token_20|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128029: AddedToken("<|reserved_special_token_21|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128030: AddedToken("<|reserved_special_token_22|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128031: AddedToken("<|reserved_special_token_23|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128032: AddedToken("<|reserved_special_token_24|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128033: AddedToken("<|reserved_special_token_25|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128034: AddedToken("<|reserved_special_token_26|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128035: AddedToken("<|reserved_special_token_27|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128036: AddedToken("<|reserved_special_token_28|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128037: AddedToken("<|reserved_special_token_29|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128038: AddedToken("<|reserved_special_token_30|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128039: AddedToken("<|reserved_special_token_31|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128040: AddedToken("<|reserved_special_token_32|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128041: AddedToken("<|reserved_special_token_33|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128042: AddedToken("<|reserved_special_token_34|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128043: AddedToken("<|reserved_special_token_35|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128044: AddedToken("<|reserved_special_token_36|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128045: AddedToken("<|reserved_special_token_37|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128046: AddedToken("<|reserved_special_token_38|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128047: AddedToken("<|reserved_special_token_39|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128048: AddedToken("<|reserved_special_token_40|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128049: AddedToken("<|reserved_special_token_41|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128050: AddedToken("<|reserved_special_token_42|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128051: AddedToken("<|reserved_special_token_43|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128052: AddedToken("<|reserved_special_token_44|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128053: AddedToken("<|reserved_special_token_45|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128054: AddedToken("<|reserved_special_token_46|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128055: AddedToken("<|reserved_special_token_47|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128056: AddedToken("<|reserved_special_token_48|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128057: AddedToken("<|reserved_special_token_49|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128058: AddedToken("<|reserved_special_token_50|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128059: AddedToken("<|reserved_special_token_51|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128060: AddedToken("<|reserved_special_token_52|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128061: AddedToken("<|reserved_special_token_53|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128062: AddedToken("<|reserved_special_token_54|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128063: AddedToken("<|reserved_special_token_55|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128064: AddedToken("<|reserved_special_token_56|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128065: AddedToken("<|reserved_special_token_57|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128066: AddedToken("<|reserved_special_token_58|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128067: AddedToken("<|reserved_special_token_59|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128068: AddedToken("<|reserved_special_token_60|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128069: AddedToken("<|reserved_special_token_61|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128070: AddedToken("<|reserved_special_token_62|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128071: AddedToken("<|reserved_special_token_63|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128072: AddedToken("<|reserved_special_token_64|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128073: AddedToken("<|reserved_special_token_65|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128074: AddedToken("<|reserved_special_token_66|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128075: AddedToken("<|reserved_special_token_67|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128076: AddedToken("<|reserved_special_token_68|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128077: AddedToken("<|reserved_special_token_69|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128078: AddedToken("<|reserved_special_token_70|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128079: AddedToken("<|reserved_special_token_71|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128080: AddedToken("<|reserved_special_token_72|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128081: AddedToken("<|reserved_special_token_73|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128082: AddedToken("<|reserved_special_token_74|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128083: AddedToken("<|reserved_special_token_75|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128084: AddedToken("<|reserved_special_token_76|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128085: AddedToken("<|reserved_special_token_77|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128086: AddedToken("<|reserved_special_token_78|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128087: AddedToken("<|reserved_special_token_79|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128088: AddedToken("<|reserved_special_token_80|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128089: AddedToken("<|reserved_special_token_81|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128090: AddedToken("<|reserved_special_token_82|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128091: AddedToken("<|reserved_special_token_83|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128092: AddedToken("<|reserved_special_token_84|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128093: AddedToken("<|reserved_special_token_85|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128094: AddedToken("<|reserved_special_token_86|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128095: AddedToken("<|reserved_special_token_87|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128096: AddedToken("<|reserved_special_token_88|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128097: AddedToken("<|reserved_special_token_89|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128098: AddedToken("<|reserved_special_token_90|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128099: AddedToken("<|reserved_special_token_91|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128100: AddedToken("<|reserved_special_token_92|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128101: AddedToken("<|reserved_special_token_93|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128102: AddedToken("<|reserved_special_token_94|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128103: AddedToken("<|reserved_special_token_95|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128104: AddedToken("<|reserved_special_token_96|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128105: AddedToken("<|reserved_special_token_97|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128106: AddedToken("<|reserved_special_token_98|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128107: AddedToken("<|reserved_special_token_99|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128108: AddedToken("<|reserved_special_token_100|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128109: AddedToken("<|reserved_special_token_101|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128110: AddedToken("<|reserved_special_token_102|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128111: AddedToken("<|reserved_special_token_103|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128112: AddedToken("<|reserved_special_token_104|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128113: AddedToken("<|reserved_special_token_105|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128114: AddedToken("<|reserved_special_token_106|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128115: AddedToken("<|reserved_special_token_107|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128116: AddedToken("<|reserved_special_token_108|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128117: AddedToken("<|reserved_special_token_109|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128118: AddedToken("<|reserved_special_token_110|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128119: AddedToken("<|reserved_special_token_111|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128120: AddedToken("<|reserved_special_token_112|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128121: AddedToken("<|reserved_special_token_113|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128122: AddedToken("<|reserved_special_token_114|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128123: AddedToken("<|reserved_special_token_115|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128124: AddedToken("<|reserved_special_token_116|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128125: AddedToken("<|reserved_special_token_117|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128126: AddedToken("<|reserved_special_token_118|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128127: AddedToken("<|reserved_special_token_119|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128128: AddedToken("<|reserved_special_token_120|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128129: AddedToken("<|reserved_special_token_121|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128130: AddedToken("<|reserved_special_token_122|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128131: AddedToken("<|reserved_special_token_123|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128132: AddedToken("<|reserved_special_token_124|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128133: AddedToken("<|reserved_special_token_125|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128134: AddedToken("<|reserved_special_token_126|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128135: AddedToken("<|reserved_special_token_127|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128136: AddedToken("<|reserved_special_token_128|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128137: AddedToken("<|reserved_special_token_129|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128138: AddedToken("<|reserved_special_token_130|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128139: AddedToken("<|reserved_special_token_131|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128140: AddedToken("<|reserved_special_token_132|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128141: AddedToken("<|reserved_special_token_133|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128142: AddedToken("<|reserved_special_token_134|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128143: AddedToken("<|reserved_special_token_135|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128144: AddedToken("<|reserved_special_token_136|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128145: AddedToken("<|reserved_special_token_137|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128146: AddedToken("<|reserved_special_token_138|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128147: AddedToken("<|reserved_special_token_139|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128148: AddedToken("<|reserved_special_token_140|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128149: AddedToken("<|reserved_special_token_141|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128150: AddedToken("<|reserved_special_token_142|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128151: AddedToken("<|reserved_special_token_143|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128152: AddedToken("<|reserved_special_token_144|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128153: AddedToken("<|reserved_special_token_145|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128154: AddedToken("<|reserved_special_token_146|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128155: AddedToken("<|reserved_special_token_147|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128156: AddedToken("<|reserved_special_token_148|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128157: AddedToken("<|reserved_special_token_149|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128158: AddedToken("<|reserved_special_token_150|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128159: AddedToken("<|reserved_special_token_151|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128160: AddedToken("<|reserved_special_token_152|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128161: AddedToken("<|reserved_special_token_153|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128162: AddedToken("<|reserved_special_token_154|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128163: AddedToken("<|reserved_special_token_155|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128164: AddedToken("<|reserved_special_token_156|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128165: AddedToken("<|reserved_special_token_157|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128166: AddedToken("<|reserved_special_token_158|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128167: AddedToken("<|reserved_special_token_159|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128168: AddedToken("<|reserved_special_token_160|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128169: AddedToken("<|reserved_special_token_161|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128170: AddedToken("<|reserved_special_token_162|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128171: AddedToken("<|reserved_special_token_163|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128172: AddedToken("<|reserved_special_token_164|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128173: AddedToken("<|reserved_special_token_165|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128174: AddedToken("<|reserved_special_token_166|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128175: AddedToken("<|reserved_special_token_167|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128176: AddedToken("<|reserved_special_token_168|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128177: AddedToken("<|reserved_special_token_169|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128178: AddedToken("<|reserved_special_token_170|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128179: AddedToken("<|reserved_special_token_171|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128180: AddedToken("<|reserved_special_token_172|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128181: AddedToken("<|reserved_special_token_173|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128182: AddedToken("<|reserved_special_token_174|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128183: AddedToken("<|reserved_special_token_175|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128184: AddedToken("<|reserved_special_token_176|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128185: AddedToken("<|reserved_special_token_177|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128186: AddedToken("<|reserved_special_token_178|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128187: AddedToken("<|reserved_special_token_179|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128188: AddedToken("<|reserved_special_token_180|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128189: AddedToken("<|reserved_special_token_181|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128190: AddedToken("<|reserved_special_token_182|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128191: AddedToken("<|reserved_special_token_183|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128192: AddedToken("<|reserved_special_token_184|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128193: AddedToken("<|reserved_special_token_185|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128194: AddedToken("<|reserved_special_token_186|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128195: AddedToken("<|reserved_special_token_187|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128196: AddedToken("<|reserved_special_token_188|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128197: AddedToken("<|reserved_special_token_189|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128198: AddedToken("<|reserved_special_token_190|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128199: AddedToken("<|reserved_special_token_191|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128200: AddedToken("<|reserved_special_token_192|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128201: AddedToken("<|reserved_special_token_193|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128202: AddedToken("<|reserved_special_token_194|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128203: AddedToken("<|reserved_special_token_195|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128204: AddedToken("<|reserved_special_token_196|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128205: AddedToken("<|reserved_special_token_197|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128206: AddedToken("<|reserved_special_token_198|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128207: AddedToken("<|reserved_special_token_199|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128208: AddedToken("<|reserved_special_token_200|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128209: AddedToken("<|reserved_special_token_201|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128210: AddedToken("<|reserved_special_token_202|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128211: AddedToken("<|reserved_special_token_203|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128212: AddedToken("<|reserved_special_token_204|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128213: AddedToken("<|reserved_special_token_205|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128214: AddedToken("<|reserved_special_token_206|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128215: AddedToken("<|reserved_special_token_207|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128216: AddedToken("<|reserved_special_token_208|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128217: AddedToken("<|reserved_special_token_209|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128218: AddedToken("<|reserved_special_token_210|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128219: AddedToken("<|reserved_special_token_211|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128220: AddedToken("<|reserved_special_token_212|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128221: AddedToken("<|reserved_special_token_213|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128222: AddedToken("<|reserved_special_token_214|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128223: AddedToken("<|reserved_special_token_215|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128224: AddedToken("<|reserved_special_token_216|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128225: AddedToken("<|reserved_special_token_217|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128226: AddedToken("<|reserved_special_token_218|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128227: AddedToken("<|reserved_special_token_219|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128228: AddedToken("<|reserved_special_token_220|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128229: AddedToken("<|reserved_special_token_221|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128230: AddedToken("<|reserved_special_token_222|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128231: AddedToken("<|reserved_special_token_223|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128232: AddedToken("<|reserved_special_token_224|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128233: AddedToken("<|reserved_special_token_225|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128234: AddedToken("<|reserved_special_token_226|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128235: AddedToken("<|reserved_special_token_227|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128236: AddedToken("<|reserved_special_token_228|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128237: AddedToken("<|reserved_special_token_229|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128238: AddedToken("<|reserved_special_token_230|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128239: AddedToken("<|reserved_special_token_231|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128240: AddedToken("<|reserved_special_token_232|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128241: AddedToken("<|reserved_special_token_233|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128242: AddedToken("<|reserved_special_token_234|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128243: AddedToken("<|reserved_special_token_235|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128244: AddedToken("<|reserved_special_token_236|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128245: AddedToken("<|reserved_special_token_237|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128246: AddedToken("<|reserved_special_token_238|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128247: AddedToken("<|reserved_special_token_239|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128248: AddedToken("<|reserved_special_token_240|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128249: AddedToken("<|reserved_special_token_241|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128250: AddedToken("<|reserved_special_token_242|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128251: AddedToken("<|reserved_special_token_243|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128252: AddedToken("<|reserved_special_token_244|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128253: AddedToken("<|reserved_special_token_245|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128254: AddedToken("<|reserved_special_token_246|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128255: AddedToken("<|reserved_special_token_247|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128256: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 	128257: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), })
wandb: - 0.025 MB of 0.025 MB uploaded
wandb: \ 0.443 MB of 0.443 MB uploaded
wandb: 
wandb: Run history:
wandb:               eval/loss █▅▂▁
wandb:            eval/runtime █▅▅▁
wandb: eval/samples_per_second ▁▄▄█
wandb:   eval/steps_per_second ▁▄▄█
wandb:             train/epoch ▁▁▁▂▂▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇███
wandb:       train/global_step ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇███
wandb:         train/grad_norm █▄▃▂▃▃▅▃▆▄▅▂▂▂▁▅▄▃▂▁▂▂▂▂▂▃▃▃▂▂▂▁▂▃▂▁▂▁▁▂
wandb:     train/learning_rate ███▇▇▇▇▇▇▆▆▆▆▆▆▅▅▅▅▅▅▄▄▄▄▄▃▃▃▃▃▂▂▂▂▂▂▁▁▁
wandb:              train/loss ▆▆█▆▇▆▆▆▇▃▄▄▄▄▅▆▃▆▅▄▃▃▄▅▃▄▃▅▄▄▅▅▂▁▁▁▁▁▃▁
wandb: 
wandb: Run summary:
wandb:                eval/loss 0.49751
wandb:             eval/runtime 32.1892
wandb:  eval/samples_per_second 82.947
wandb:    eval/steps_per_second 5.188
wandb:               total_flos 559881199288320.0
wandb:              train/epoch 2.95196
wandb:        train/global_step 669
wandb:          train/grad_norm 0.47606
wandb:      train/learning_rate 0.0
wandb:               train/loss 0.4504
wandb:               train_loss 0.39771
wandb:            train_runtime 26579.6548
wandb: train_samples_per_second 29.834
wandb:   train_steps_per_second 0.025
wandb: 
wandb: 🚀 View run llama_3.1_8b_function_calling at: https://wandb.ai/therealagi-reviewramp/function-calling-spectrum/runs/svh2equ1
wandb: ⭐️ View project at: https://wandb.ai/therealagi-reviewramp/function-calling-spectrum
wandb: Synced 6 W&B file(s), 0 media file(s), 1 artifact file(s) and 1 other file(s)
wandb: Find logs at: ./wandb/run-20240802_033958-svh2equ1/logs
wandb: WARNING The new W&B backend becomes opt-out in version 0.18.0; try it out with `wandb.require("core")`! See https://wandb.me/wandb-core for more information.