Upload folder using huggingface_hub

README.md

@@ -0,0 +1,66 @@
+---
+license: llama2
+base_model: KoboldAI/LLaMA2-13B-TiefighterLR
+tags:
+- generated_from_trainer
+model-index:
+- name: lora-outC
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
+# lora-outC
+
+This model is a fine-tuned version of [KoboldAI/LLaMA2-13B-TiefighterLR](https://huggingface.co/KoboldAI/LLaMA2-13B-TiefighterLR) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 1.9844
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 1.5e-05
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- gradient_accumulation_steps: 5
+- total_train_batch_size: 40
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 10
+- num_epochs: 5
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| 1.8419        | 0.75  | 25   | 2.5257          |
+| 1.7748        | 1.5   | 50   | 2.2467          |
+| 1.813         | 2.25  | 75   | 2.0914          |
+| 1.8067        | 2.99  | 100  | 2.0235          |
+| 1.5346        | 3.74  | 125  | 1.9939          |
+| 1.5869        | 4.49  | 150  | 1.9844          |
+
+
+### Framework versions
+
+- Transformers 4.34.1
+- Pytorch 2.1.0+cu118
+- Datasets 2.14.6
+- Tokenizers 0.14.1

adapter_config.json

@@ -0,0 +1,28 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "KoboldAI/LLaMA2-13B-TiefighterLR",
+  "bias": "none",
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "gate_proj",
+    "q_proj",
+    "up_proj",
+    "o_proj",
+    "v_proj",
+    "down_proj"
+  ],
+  "task_type": "CAUSAL_LM"
+}

adapter_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:af84eb1ef393f0e792e09998b8c61789f659fbc8aaf8dd5367a0abc3ad6994a1
+size 2002982666

checkpoint-100/README.md

@@ -0,0 +1,219 @@
+---
+library_name: peft
+base_model: KoboldAI/LLaMA2-13B-TiefighterLR
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure 
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Data Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+
+
+## Training procedure
+
+
+The following `bitsandbytes` quantization config was used during training:
+- quant_method: bitsandbytes
+- load_in_8bit: True
+- load_in_4bit: False
+- llm_int8_threshold: 6.0
+- llm_int8_skip_modules: None
+- llm_int8_enable_fp32_cpu_offload: False
+- llm_int8_has_fp16_weight: False
+- bnb_4bit_quant_type: fp4
+- bnb_4bit_use_double_quant: False
+- bnb_4bit_compute_dtype: float32
+
+### Framework versions
+
+
+- PEFT 0.6.0

checkpoint-100/adapter_config.json

@@ -0,0 +1,28 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "KoboldAI/LLaMA2-13B-TiefighterLR",
+  "bias": "none",
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "gate_proj",
+    "q_proj",
+    "up_proj",
+    "o_proj",
+    "v_proj",
+    "down_proj"
+  ],
+  "task_type": "CAUSAL_LM"
+}

checkpoint-100/adapter_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:076c1ba8e906cad1fd9c5f494cbcbcfd9366e87ee2fe0c579dcf19fe01a192a0
+size 2002982666

checkpoint-100/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:053aac405e7993c1610e69212c961062f8012b152fe4fdad8566968395b08ef4
+size 4005907122

checkpoint-100/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23e56defc3c5796592aa967f64ef5da2f70447f5f0a0db865dd61700e02105e2
+size 14244

checkpoint-100/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:78bc57bd6db4182773017808792e92afe4a08b71615329b42a01985cad6c0ce9
+size 1064

checkpoint-100/trainer_state.json

@@ -0,0 +1,651 @@
+{
+  "best_metric": 2.023528575897217,
+  "best_model_checkpoint": "./lora-outC/checkpoint-100",
+  "epoch": 2.9940119760479043,
+  "eval_steps": 25,
+  "global_step": 100,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.03,
+      "learning_rate": 1.5e-06,
+      "loss": 1.8176,
+      "step": 1
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 3e-06,
+      "loss": 2.0942,
+      "step": 2
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.5e-06,
+      "loss": 3.1029,
+      "step": 3
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 6e-06,
+      "loss": 2.0346,
+      "step": 4
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 7.5e-06,
+      "loss": 2.1768,
+      "step": 5
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 9e-06,
+      "loss": 1.9869,
+      "step": 6
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 1.05e-05,
+      "loss": 2.7137,
+      "step": 7
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 1.2e-05,
+      "loss": 1.9197,
+      "step": 8
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 1.3500000000000001e-05,
+      "loss": 2.405,
+      "step": 9
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 1.5e-05,
+      "loss": 3.0781,
+      "step": 10
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 1.4998459531758846e-05,
+      "loss": 1.8125,
+      "step": 11
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 1.4993838759846692e-05,
+      "loss": 2.0828,
+      "step": 12
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 1.4986139582437507e-05,
+      "loss": 1.9523,
+      "step": 13
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 1.4975365162288163e-05,
+      "loss": 2.1143,
+      "step": 14
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 1.4961519925439214e-05,
+      "loss": 2.1367,
+      "step": 15
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 1.4944609559396697e-05,
+      "loss": 1.8429,
+      "step": 16
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 1.4924641010795769e-05,
+      "loss": 2.217,
+      "step": 17
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 1.4901622482547082e-05,
+      "loss": 1.7973,
+      "step": 18
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 1.4875563430467096e-05,
+      "loss": 2.1631,
+      "step": 19
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 1.484647455939371e-05,
+      "loss": 1.9283,
+      "step": 20
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 1.4814367818788806e-05,
+      "loss": 1.9236,
+      "step": 21
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 1.4779256397829512e-05,
+      "loss": 2.5061,
+      "step": 22
+    },
+    {
+      "epoch": 0.69,
+      "learning_rate": 1.4741154719990202e-05,
+      "loss": 2.2137,
+      "step": 23
+    },
+    {
+      "epoch": 0.72,
+      "learning_rate": 1.4700078437117444e-05,
+      "loss": 2.7564,
+      "step": 24
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 1.4656044423000367e-05,
+      "loss": 1.8419,
+      "step": 25
+    },
+    {
+      "epoch": 0.75,
+      "eval_loss": 2.5257418155670166,
+      "eval_runtime": 5.8091,
+      "eval_samples_per_second": 4.82,
+      "eval_steps_per_second": 0.689,
+      "step": 25
+    },
+    {
+      "epoch": 0.78,
+      "learning_rate": 1.4609070766439045e-05,
+      "loss": 2.3377,
+      "step": 26
+    },
+    {
+      "epoch": 0.81,
+      "learning_rate": 1.4559176763813775e-05,
+      "loss": 1.7976,
+      "step": 27
+    },
+    {
+      "epoch": 0.84,
+      "learning_rate": 1.450638291115828e-05,
+      "loss": 2.0391,
+      "step": 28
+    },
+    {
+      "epoch": 0.87,
+      "learning_rate": 1.4450710895740116e-05,
+      "loss": 1.9452,
+      "step": 29
+    },
+    {
+      "epoch": 0.9,
+      "learning_rate": 1.439218358715173e-05,
+      "loss": 1.8263,
+      "step": 30
+    },
+    {
+      "epoch": 0.93,
+      "learning_rate": 1.4330825027915823e-05,
+      "loss": 2.0638,
+      "step": 31
+    },
+    {
+      "epoch": 0.96,
+      "learning_rate": 1.4266660423608882e-05,
+      "loss": 1.8948,
+      "step": 32
+    },
+    {
+      "epoch": 0.99,
+      "learning_rate": 1.4199716132506944e-05,
+      "loss": 1.985,
+      "step": 33
+    },
+    {
+      "epoch": 1.02,
+      "learning_rate": 1.4130019654757838e-05,
+      "loss": 2.2594,
+      "step": 34
+    },
+    {
+      "epoch": 1.05,
+      "learning_rate": 1.4057599621084365e-05,
+      "loss": 2.2773,
+      "step": 35
+    },
+    {
+      "epoch": 1.08,
+      "learning_rate": 1.3982485781023034e-05,
+      "loss": 1.659,
+      "step": 36
+    },
+    {
+      "epoch": 1.11,
+      "learning_rate": 1.3904708990703192e-05,
+      "loss": 1.834,
+      "step": 37
+    },
+    {
+      "epoch": 1.14,
+      "learning_rate": 1.3824301200171617e-05,
+      "loss": 1.6516,
+      "step": 38
+    },
+    {
+      "epoch": 1.17,
+      "learning_rate": 1.374129544026768e-05,
+      "loss": 2.1114,
+      "step": 39
+    },
+    {
+      "epoch": 1.2,
+      "learning_rate": 1.3655725809054574e-05,
+      "loss": 1.6287,
+      "step": 40
+    },
+    {
+      "epoch": 1.23,
+      "learning_rate": 1.3567627457812107e-05,
+      "loss": 1.6146,
+      "step": 41
+    },
+    {
+      "epoch": 1.26,
+      "learning_rate": 1.3477036576596862e-05,
+      "loss": 2.2355,
+      "step": 42
+    },
+    {
+      "epoch": 1.29,
+      "learning_rate": 1.3383990379375632e-05,
+      "loss": 1.8845,
+      "step": 43
+    },
+    {
+      "epoch": 1.32,
+      "learning_rate": 1.3288527088738215e-05,
+      "loss": 2.0984,
+      "step": 44
+    },
+    {
+      "epoch": 1.35,
+      "learning_rate": 1.3190685920195931e-05,
+      "loss": 1.7909,
+      "step": 45
+    },
+    {
+      "epoch": 1.38,
+      "learning_rate": 1.3090507066072199e-05,
+      "loss": 1.9248,
+      "step": 46
+    },
+    {
+      "epoch": 1.41,
+      "learning_rate": 1.2988031678991868e-05,
+      "loss": 1.651,
+      "step": 47
+    },
+    {
+      "epoch": 1.44,
+      "learning_rate": 1.2883301854976092e-05,
+      "loss": 1.8998,
+      "step": 48
+    },
+    {
+      "epoch": 1.47,
+      "learning_rate": 1.277636061614961e-05,
+      "loss": 2.0732,
+      "step": 49
+    },
+    {
+      "epoch": 1.5,
+      "learning_rate": 1.266725189306765e-05,
+      "loss": 1.7748,
+      "step": 50
+    },
+    {
+      "epoch": 1.5,
+      "eval_loss": 2.2467305660247803,
+      "eval_runtime": 5.8123,
+      "eval_samples_per_second": 4.817,
+      "eval_steps_per_second": 0.688,
+      "step": 50
+    },
+    {
+      "epoch": 1.53,
+      "learning_rate": 1.2556020506669606e-05,
+      "loss": 1.896,
+      "step": 51
+    },
+    {
+      "epoch": 1.56,
+      "learning_rate": 1.244271214986699e-05,
+      "loss": 1.6864,
+      "step": 52
+    },
+    {
+      "epoch": 1.59,
+      "learning_rate": 1.2327373368773164e-05,
+      "loss": 1.7704,
+      "step": 53
+    },
+    {
+      "epoch": 1.62,
+      "learning_rate": 1.2210051543582597e-05,
+      "loss": 1.8227,
+      "step": 54
+    },
+    {
+      "epoch": 1.65,
+      "learning_rate": 1.2090794869107473e-05,
+      "loss": 1.6988,
+      "step": 55
+    },
+    {
+      "epoch": 1.68,
+      "learning_rate": 1.1969652334979677e-05,
+      "loss": 2.3322,
+      "step": 56
+    },
+    {
+      "epoch": 1.71,
+      "learning_rate": 1.1846673705526274e-05,
+      "loss": 1.8776,
+      "step": 57
+    },
+    {
+      "epoch": 1.74,
+      "learning_rate": 1.1721909499326732e-05,
+      "loss": 1.9751,
+      "step": 58
+    },
+    {
+      "epoch": 1.77,
+      "learning_rate": 1.1595410968460328e-05,
+      "loss": 1.8121,
+      "step": 59
+    },
+    {
+      "epoch": 1.8,
+      "learning_rate": 1.1467230077452218e-05,
+      "loss": 1.7561,
+      "step": 60
+    },
+    {
+      "epoch": 1.83,
+      "learning_rate": 1.1337419481926869e-05,
+      "loss": 1.8807,
+      "step": 61
+    },
+    {
+      "epoch": 1.86,
+      "learning_rate": 1.1206032506977544e-05,
+      "loss": 1.6302,
+      "step": 62
+    },
+    {
+      "epoch": 1.89,
+      "learning_rate": 1.1073123125260839e-05,
+      "loss": 1.755,
+      "step": 63
+    },
+    {
+      "epoch": 1.92,
+      "learning_rate": 1.0938745934825148e-05,
+      "loss": 1.9796,
+      "step": 64
+    },
+    {
+      "epoch": 1.95,
+      "learning_rate": 1.0802956136682259e-05,
+      "loss": 1.7132,
+      "step": 65
+    },
+    {
+      "epoch": 1.98,
+      "learning_rate": 1.066580951213124e-05,
+      "loss": 1.7189,
+      "step": 66
+    },
+    {
+      "epoch": 2.01,
+      "learning_rate": 1.052736239984396e-05,
+      "loss": 1.6825,
+      "step": 67
+    },
+    {
+      "epoch": 2.04,
+      "learning_rate": 1.0387671672721633e-05,
+      "loss": 1.7156,
+      "step": 68
+    },
+    {
+      "epoch": 2.07,
+      "learning_rate": 1.024679471453192e-05,
+      "loss": 1.6693,
+      "step": 69
+    },
+    {
+      "epoch": 2.1,
+      "learning_rate": 1.0104789396336152e-05,
+      "loss": 1.8269,
+      "step": 70
+    },
+    {
+      "epoch": 2.13,
+      "learning_rate": 9.961714052716398e-06,
+      "loss": 1.6614,
+      "step": 71
+    },
+    {
+      "epoch": 2.16,
+      "learning_rate": 9.817627457812105e-06,
+      "loss": 1.8712,
+      "step": 72
+    },
+    {
+      "epoch": 2.19,
+      "learning_rate": 9.672588801176193e-06,
+      "loss": 1.8181,
+      "step": 73
+    },
+    {
+      "epoch": 2.22,
+      "learning_rate": 9.526657663460466e-06,
+      "loss": 1.6211,
+      "step": 74
+    },
+    {
+      "epoch": 2.25,
+      "learning_rate": 9.379893991940404e-06,
+      "loss": 1.813,
+      "step": 75
+    },
+    {
+      "epoch": 2.25,
+      "eval_loss": 2.0914103984832764,
+      "eval_runtime": 5.7977,
+      "eval_samples_per_second": 4.83,
+      "eval_steps_per_second": 0.69,
+      "step": 75
+    },
+    {
+      "epoch": 2.28,
+      "learning_rate": 9.232358075889326e-06,
+      "loss": 1.6357,
+      "step": 76
+    },
+    {
+      "epoch": 2.31,
+      "learning_rate": 9.084110521812056e-06,
+      "loss": 1.5066,
+      "step": 77
+    },
+    {
+      "epoch": 2.34,
+      "learning_rate": 8.935212228548297e-06,
+      "loss": 1.7587,
+      "step": 78
+    },
+    {
+      "epoch": 2.37,
+      "learning_rate": 8.785724362255902e-06,
+      "loss": 2.1006,
+      "step": 79
+    },
+    {
+      "epoch": 2.4,
+      "learning_rate": 8.635708331284326e-06,
+      "loss": 1.5684,
+      "step": 80
+    },
+    {
+      "epoch": 2.43,
+      "learning_rate": 8.485225760948599e-06,
+      "loss": 1.8519,
+      "step": 81
+    },
+    {
+      "epoch": 2.46,
+      "learning_rate": 8.334338468214178e-06,
+      "loss": 2.0052,
+      "step": 82
+    },
+    {
+      "epoch": 2.49,
+      "learning_rate": 8.183108436303049e-06,
+      "loss": 1.5687,
+      "step": 83
+    },
+    {
+      "epoch": 2.51,
+      "learning_rate": 8.031597789231547e-06,
+      "loss": 1.6001,
+      "step": 84
+    },
+    {
+      "epoch": 2.54,
+      "learning_rate": 7.879868766290346e-06,
+      "loss": 1.6623,
+      "step": 85
+    },
+    {
+      "epoch": 2.57,
+      "learning_rate": 7.727983696477072e-06,
+      "loss": 1.6158,
+      "step": 86
+    },
+    {
+      "epoch": 2.6,
+      "learning_rate": 7.576004972892083e-06,
+      "loss": 1.8351,
+      "step": 87
+    },
+    {
+      "epoch": 2.63,
+      "learning_rate": 7.42399502710792e-06,
+      "loss": 1.9375,
+      "step": 88
+    },
+    {
+      "epoch": 2.66,
+      "learning_rate": 7.272016303522931e-06,
+      "loss": 1.6786,
+      "step": 89
+    },
+    {
+      "epoch": 2.69,
+      "learning_rate": 7.120131233709655e-06,
+      "loss": 1.4921,
+      "step": 90
+    },
+    {
+      "epoch": 2.72,
+      "learning_rate": 6.968402210768453e-06,
+      "loss": 1.7724,
+      "step": 91
+    },
+    {
+      "epoch": 2.75,
+      "learning_rate": 6.816891563696952e-06,
+      "loss": 1.4894,
+      "step": 92
+    },
+    {
+      "epoch": 2.78,
+      "learning_rate": 6.665661531785824e-06,
+      "loss": 1.5464,
+      "step": 93
+    },
+    {
+      "epoch": 2.81,
+      "learning_rate": 6.5147742390514015e-06,
+      "loss": 1.6199,
+      "step": 94
+    },
+    {
+      "epoch": 2.84,
+      "learning_rate": 6.364291668715678e-06,
+      "loss": 1.516,
+      "step": 95
+    },
+    {
+      "epoch": 2.87,
+      "learning_rate": 6.214275637744099e-06,
+      "loss": 1.733,
+      "step": 96
+    },
+    {
+      "epoch": 2.9,
+      "learning_rate": 6.064787771451703e-06,
+      "loss": 1.6849,
+      "step": 97
+    },
+    {
+      "epoch": 2.93,
+      "learning_rate": 5.915889478187945e-06,
+      "loss": 1.5236,
+      "step": 98
+    },
+    {
+      "epoch": 2.96,
+      "learning_rate": 5.767641924110675e-06,
+      "loss": 1.6909,
+      "step": 99
+    },
+    {
+      "epoch": 2.99,
+      "learning_rate": 5.620106008059595e-06,
+      "loss": 1.8067,
+      "step": 100
+    },
+    {
+      "epoch": 2.99,
+      "eval_loss": 2.023528575897217,
+      "eval_runtime": 5.7889,
+      "eval_samples_per_second": 4.837,
+      "eval_steps_per_second": 0.691,
+      "step": 100
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 165,
+  "num_train_epochs": 5,
+  "save_steps": 50,
+  "total_flos": 2.1461430586834944e+17,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-100/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4f49d59d3dd021439d5df3b0814b6dcba6ff9d4dd59d1bee8fc86f469e41e6e
+size 4920

checkpoint-150/README.md

@@ -0,0 +1,219 @@
+---
+library_name: peft
+base_model: KoboldAI/LLaMA2-13B-TiefighterLR
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure 
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Data Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+
+
+## Training procedure
+
+
+The following `bitsandbytes` quantization config was used during training:
+- quant_method: bitsandbytes
+- load_in_8bit: True
+- load_in_4bit: False
+- llm_int8_threshold: 6.0
+- llm_int8_skip_modules: None
+- llm_int8_enable_fp32_cpu_offload: False
+- llm_int8_has_fp16_weight: False
+- bnb_4bit_quant_type: fp4
+- bnb_4bit_use_double_quant: False
+- bnb_4bit_compute_dtype: float32
+
+### Framework versions
+
+
+- PEFT 0.6.0

checkpoint-150/adapter_config.json

@@ -0,0 +1,28 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "KoboldAI/LLaMA2-13B-TiefighterLR",
+  "bias": "none",
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "gate_proj",
+    "q_proj",
+    "up_proj",
+    "o_proj",
+    "v_proj",
+    "down_proj"
+  ],
+  "task_type": "CAUSAL_LM"
+}

checkpoint-150/adapter_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:af84eb1ef393f0e792e09998b8c61789f659fbc8aaf8dd5367a0abc3ad6994a1
+size 2002982666

checkpoint-150/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4f73ce244c4aee38af5598b20310c1776f064f74f574f8a8112dae96bf74f90
+size 4005907122

checkpoint-150/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c13afc35b0ed6289d605a79e1ec603d693631c4a8e8e2126ce86e13efbdc1872
+size 14244

checkpoint-150/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d89149726f6f60010e65929fc9e5eb5e3137685f0d9df7260a412b574ba7177
+size 1064

checkpoint-150/trainer_state.json

@@ -0,0 +1,967 @@
+{
+  "best_metric": 1.9844228029251099,
+  "best_model_checkpoint": "./lora-outC/checkpoint-150",
+  "epoch": 4.491017964071856,
+  "eval_steps": 25,
+  "global_step": 150,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.03,
+      "learning_rate": 1.5e-06,
+      "loss": 1.8176,
+      "step": 1
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 3e-06,
+      "loss": 2.0942,
+      "step": 2
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.5e-06,
+      "loss": 3.1029,
+      "step": 3
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 6e-06,
+      "loss": 2.0346,
+      "step": 4
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 7.5e-06,
+      "loss": 2.1768,
+      "step": 5
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 9e-06,
+      "loss": 1.9869,
+      "step": 6
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 1.05e-05,
+      "loss": 2.7137,
+      "step": 7
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 1.2e-05,
+      "loss": 1.9197,
+      "step": 8
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 1.3500000000000001e-05,
+      "loss": 2.405,
+      "step": 9
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 1.5e-05,
+      "loss": 3.0781,
+      "step": 10
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 1.4998459531758846e-05,
+      "loss": 1.8125,
+      "step": 11
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 1.4993838759846692e-05,
+      "loss": 2.0828,
+      "step": 12
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 1.4986139582437507e-05,
+      "loss": 1.9523,
+      "step": 13
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 1.4975365162288163e-05,
+      "loss": 2.1143,
+      "step": 14
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 1.4961519925439214e-05,
+      "loss": 2.1367,
+      "step": 15
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 1.4944609559396697e-05,
+      "loss": 1.8429,
+      "step": 16
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 1.4924641010795769e-05,
+      "loss": 2.217,
+      "step": 17
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 1.4901622482547082e-05,
+      "loss": 1.7973,
+      "step": 18
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 1.4875563430467096e-05,
+      "loss": 2.1631,
+      "step": 19
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 1.484647455939371e-05,
+      "loss": 1.9283,
+      "step": 20
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 1.4814367818788806e-05,
+      "loss": 1.9236,
+      "step": 21
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 1.4779256397829512e-05,
+      "loss": 2.5061,
+      "step": 22
+    },
+    {
+      "epoch": 0.69,
+      "learning_rate": 1.4741154719990202e-05,
+      "loss": 2.2137,
+      "step": 23
+    },
+    {
+      "epoch": 0.72,
+      "learning_rate": 1.4700078437117444e-05,
+      "loss": 2.7564,
+      "step": 24
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 1.4656044423000367e-05,
+      "loss": 1.8419,
+      "step": 25
+    },
+    {
+      "epoch": 0.75,
+      "eval_loss": 2.5257418155670166,
+      "eval_runtime": 5.8091,
+      "eval_samples_per_second": 4.82,
+      "eval_steps_per_second": 0.689,
+      "step": 25
+    },
+    {
+      "epoch": 0.78,
+      "learning_rate": 1.4609070766439045e-05,
+      "loss": 2.3377,
+      "step": 26
+    },
+    {
+      "epoch": 0.81,
+      "learning_rate": 1.4559176763813775e-05,
+      "loss": 1.7976,
+      "step": 27
+    },
+    {
+      "epoch": 0.84,
+      "learning_rate": 1.450638291115828e-05,
+      "loss": 2.0391,
+      "step": 28
+    },
+    {
+      "epoch": 0.87,
+      "learning_rate": 1.4450710895740116e-05,
+      "loss": 1.9452,
+      "step": 29
+    },
+    {
+      "epoch": 0.9,
+      "learning_rate": 1.439218358715173e-05,
+      "loss": 1.8263,
+      "step": 30
+    },
+    {
+      "epoch": 0.93,
+      "learning_rate": 1.4330825027915823e-05,
+      "loss": 2.0638,
+      "step": 31
+    },
+    {
+      "epoch": 0.96,
+      "learning_rate": 1.4266660423608882e-05,
+      "loss": 1.8948,
+      "step": 32
+    },
+    {
+      "epoch": 0.99,
+      "learning_rate": 1.4199716132506944e-05,
+      "loss": 1.985,
+      "step": 33
+    },
+    {
+      "epoch": 1.02,
+      "learning_rate": 1.4130019654757838e-05,
+      "loss": 2.2594,
+      "step": 34
+    },
+    {
+      "epoch": 1.05,
+      "learning_rate": 1.4057599621084365e-05,
+      "loss": 2.2773,
+      "step": 35
+    },
+    {
+      "epoch": 1.08,
+      "learning_rate": 1.3982485781023034e-05,
+      "loss": 1.659,
+      "step": 36
+    },
+    {
+      "epoch": 1.11,
+      "learning_rate": 1.3904708990703192e-05,
+      "loss": 1.834,
+      "step": 37
+    },
+    {
+      "epoch": 1.14,
+      "learning_rate": 1.3824301200171617e-05,
+      "loss": 1.6516,
+      "step": 38
+    },
+    {
+      "epoch": 1.17,
+      "learning_rate": 1.374129544026768e-05,
+      "loss": 2.1114,
+      "step": 39
+    },
+    {
+      "epoch": 1.2,
+      "learning_rate": 1.3655725809054574e-05,
+      "loss": 1.6287,
+      "step": 40
+    },
+    {
+      "epoch": 1.23,
+      "learning_rate": 1.3567627457812107e-05,
+      "loss": 1.6146,
+      "step": 41
+    },
+    {
+      "epoch": 1.26,
+      "learning_rate": 1.3477036576596862e-05,
+      "loss": 2.2355,
+      "step": 42
+    },
+    {
+      "epoch": 1.29,
+      "learning_rate": 1.3383990379375632e-05,
+      "loss": 1.8845,
+      "step": 43
+    },
+    {
+      "epoch": 1.32,
+      "learning_rate": 1.3288527088738215e-05,
+      "loss": 2.0984,
+      "step": 44
+    },
+    {
+      "epoch": 1.35,
+      "learning_rate": 1.3190685920195931e-05,
+      "loss": 1.7909,
+      "step": 45
+    },
+    {
+      "epoch": 1.38,
+      "learning_rate": 1.3090507066072199e-05,
+      "loss": 1.9248,
+      "step": 46
+    },
+    {
+      "epoch": 1.41,
+      "learning_rate": 1.2988031678991868e-05,
+      "loss": 1.651,
+      "step": 47
+    },
+    {
+      "epoch": 1.44,
+      "learning_rate": 1.2883301854976092e-05,
+      "loss": 1.8998,
+      "step": 48
+    },
+    {
+      "epoch": 1.47,
+      "learning_rate": 1.277636061614961e-05,
+      "loss": 2.0732,
+      "step": 49
+    },
+    {
+      "epoch": 1.5,
+      "learning_rate": 1.266725189306765e-05,
+      "loss": 1.7748,
+      "step": 50
+    },
+    {
+      "epoch": 1.5,
+      "eval_loss": 2.2467305660247803,
+      "eval_runtime": 5.8123,
+      "eval_samples_per_second": 4.817,
+      "eval_steps_per_second": 0.688,
+      "step": 50
+    },
+    {
+      "epoch": 1.53,
+      "learning_rate": 1.2556020506669606e-05,
+      "loss": 1.896,
+      "step": 51
+    },
+    {
+      "epoch": 1.56,
+      "learning_rate": 1.244271214986699e-05,
+      "loss": 1.6864,
+      "step": 52
+    },
+    {
+      "epoch": 1.59,
+      "learning_rate": 1.2327373368773164e-05,
+      "loss": 1.7704,
+      "step": 53
+    },
+    {
+      "epoch": 1.62,
+      "learning_rate": 1.2210051543582597e-05,
+      "loss": 1.8227,
+      "step": 54
+    },
+    {
+      "epoch": 1.65,
+      "learning_rate": 1.2090794869107473e-05,
+      "loss": 1.6988,
+      "step": 55
+    },
+    {
+      "epoch": 1.68,
+      "learning_rate": 1.1969652334979677e-05,
+      "loss": 2.3322,
+      "step": 56
+    },
+    {
+      "epoch": 1.71,
+      "learning_rate": 1.1846673705526274e-05,
+      "loss": 1.8776,
+      "step": 57
+    },
+    {
+      "epoch": 1.74,
+      "learning_rate": 1.1721909499326732e-05,
+      "loss": 1.9751,
+      "step": 58
+    },
+    {
+      "epoch": 1.77,
+      "learning_rate": 1.1595410968460328e-05,
+      "loss": 1.8121,
+      "step": 59
+    },
+    {
+      "epoch": 1.8,
+      "learning_rate": 1.1467230077452218e-05,
+      "loss": 1.7561,
+      "step": 60
+    },
+    {
+      "epoch": 1.83,
+      "learning_rate": 1.1337419481926869e-05,
+      "loss": 1.8807,
+      "step": 61
+    },
+    {
+      "epoch": 1.86,
+      "learning_rate": 1.1206032506977544e-05,
+      "loss": 1.6302,
+      "step": 62
+    },
+    {
+      "epoch": 1.89,
+      "learning_rate": 1.1073123125260839e-05,
+      "loss": 1.755,
+      "step": 63
+    },
+    {
+      "epoch": 1.92,
+      "learning_rate": 1.0938745934825148e-05,
+      "loss": 1.9796,
+      "step": 64
+    },
+    {
+      "epoch": 1.95,
+      "learning_rate": 1.0802956136682259e-05,
+      "loss": 1.7132,
+      "step": 65
+    },
+    {
+      "epoch": 1.98,
+      "learning_rate": 1.066580951213124e-05,
+      "loss": 1.7189,
+      "step": 66
+    },
+    {
+      "epoch": 2.01,
+      "learning_rate": 1.052736239984396e-05,
+      "loss": 1.6825,
+      "step": 67
+    },
+    {
+      "epoch": 2.04,
+      "learning_rate": 1.0387671672721633e-05,
+      "loss": 1.7156,
+      "step": 68
+    },
+    {
+      "epoch": 2.07,
+      "learning_rate": 1.024679471453192e-05,
+      "loss": 1.6693,
+      "step": 69
+    },
+    {
+      "epoch": 2.1,
+      "learning_rate": 1.0104789396336152e-05,
+      "loss": 1.8269,
+      "step": 70
+    },
+    {
+      "epoch": 2.13,
+      "learning_rate": 9.961714052716398e-06,
+      "loss": 1.6614,
+      "step": 71
+    },
+    {
+      "epoch": 2.16,
+      "learning_rate": 9.817627457812105e-06,
+      "loss": 1.8712,
+      "step": 72
+    },
+    {
+      "epoch": 2.19,
+      "learning_rate": 9.672588801176193e-06,
+      "loss": 1.8181,
+      "step": 73
+    },
+    {
+      "epoch": 2.22,
+      "learning_rate": 9.526657663460466e-06,
+      "loss": 1.6211,
+      "step": 74
+    },
+    {
+      "epoch": 2.25,
+      "learning_rate": 9.379893991940404e-06,
+      "loss": 1.813,
+      "step": 75
+    },
+    {
+      "epoch": 2.25,
+      "eval_loss": 2.0914103984832764,
+      "eval_runtime": 5.7977,
+      "eval_samples_per_second": 4.83,
+      "eval_steps_per_second": 0.69,
+      "step": 75
+    },
+    {
+      "epoch": 2.28,
+      "learning_rate": 9.232358075889326e-06,
+      "loss": 1.6357,
+      "step": 76
+    },
+    {
+      "epoch": 2.31,
+      "learning_rate": 9.084110521812056e-06,
+      "loss": 1.5066,
+      "step": 77
+    },
+    {
+      "epoch": 2.34,
+      "learning_rate": 8.935212228548297e-06,
+      "loss": 1.7587,
+      "step": 78
+    },
+    {
+      "epoch": 2.37,
+      "learning_rate": 8.785724362255902e-06,
+      "loss": 2.1006,
+      "step": 79
+    },
+    {
+      "epoch": 2.4,
+      "learning_rate": 8.635708331284326e-06,
+      "loss": 1.5684,
+      "step": 80
+    },
+    {
+      "epoch": 2.43,
+      "learning_rate": 8.485225760948599e-06,
+      "loss": 1.8519,
+      "step": 81
+    },
+    {
+      "epoch": 2.46,
+      "learning_rate": 8.334338468214178e-06,
+      "loss": 2.0052,
+      "step": 82
+    },
+    {
+      "epoch": 2.49,
+      "learning_rate": 8.183108436303049e-06,
+      "loss": 1.5687,
+      "step": 83
+    },
+    {
+      "epoch": 2.51,
+      "learning_rate": 8.031597789231547e-06,
+      "loss": 1.6001,
+      "step": 84
+    },
+    {
+      "epoch": 2.54,
+      "learning_rate": 7.879868766290346e-06,
+      "loss": 1.6623,
+      "step": 85
+    },
+    {
+      "epoch": 2.57,
+      "learning_rate": 7.727983696477072e-06,
+      "loss": 1.6158,
+      "step": 86
+    },
+    {
+      "epoch": 2.6,
+      "learning_rate": 7.576004972892083e-06,
+      "loss": 1.8351,
+      "step": 87
+    },
+    {
+      "epoch": 2.63,
+      "learning_rate": 7.42399502710792e-06,
+      "loss": 1.9375,
+      "step": 88
+    },
+    {
+      "epoch": 2.66,
+      "learning_rate": 7.272016303522931e-06,
+      "loss": 1.6786,
+      "step": 89
+    },
+    {
+      "epoch": 2.69,
+      "learning_rate": 7.120131233709655e-06,
+      "loss": 1.4921,
+      "step": 90
+    },
+    {
+      "epoch": 2.72,
+      "learning_rate": 6.968402210768453e-06,
+      "loss": 1.7724,
+      "step": 91
+    },
+    {
+      "epoch": 2.75,
+      "learning_rate": 6.816891563696952e-06,
+      "loss": 1.4894,
+      "step": 92
+    },
+    {
+      "epoch": 2.78,
+      "learning_rate": 6.665661531785824e-06,
+      "loss": 1.5464,
+      "step": 93
+    },
+    {
+      "epoch": 2.81,
+      "learning_rate": 6.5147742390514015e-06,
+      "loss": 1.6199,
+      "step": 94
+    },
+    {
+      "epoch": 2.84,
+      "learning_rate": 6.364291668715678e-06,
+      "loss": 1.516,
+      "step": 95
+    },
+    {
+      "epoch": 2.87,
+      "learning_rate": 6.214275637744099e-06,
+      "loss": 1.733,
+      "step": 96
+    },
+    {
+      "epoch": 2.9,
+      "learning_rate": 6.064787771451703e-06,
+      "loss": 1.6849,
+      "step": 97
+    },
+    {
+      "epoch": 2.93,
+      "learning_rate": 5.915889478187945e-06,
+      "loss": 1.5236,
+      "step": 98
+    },
+    {
+      "epoch": 2.96,
+      "learning_rate": 5.767641924110675e-06,
+      "loss": 1.6909,
+      "step": 99
+    },
+    {
+      "epoch": 2.99,
+      "learning_rate": 5.620106008059595e-06,
+      "loss": 1.8067,
+      "step": 100
+    },
+    {
+      "epoch": 2.99,
+      "eval_loss": 2.023528575897217,
+      "eval_runtime": 5.7889,
+      "eval_samples_per_second": 4.837,
+      "eval_steps_per_second": 0.691,
+      "step": 100
+    },
+    {
+      "epoch": 3.02,
+      "learning_rate": 5.473342336539537e-06,
+      "loss": 1.5651,
+      "step": 101
+    },
+    {
+      "epoch": 3.05,
+      "learning_rate": 5.32741119882381e-06,
+      "loss": 1.612,
+      "step": 102
+    },
+    {
+      "epoch": 3.08,
+      "learning_rate": 5.182372542187895e-06,
+      "loss": 1.6852,
+      "step": 103
+    },
+    {
+      "epoch": 3.11,
+      "learning_rate": 5.038285947283603e-06,
+      "loss": 1.7725,
+      "step": 104
+    },
+    {
+      "epoch": 3.14,
+      "learning_rate": 4.895210603663849e-06,
+      "loss": 1.6168,
+      "step": 105
+    },
+    {
+      "epoch": 3.17,
+      "learning_rate": 4.75320528546808e-06,
+      "loss": 1.4268,
+      "step": 106
+    },
+    {
+      "epoch": 3.2,
+      "learning_rate": 4.612328327278365e-06,
+      "loss": 1.7011,
+      "step": 107
+    },
+    {
+      "epoch": 3.23,
+      "learning_rate": 4.472637600156042e-06,
+      "loss": 1.6768,
+      "step": 108
+    },
+    {
+      "epoch": 3.26,
+      "learning_rate": 4.334190487868761e-06,
+      "loss": 1.5086,
+      "step": 109
+    },
+    {
+      "epoch": 3.29,
+      "learning_rate": 4.197043863317742e-06,
+      "loss": 1.5546,
+      "step": 110
+    },
+    {
+      "epoch": 3.32,
+      "learning_rate": 4.061254065174856e-06,
+      "loss": 1.4834,
+      "step": 111
+    },
+    {
+      "epoch": 3.35,
+      "learning_rate": 3.9268768747391635e-06,
+      "loss": 1.7054,
+      "step": 112
+    },
+    {
+      "epoch": 3.38,
+      "learning_rate": 3.793967493022458e-06,
+      "loss": 1.5152,
+      "step": 113
+    },
+    {
+      "epoch": 3.41,
+      "learning_rate": 3.662580518073134e-06,
+      "loss": 1.794,
+      "step": 114
+    },
+    {
+      "epoch": 3.44,
+      "learning_rate": 3.5327699225477836e-06,
+      "loss": 1.7234,
+      "step": 115
+    },
+    {
+      "epoch": 3.47,
+      "learning_rate": 3.4045890315396753e-06,
+      "loss": 2.276,
+      "step": 116
+    },
+    {
+      "epoch": 3.5,
+      "learning_rate": 3.278090500673269e-06,
+      "loss": 1.7764,
+      "step": 117
+    },
+    {
+      "epoch": 3.53,
+      "learning_rate": 3.153326294473727e-06,
+      "loss": 1.6114,
+      "step": 118
+    },
+    {
+      "epoch": 3.56,
+      "learning_rate": 3.030347665020323e-06,
+      "loss": 1.469,
+      "step": 119
+    },
+    {
+      "epoch": 3.59,
+      "learning_rate": 2.909205130892528e-06,
+      "loss": 1.7046,
+      "step": 120
+    },
+    {
+      "epoch": 3.62,
+      "learning_rate": 2.7899484564174036e-06,
+      "loss": 1.9297,
+      "step": 121
+    },
+    {
+      "epoch": 3.65,
+      "learning_rate": 2.6726266312268373e-06,
+      "loss": 1.6532,
+      "step": 122
+    },
+    {
+      "epoch": 3.68,
+      "learning_rate": 2.557287850133012e-06,
+      "loss": 2.4251,
+      "step": 123
+    },
+    {
+      "epoch": 3.71,
+      "learning_rate": 2.443979493330395e-06,
+      "loss": 1.6847,
+      "step": 124
+    },
+    {
+      "epoch": 3.74,
+      "learning_rate": 2.3327481069323503e-06,
+      "loss": 1.5346,
+      "step": 125
+    },
+    {
+      "epoch": 3.74,
+      "eval_loss": 1.9938760995864868,
+      "eval_runtime": 5.8028,
+      "eval_samples_per_second": 4.825,
+      "eval_steps_per_second": 0.689,
+      "step": 125
+    },
+    {
+      "epoch": 3.77,
+      "learning_rate": 2.223639383850389e-06,
+      "loss": 1.4916,
+      "step": 126
+    },
+    {
+      "epoch": 3.8,
+      "learning_rate": 2.1166981450239087e-06,
+      "loss": 1.4439,
+      "step": 127
+    },
+    {
+      "epoch": 3.83,
+      "learning_rate": 2.011968321008133e-06,
+      "loss": 1.5444,
+      "step": 128
+    },
+    {
+      "epoch": 3.86,
+      "learning_rate": 1.9094929339278032e-06,
+      "loss": 1.6214,
+      "step": 129
+    },
+    {
+      "epoch": 3.89,
+      "learning_rate": 1.8093140798040686e-06,
+      "loss": 1.7051,
+      "step": 130
+    },
+    {
+      "epoch": 3.92,
+      "learning_rate": 1.7114729112617848e-06,
+      "loss": 1.5758,
+      "step": 131
+    },
+    {
+      "epoch": 3.95,
+      "learning_rate": 1.6160096206243715e-06,
+      "loss": 2.1254,
+      "step": 132
+    },
+    {
+      "epoch": 3.98,
+      "learning_rate": 1.5229634234031384e-06,
+      "loss": 1.7538,
+      "step": 133
+    },
+    {
+      "epoch": 4.01,
+      "learning_rate": 1.432372542187895e-06,
+      "loss": 1.5902,
+      "step": 134
+    },
+    {
+      "epoch": 4.04,
+      "learning_rate": 1.3442741909454295e-06,
+      "loss": 1.6988,
+      "step": 135
+    },
+    {
+      "epoch": 4.07,
+      "learning_rate": 1.2587045597323219e-06,
+      "loss": 2.1365,
+      "step": 136
+    },
+    {
+      "epoch": 4.1,
+      "learning_rate": 1.1756987998283847e-06,
+      "loss": 1.5347,
+      "step": 137
+    },
+    {
+      "epoch": 4.13,
+      "learning_rate": 1.0952910092968086e-06,
+      "loss": 1.5637,
+      "step": 138
+    },
+    {
+      "epoch": 4.16,
+      "learning_rate": 1.0175142189769688e-06,
+      "loss": 1.6597,
+      "step": 139
+    },
+    {
+      "epoch": 4.19,
+      "learning_rate": 9.424003789156343e-07,
+      "loss": 1.5579,
+      "step": 140
+    },
+    {
+      "epoch": 4.22,
+      "learning_rate": 8.699803452421612e-07,
+      "loss": 1.5092,
+      "step": 141
+    },
+    {
+      "epoch": 4.25,
+      "learning_rate": 8.002838674930577e-07,
+      "loss": 1.8963,
+      "step": 142
+    },
+    {
+      "epoch": 4.28,
+      "learning_rate": 7.333395763911189e-07,
+      "loss": 1.672,
+      "step": 143
+    },
+    {
+      "epoch": 4.31,
+      "learning_rate": 6.691749720841769e-07,
+      "loss": 1.8841,
+      "step": 144
+    },
+    {
+      "epoch": 4.34,
+      "learning_rate": 6.078164128482705e-07,
+      "loss": 1.5356,
+      "step": 145
+    },
+    {
+      "epoch": 4.37,
+      "learning_rate": 5.492891042598855e-07,
+      "loss": 1.6926,
+      "step": 146
+    },
+    {
+      "epoch": 4.4,
+      "learning_rate": 4.936170888417221e-07,
+      "loss": 1.6317,
+      "step": 147
+    },
+    {
+      "epoch": 4.43,
+      "learning_rate": 4.408232361862252e-07,
+      "loss": 1.5067,
+      "step": 148
+    },
+    {
+      "epoch": 4.46,
+      "learning_rate": 3.9092923356095487e-07,
+      "loss": 1.7316,
+      "step": 149
+    },
+    {
+      "epoch": 4.49,
+      "learning_rate": 3.439555769996339e-07,
+      "loss": 1.5869,
+      "step": 150
+    },
+    {
+      "epoch": 4.49,
+      "eval_loss": 1.9844228029251099,
+      "eval_runtime": 5.8337,
+      "eval_samples_per_second": 4.8,
+      "eval_steps_per_second": 0.686,
+      "step": 150
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 165,
+  "num_train_epochs": 5,
+  "save_steps": 50,
+  "total_flos": 3.206088330785587e+17,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-150/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4f49d59d3dd021439d5df3b0814b6dcba6ff9d4dd59d1bee8fc86f469e41e6e
+size 4920

checkpoint-50/README.md

@@ -0,0 +1,219 @@
+---
+library_name: peft
+base_model: KoboldAI/LLaMA2-13B-TiefighterLR
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure 
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Data Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+
+
+## Training procedure
+
+
+The following `bitsandbytes` quantization config was used during training:
+- quant_method: bitsandbytes
+- load_in_8bit: True
+- load_in_4bit: False
+- llm_int8_threshold: 6.0
+- llm_int8_skip_modules: None
+- llm_int8_enable_fp32_cpu_offload: False
+- llm_int8_has_fp16_weight: False
+- bnb_4bit_quant_type: fp4
+- bnb_4bit_use_double_quant: False
+- bnb_4bit_compute_dtype: float32
+
+### Framework versions
+
+
+- PEFT 0.6.0

checkpoint-50/adapter_config.json

@@ -0,0 +1,28 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "KoboldAI/LLaMA2-13B-TiefighterLR",
+  "bias": "none",
+  "fan_in_fan_out": null,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "gate_proj",
+    "q_proj",
+    "up_proj",
+    "o_proj",
+    "v_proj",
+    "down_proj"
+  ],
+  "task_type": "CAUSAL_LM"
+}

checkpoint-50/adapter_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e811a7f9912f7d523b720f9ddc404e8dd597dd5b22fc2e304480fb713932d5b
+size 2002982666

checkpoint-50/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f75041f5770fc82dd9645769135031d6a61056501285c4520a56a385f4569d84
+size 14244

checkpoint-50/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d50702886eb30f6095dcc4ac5425699e01b239967c3b99bced0ededf97187fba
+size 1064

checkpoint-50/trainer_state.json

@@ -0,0 +1,335 @@
+{
+  "best_metric": 2.2467305660247803,
+  "best_model_checkpoint": "./lora-outC/checkpoint-50",
+  "epoch": 1.4970059880239521,
+  "eval_steps": 25,
+  "global_step": 50,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.03,
+      "learning_rate": 1.5e-06,
+      "loss": 1.8176,
+      "step": 1
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 3e-06,
+      "loss": 2.0942,
+      "step": 2
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 4.5e-06,
+      "loss": 3.1029,
+      "step": 3
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 6e-06,
+      "loss": 2.0346,
+      "step": 4
+    },
+    {
+      "epoch": 0.15,
+      "learning_rate": 7.5e-06,
+      "loss": 2.1768,
+      "step": 5
+    },
+    {
+      "epoch": 0.18,
+      "learning_rate": 9e-06,
+      "loss": 1.9869,
+      "step": 6
+    },
+    {
+      "epoch": 0.21,
+      "learning_rate": 1.05e-05,
+      "loss": 2.7137,
+      "step": 7
+    },
+    {
+      "epoch": 0.24,
+      "learning_rate": 1.2e-05,
+      "loss": 1.9197,
+      "step": 8
+    },
+    {
+      "epoch": 0.27,
+      "learning_rate": 1.3500000000000001e-05,
+      "loss": 2.405,
+      "step": 9
+    },
+    {
+      "epoch": 0.3,
+      "learning_rate": 1.5e-05,
+      "loss": 3.0781,
+      "step": 10
+    },
+    {
+      "epoch": 0.33,
+      "learning_rate": 1.4998459531758846e-05,
+      "loss": 1.8125,
+      "step": 11
+    },
+    {
+      "epoch": 0.36,
+      "learning_rate": 1.4993838759846692e-05,
+      "loss": 2.0828,
+      "step": 12
+    },
+    {
+      "epoch": 0.39,
+      "learning_rate": 1.4986139582437507e-05,
+      "loss": 1.9523,
+      "step": 13
+    },
+    {
+      "epoch": 0.42,
+      "learning_rate": 1.4975365162288163e-05,
+      "loss": 2.1143,
+      "step": 14
+    },
+    {
+      "epoch": 0.45,
+      "learning_rate": 1.4961519925439214e-05,
+      "loss": 2.1367,
+      "step": 15
+    },
+    {
+      "epoch": 0.48,
+      "learning_rate": 1.4944609559396697e-05,
+      "loss": 1.8429,
+      "step": 16
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 1.4924641010795769e-05,
+      "loss": 2.217,
+      "step": 17
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 1.4901622482547082e-05,
+      "loss": 1.7973,
+      "step": 18
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 1.4875563430467096e-05,
+      "loss": 2.1631,
+      "step": 19
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 1.484647455939371e-05,
+      "loss": 1.9283,
+      "step": 20
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 1.4814367818788806e-05,
+      "loss": 1.9236,
+      "step": 21
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 1.4779256397829512e-05,
+      "loss": 2.5061,
+      "step": 22
+    },
+    {
+      "epoch": 0.69,
+      "learning_rate": 1.4741154719990202e-05,
+      "loss": 2.2137,
+      "step": 23
+    },
+    {
+      "epoch": 0.72,
+      "learning_rate": 1.4700078437117444e-05,
+      "loss": 2.7564,
+      "step": 24
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 1.4656044423000367e-05,
+      "loss": 1.8419,
+      "step": 25
+    },
+    {
+      "epoch": 0.75,
+      "eval_loss": 2.5257418155670166,
+      "eval_runtime": 5.8091,
+      "eval_samples_per_second": 4.82,
+      "eval_steps_per_second": 0.689,
+      "step": 25
+    },
+    {
+      "epoch": 0.78,
+      "learning_rate": 1.4609070766439045e-05,
+      "loss": 2.3377,
+      "step": 26
+    },
+    {
+      "epoch": 0.81,
+      "learning_rate": 1.4559176763813775e-05,
+      "loss": 1.7976,
+      "step": 27
+    },
+    {
+      "epoch": 0.84,
+      "learning_rate": 1.450638291115828e-05,
+      "loss": 2.0391,
+      "step": 28
+    },
+    {
+      "epoch": 0.87,
+      "learning_rate": 1.4450710895740116e-05,
+      "loss": 1.9452,
+      "step": 29
+    },
+    {
+      "epoch": 0.9,
+      "learning_rate": 1.439218358715173e-05,
+      "loss": 1.8263,
+      "step": 30
+    },
+    {
+      "epoch": 0.93,
+      "learning_rate": 1.4330825027915823e-05,
+      "loss": 2.0638,
+      "step": 31
+    },
+    {
+      "epoch": 0.96,
+      "learning_rate": 1.4266660423608882e-05,
+      "loss": 1.8948,
+      "step": 32
+    },
+    {
+      "epoch": 0.99,
+      "learning_rate": 1.4199716132506944e-05,
+      "loss": 1.985,
+      "step": 33
+    },
+    {
+      "epoch": 1.02,
+      "learning_rate": 1.4130019654757838e-05,
+      "loss": 2.2594,
+      "step": 34
+    },
+    {
+      "epoch": 1.05,
+      "learning_rate": 1.4057599621084365e-05,
+      "loss": 2.2773,
+      "step": 35
+    },
+    {
+      "epoch": 1.08,
+      "learning_rate": 1.3982485781023034e-05,
+      "loss": 1.659,
+      "step": 36
+    },
+    {
+      "epoch": 1.11,
+      "learning_rate": 1.3904708990703192e-05,
+      "loss": 1.834,
+      "step": 37
+    },
+    {
+      "epoch": 1.14,
+      "learning_rate": 1.3824301200171617e-05,
+      "loss": 1.6516,
+      "step": 38
+    },
+    {
+      "epoch": 1.17,
+      "learning_rate": 1.374129544026768e-05,
+      "loss": 2.1114,
+      "step": 39
+    },
+    {
+      "epoch": 1.2,
+      "learning_rate": 1.3655725809054574e-05,
+      "loss": 1.6287,
+      "step": 40
+    },
+    {
+      "epoch": 1.23,
+      "learning_rate": 1.3567627457812107e-05,
+      "loss": 1.6146,
+      "step": 41
+    },
+    {
+      "epoch": 1.26,
+      "learning_rate": 1.3477036576596862e-05,
+      "loss": 2.2355,
+      "step": 42
+    },
+    {
+      "epoch": 1.29,
+      "learning_rate": 1.3383990379375632e-05,
+      "loss": 1.8845,
+      "step": 43
+    },
+    {
+      "epoch": 1.32,
+      "learning_rate": 1.3288527088738215e-05,
+      "loss": 2.0984,
+      "step": 44
+    },
+    {
+      "epoch": 1.35,
+      "learning_rate": 1.3190685920195931e-05,
+      "loss": 1.7909,
+      "step": 45
+    },
+    {
+      "epoch": 1.38,
+      "learning_rate": 1.3090507066072199e-05,
+      "loss": 1.9248,
+      "step": 46
+    },
+    {
+      "epoch": 1.41,
+      "learning_rate": 1.2988031678991868e-05,
+      "loss": 1.651,
+      "step": 47
+    },
+    {
+      "epoch": 1.44,
+      "learning_rate": 1.2883301854976092e-05,
+      "loss": 1.8998,
+      "step": 48
+    },
+    {
+      "epoch": 1.47,
+      "learning_rate": 1.277636061614961e-05,
+      "loss": 2.0732,
+      "step": 49
+    },
+    {
+      "epoch": 1.5,
+      "learning_rate": 1.266725189306765e-05,
+      "loss": 1.7748,
+      "step": 50
+    },
+    {
+      "epoch": 1.5,
+      "eval_loss": 2.2467305660247803,
+      "eval_runtime": 5.8123,
+      "eval_samples_per_second": 4.817,
+      "eval_steps_per_second": 0.688,
+      "step": 50
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 165,
+  "num_train_epochs": 5,
+  "save_steps": 50,
+  "total_flos": 1.0730715293417472e+17,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-50/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4f49d59d3dd021439d5df3b0814b6dcba6ff9d4dd59d1bee8fc86f469e41e6e
+size 4920

config.json

@@ -0,0 +1,41 @@
+{
+  "_name_or_path": "KoboldAI/LLaMA2-13B-TiefighterLR",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 13824,
+  "max_position_embeddings": 4096,
+  "model_type": "llama",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 40,
+  "num_key_value_heads": 40,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "quantization_config": {
+    "bnb_4bit_compute_dtype": "float32",
+    "bnb_4bit_quant_type": "fp4",
+    "bnb_4bit_use_double_quant": false,
+    "llm_int8_enable_fp32_cpu_offload": false,
+    "llm_int8_has_fp16_weight": false,
+    "llm_int8_skip_modules": null,
+    "llm_int8_threshold": 6.0,
+    "load_in_4bit": false,
+    "load_in_8bit": true,
+    "quant_method": "bitsandbytes"
+  },
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.34.1",
+  "use_cache": false,
+  "vocab_size": 32000,
+  "welcome": "# Welcome to Tiefighter by KoboldAI Henky!! \nTiefighter is a unique merge model that combines the best of fictional use cases into one model, a true all-rounder.\nThis does mean that the model may take some steering in the beginning of your session to guide it towards what you want to do.\n\n## Story Writing\nRegular story writing in the traditional way is supported, simply copy paste your story and continue writing. Optionally use an instruction in memory or an authors note to guide the direction of your story.\n\n### Generate a story on demand\nTo generate stories on demand you can use an instruction (tested in the Alpaca format) such as \"Write a novel about X, use chapters and dialogue\" this will generate a story. The format can vary between generations depending on how the model chooses to begin, either write what you want as shown in the earlier example or write the beginning of the story yourself so the model can follow your style. A few retries can also help if the model gets it wrong.\n\n## Chatbots and persona's\nThis model has been tested with various forms of chatting, testers have found that typically less is more and the model is good at improvising. Don't drown the model in paragraphs of detailed information, instead keep it simple first and see how far you can lean on the models own ability to figure out your character. Copy pasting paragraphs of background information is not suitable for a 13B model such as this one, code formatted characters or an instruction prompt describing who you wish to talk to goes much further.\n\nFor example, you can put this in memory in regular chat mode:\n``` \n### Instruction: \nGenerate a conversation between Alice and Henk where they discuss language models.\nIn this conversation Henk is excited to teach Alice about Tiefigther. \n### Response: \n```\n\nBecause the model is a merge of a variety of models, it should support a broad range of instruct formats, or plain chat mode. If you have a particular favourite try it, otherwise we recommend to either use the regular chat mode or Alpaca's format.\n\n## Instruct Prompting\nThis model features various instruct models on a variety of instruction styles, when testing the model we have used Alpaca for our own tests. If you prefer a different format chances are it can work.\n\nDuring instructions we have observed that in some cases the adventure data can leak, it may also be worth experimenting using > as the prefix for a user command to remedy this. But this may result in a stronger fiction bias.\n\nKeep in mind that while this model can be used as a factual instruct model, the focus was on fiction. Information provided by the model can be made up.\n\n## Adventuring and Adventure Games\nThis model contains a lora that was trained on the same adventure dataset as the KoboldAI Skein model. Adventuring is best done using an small introduction to the world and your objective while using the > prefix for a user command (KoboldAI's adventure mode).\n\nIt is possible that the model does not immediately pick up on what you wish to do and does not engage in its Adventure mode behaviour right away. Simply manually correct the output to trim excess dialogue or other undesirable behaviour and continue to submit your actions using the appropriate mode. The model should pick up on this style quickly and will correctly follow this format within 3 turns.\n\n## Discovered something cool and want to engage with us? \nJoin our community at https://koboldai.org/discord !\n\n### This model would not be possible without the awesome work from: \nUndi95, PocketDoc, Blackroot, Brouz, The Face of Goonery, zattio770, PygmalionAI, TokenBender, nRuaif, lemonilia, Xwin-LM, elinas, jondurbin, NousResearch, CalderaAI, MrSeeker, OpenAssistant, ehartford, Henk717, AI Dungeon, StabilityAI and zattio770."
+}

special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

tokenizer_config.json

@@ -0,0 +1,44 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<unk>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "trust_remote_code": false,
+  "unk_token": "<unk>",
+  "use_default_system_prompt": true,
+  "use_fast": true
+}