---
library_name: peft
license: llama2
language:
- en
tags:
- medical
base_model: lmsys/vicuna-7b-v1.5
inference: false
---

# MindWell

MindWell is a chat assistant trained by fine-tuning [lmsys/vicuna-7b-v1.5](https://huggingface.co/lmsys/vicuna-7b-v1.5) on two symptom-based depression datasets: [BDI-Sen](https://dl.acm.org/doi/abs/10.1145/3539618.3591905) and [PsySym](https://aclanthology.org/2022.emnlp-main.677/). To expedite and optimize the fine-tuning process, we have implemented Low Rank Adaptation (LoRA) techniques, ensuring enhanced efficiency and faster adaptation.

## How to Get Started with MindWell

Use the code below to get started with the model.

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, LlamaTokenizer, GenerationConfig

config = PeftConfig.from_pretrained("irlab-udc/MindWell")
model = AutoModelForCausalLM.from_pretrained("lmsys/vicuna-13b-v1.5", device_map="auto")
model = PeftModel.from_pretrained(model, "irlab-udc/MindWell")
tokenizer = AutoTokenizer.from_pretrained("lmsys/vicuna-13b-v1.5")

def evaluate(input):
  inputs = tokenizer(input, return_tensors="pt")
  input_ids = inputs["input_ids"].cuda()
  generation_output = model.generate(
      input_ids=input_ids,
      generation_config=GenerationConfig(do_sample=True),
      return_dict_in_generate=True,
      output_scores=True,
      max_new_tokens=512,
  )
  for s in generation_output.sequences:
    output = tokenizer.decode(s)
    print("Answer:", output)

evaluate("What can you do?")
```

```
Answer: I can analyze the user's comments to determine if they exhibit any signs of depressive symptoms based on the provided list of symptoms. I will justify my decisions by means of excerpts from the user's comments. If I don't know the answer, I will truthfully say that I don't know.
```

## Training

#### Configurations and Hyperparameters

The following `LoraConfig` config was used during training:
- r: 8
- lora_alpha: 16
- target_modules: ["q_proj", "v_proj"]
- lora_dropout: 0.05
- bias: "none"
- task_type: "CAUSAL_LM"

The following `TrainingArguments` config was used during training:
- per_device_train_batch_size: 64
- gradient_accumulation_steps: 32
- warmup_steps: 100
- num_train_epochs: 20
- learning_rate: 3e-4
- fp16=True

The following `bitsandbytes` quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: True
- load_in_4bit: False
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: fp4
- bnb_4bit_use_double_quant: False
- bnb_4bit_compute_dtype: float32

#### Framework versions

- PyTorch 2.1.0
- PEFT 0.5.0
- 🤗 Transformers 4.34.0
- 🤗 Datasets 2.14.5
- 🤗 Tokenizers 0.14.0


## Environmental Impact

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** NVIDIA RTX 6000 Ada Generation.
- **Hours used:** 20.
- **Cloud Provider:** Private infrastructure.
- **Carbon Emitted:** 2.59 Kg. CO2 eq.