|
--- |
|
library_name: transformers |
|
license: mit |
|
base_model: microsoft/Phi-4-multimodal-instruct |
|
tags: |
|
- generated_from_trainer |
|
model-index: |
|
- name: Phi-4-multimodal-instruct-asr-tr |
|
results: [] |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# Phi-4-multimodal-instruct-asr-tr |
|
|
|
This model is a fine-tuned version of [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) on a 600-hour Turkish audio dataset, trained for a single epoch because of resource constraints. |
|
|
|
## Trained with Prompt: "Transcribe the Turkish audio" |
|
Including the source language during inference helps reduce hallucinations and improve accuracy, even with the base model. This model has been fine-tuned using the same prompt. |
|
|
|
### Training results |
|
|
|
- **Evaluation Results:** |
|
- *Before Fine-Tuning:* |
|
- WER: 127.29 |
|
- CER: 78.22 |
|
- *After Fine-Tuning:* |
|
- WER: 47.57 |
|
- CER: 20.52 |
|
|
|
- **Training Loss:** |
|
- Decreased from 1.423 to 0.176 |
|
|
|
## Inference |
|
|
|
Load `generation_config` and `processor` from the base model as a quick fix to use the default generation settings. |
|
|
|
*Note: The new models currently lack high-quality fine-tuning scripts. When saving a fine-tuned model using `model.save_pretrained()`, the processor configuration—including essential audio parameters—is not automatically saved. This omission can lead to errors during inference due to the model’s complex architecture. Loading these components from the base model ensures that all critical settings are properly included.* |
|
|
|
```python |
|
generation_config = GenerationConfig.from_pretrained( |
|
'microsoft/Phi-4-multimodal-instruct', 'generation_config.json' |
|
) |
|
processor = AutoProcessor.from_pretrained( |
|
'microsoft/Phi-4-multimodal-instruct', trust_remote_code=True |
|
) |
|
``` |
|
|
|
|
|
## Model description |
|
|
|
More information needed |
|
|
|
## Intended uses & limitations |
|
|
|
More information needed |
|
|
|
## Training and evaluation data |
|
|
|
More information needed |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 1e-05 |
|
- train_batch_size: 4 |
|
- eval_batch_size: 8 |
|
- seed: 42 |
|
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.95) and epsilon=1e-07 and optimizer_args=No additional optimizer arguments |
|
- lr_scheduler_type: linear |
|
- lr_scheduler_warmup_steps: 5000 |
|
- num_epochs: 1 |
|
- mixed_precision_training: Native AMP |
|
|
|
### Framework versions |
|
|
|
- Transformers 4.46.1 |
|
- Pytorch 2.5.1+cu124 |
|
- Datasets 3.3.2 |
|
- Tokenizers 0.20.3 |
