microsoft

/

Phi-4-multimodal-instruct

like 1.06k

Follow

Microsoft 10.1k

Thanks for sharing the fine-tuning script! I found a bug here. The .rstrip(ANSWER_SUFFIX) can remove extra characters at the end. I suggest to replace it with .replace(ANSWER_SUFFIX, "").
Example:

ipdb> processor.decode(inputs["labels"][1]).rstrip(ANSWER_SUFFIX)
'!!!!!!!!!!quite the opposi'
ipdb> processor.decode(inputs["labels"][1])
'!!!!!!!!!!quite the opposite<|end|><|endoftext|>'
ipdb> processor.decode(inputs["labels"][1]).replace(ANSWER_SUFFIX, "")
'!!!!!!!!!!quite the opposite'

Also, just a comment: for anyone wanting to use the Whisper normalizer for the predicted / reference text, here's one way to do it (data_utils from here)

        generated_text = [
            data_utils.normalizer(processor.decode(_pred_ids[inputs["input_ids"].shape[1] : _stop_tokens_idx],
                               skip_special_tokens=True,
                               clean_up_tokenization_spaces=False)) # add whisper normalizer
            for _pred_ids, _stop_tokens_idx in zip(generated_ids, stop_tokens_idx)
        ]
        all_generated_texts.extend(generated_text)

        labels = [data_utils.normalizer(processor.decode(_label_ids[_label_ids != 0]).replace(ANSWER_SUFFIX, "")) for _label_ids in inputs["labels"]]