COMET-partial

This model is based on COMET-early-exit, which is a fork but not compatible with original Unbabel's COMET. To run the model, you need to first install this version of COMET either with:

pip install "git+https://github.com/zouharvi/COMET-early-exit#egg=comet-early-exit&subdirectory=comet_early_exit"

or in editable mode:

git clone https://github.com/zouharvi/COMET-early-exit.git
cd COMET-early-exit
pip3 install -e comet_early_exit

This model is described in the appendix in the paper. It is able to score even incomplete translations (i.e. prefixes of translations):

import comet_early_exit
model = comet_early_exit.load_from_checkpoint(comet_early_exit.download_model("zouharvi/COMET-partial"))
data = [
    {
        "src": "I want to receive my food in 10 to 15 minutes.",
        "mt": "Ich werde",
    },
    {
        "src": "I want to receive my food in 10 to 15 minutes.",
        "mt": "Ich möchte",
    },
    {
        "src": "I want to receive my food in 10 to 15 minutes.",
        "mt": "Ich möchte mein Essen in",
    },
    {
        "src": "I want to receive my food in 10 to 15 minutes.",
        "mt": "Ich möchte mein Essen in 10 bis 15 Minuten erhalten.",
    },
    {
        "src": "I want to receive my food in 10 to 15 minutes.",
        "mt": "Ich möchte mein Essen in 10 bis 15 Minuten bekommen.",
    }
]
model_output = model.predict(data, batch_size=8, gpus=1)
print("scores", model_output["scores"])

Outputs (formatted):

scores 89.26  89.45  89.51  89.48  89.66

This model is based on the work Early-Exit and Instant Confidence Translation Quality Estimation which can be cited as:

@misc{zouhar2025earlyexitinstantconfidencetranslation,
      title={Early-Exit and Instant Confidence Translation Quality Estimation}, 
      author={Vilém Zouhar and Maike Züfle and Beni Egressy and Julius Cheng and Jan Niehues},
      year={2025},
      eprint={2502.14429},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.14429}, 
}