rryisthebest
commited on
Update README.md
Browse files
README.md
CHANGED
@@ -5,13 +5,13 @@ license: mit
|
|
5 |
# Model Card for FIRST
|
6 |
|
7 |
<!-- Provide a quick summary of what the model is/does. -->
|
8 |
-
FIRST is a language models trained
|
9 |
|
10 |
### Model Description
|
11 |
|
12 |
<!-- Provide a longer summary of what this model is. -->
|
13 |
-
- **Model type:**
|
14 |
-
- **Language(s) (NLP):**
|
15 |
- **License:** MIT
|
16 |
- **Finetuned from model:** [HuggingFaceH4/zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta)
|
17 |
|
@@ -26,24 +26,13 @@ FIRST is a language models trained specifically for listwise reranking tasks, le
|
|
26 |
### Evaluations
|
27 |
|
28 |
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
|
29 |
-
At the time of release, FIRST demonstrates superior performance across a variety of reranking datasets. The table below provides a detailed performance comparison against other LLM rerankers on the BEIR benchmark.
|
30 |
| Reranker | Training Data | Avg. | Climate FEVER | DBPedia | FEVER | FiQA | Hotpot QA | MS Marco | NFCorpus | NQ | Sci-docs | Sci-fact | Trec-COVID |
|
31 |
|---------------|----------------|-------|---------------|---------|-------|-------|-----------|----------|----------|-------|----------|----------|------------|
|
32 |
| Rank Vicuna | GPT 3.5 | 50.7 | **28.2** | 50.0 | 81.0 | 35.9 | 73.5 | 36.7 | 33.1 | 58.6 | 18.4 | 70.5 | 71.3 |
|
33 |
| Rank Zephyr | GPT 3.5 + 3.5 | 53.7 | 25.6 | 50.0 | 80.1 | **42.2** | 71.6 | 42.7 | **37.7** | 65.6 | **20.5** | **76.7** | 78.4 |
|
34 |
| **FIRST** | GPT-4 | **54.3** | 26.7 | **50.9**| **81.7**| **42.2** | **74.2** | **44.4** | 37.4 | **66.4**| 20.4 | 74.6 | **78.8** |
|
35 |
|
36 |
-
|
37 |
-
More details can be found in the paper.
|
38 |
-
## Bias, Risks, and Limitations
|
39 |
-
|
40 |
-
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
|
41 |
-
We forward here an excerpt from the [Zephyr-7B-β model card](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta/blob/main/README.md#bias-risks--limitations):
|
42 |
-
|
43 |
-
> Zephyr-7B-β has not been aligned to human preferences for safety within the RLHF phase or deployed with in-the-loop filtering of responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so). It is also unknown what the size and composition of the corpus was used to train the base model (mistralai/Mistral-7B-v0.1), however it is likely to have included a mix of Web data and technical sources like books and code. See the [Falcon 180B model card](https://huggingface.co/tiiuae/falcon-180B#training-data) for an example of this.
|
44 |
-
|
45 |
-
FIRST is trained specifically on monolingual English data, effectiveness on multilingual sets is not guaranteed.
|
46 |
-
|
47 |
## Citation
|
48 |
|
49 |
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
|
|
|
5 |
# Model Card for FIRST
|
6 |
|
7 |
<!-- Provide a quick summary of what the model is/does. -->
|
8 |
+
FIRST is a language models trained for listwise reranking tasks leveraging the output logits of the first generated identifier to directly produce a ranked ordering of candidates. Built on the Zephyr-7B-β model, FIRST undergoes single-stage fine-tuning on a converted alphabetic version of the RankZephyr dataset (i,e RankGPT-4 reorderings of OpenAI's Ada2 orderings for 5k queries) More details can be found in the paper.
|
9 |
|
10 |
### Model Description
|
11 |
|
12 |
<!-- Provide a longer summary of what this model is. -->
|
13 |
+
- **Model type:** Fine-tuned on listwise reranking data from Zephyr-7B-β model.
|
14 |
+
- **Language(s) (NLP):** English
|
15 |
- **License:** MIT
|
16 |
- **Finetuned from model:** [HuggingFaceH4/zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta)
|
17 |
|
|
|
26 |
### Evaluations
|
27 |
|
28 |
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
|
29 |
+
At the time of release, FIRST demonstrates superior performance across a variety of reranking datasets. The table below provides a detailed performance comparison against other LLM rerankers on the BEIR benchmark. (More details can be found in the paper.)
|
30 |
| Reranker | Training Data | Avg. | Climate FEVER | DBPedia | FEVER | FiQA | Hotpot QA | MS Marco | NFCorpus | NQ | Sci-docs | Sci-fact | Trec-COVID |
|
31 |
|---------------|----------------|-------|---------------|---------|-------|-------|-----------|----------|----------|-------|----------|----------|------------|
|
32 |
| Rank Vicuna | GPT 3.5 | 50.7 | **28.2** | 50.0 | 81.0 | 35.9 | 73.5 | 36.7 | 33.1 | 58.6 | 18.4 | 70.5 | 71.3 |
|
33 |
| Rank Zephyr | GPT 3.5 + 3.5 | 53.7 | 25.6 | 50.0 | 80.1 | **42.2** | 71.6 | 42.7 | **37.7** | 65.6 | **20.5** | **76.7** | 78.4 |
|
34 |
| **FIRST** | GPT-4 | **54.3** | 26.7 | **50.9**| **81.7**| **42.2** | **74.2** | **44.4** | 37.4 | **66.4**| 20.4 | 74.6 | **78.8** |
|
35 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
36 |
## Citation
|
37 |
|
38 |
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
|