Post
1664
Discover all the improvements in the new version of Lighteval: https://huggingface.co/docs/lighteval/
AI & ML interests
The FEVER workshops are a venue for work in verifiable knowledge extraction and to stimulate progress in this direction.
Recent Activity
View all activity
fever's activity
albertvillanovaย
posted
an
update
6 days ago
andreasvlachosย
authored
a
paper
13 days ago
albertvillanovaย
posted
an
update
about 2 months ago
Post
1523
๐จ How green is your model? ๐ฑ Introducing a new feature in the Comparator tool: Environmental Impact for responsible #LLM research!
๐ open-llm-leaderboard/comparator
Now, you can not only compare models by performance, but also by their environmental footprint!
๐ The Comparator calculates COโ emissions during evaluation and shows key model characteristics: evaluation score, number of parameters, architecture, precision, type... ๐ ๏ธ
Make informed decisions about your model's impact on the planet and join the movement towards greener AI!
๐ open-llm-leaderboard/comparator
Now, you can not only compare models by performance, but also by their environmental footprint!
๐ The Comparator calculates COโ emissions during evaluation and shows key model characteristics: evaluation score, number of parameters, architecture, precision, type... ๐ ๏ธ
Make informed decisions about your model's impact on the planet and join the movement towards greener AI!
albertvillanovaย
posted
an
update
2 months ago
Post
1562
๐ New feature of the Comparator of the ๐ค Open LLM Leaderboard: now compare models with their base versions & derivatives (finetunes, adapters, etc.). Perfect for tracking how adjustments affect performance & seeing innovations in action. Dive deeper into the leaderboard!
๐ ๏ธ Here's how to use it:
1. Select your model from the leaderboard.
2. Load its model tree.
3. Choose any base & derived models (adapters, finetunes, merges, quantizations) for comparison.
4. Press Load.
See side-by-side performance metrics instantly!
Ready to dive in? ๐ Try the ๐ค Open LLM Leaderboard Comparator now! See how models stack up against their base versions and derivatives to understand fine-tuning and other adjustments. Easier model analysis for better insights! Check it out here: open-llm-leaderboard/comparator ๐
๐ ๏ธ Here's how to use it:
1. Select your model from the leaderboard.
2. Load its model tree.
3. Choose any base & derived models (adapters, finetunes, merges, quantizations) for comparison.
4. Press Load.
See side-by-side performance metrics instantly!
Ready to dive in? ๐ Try the ๐ค Open LLM Leaderboard Comparator now! See how models stack up against their base versions and derivatives to understand fine-tuning and other adjustments. Easier model analysis for better insights! Check it out here: open-llm-leaderboard/comparator ๐
albertvillanovaย
posted
an
update
3 months ago
Post
3141
๐ Exciting update! You can now compare multiple models side-by-side with the Hugging Face Open LLM Comparator! ๐
open-llm-leaderboard/comparator
Dive into multi-model evaluations, pinpoint the best model for your needs, and explore insights across top open LLMs all in one place. Ready to level up your model comparison game?
open-llm-leaderboard/comparator
Dive into multi-model evaluations, pinpoint the best model for your needs, and explore insights across top open LLMs all in one place. Ready to level up your model comparison game?
albertvillanovaย
posted
an
update
3 months ago
Post
1232
๐จ Instruct-tuning impacts models differently across families! Qwen2.5-72B-Instruct excels on IFEval but struggles with MATH-Hard, while Llama-3.1-70B-Instruct avoids MATH performance loss! Why? Can they follow the format in examples? ๐ Compare models:
open-llm-leaderboard/comparator
albertvillanovaย
posted
an
update
3 months ago
Post
1923
Finding the Best SmolLM for Your Project
Need an LLM assistant but unsure which hashtag#smolLM to run locally? With so many models available, how can you decide which one suits your needs best? ๐ค
If the model youโre interested in is evaluated on the Hugging Face Open LLM Leaderboard, thereโs an easy way to compare them: use the model Comparator tool: open-llm-leaderboard/comparator
Letโs walk through an example๐
Letโs compare two solid options:
- Qwen2.5-1.5B-Instruct from Alibaba Cloud Qwen (1.5B params)
- gemma-2-2b-it from Google (2.5B params)
For an assistant, you want a model thatโs great at instruction following. So, how do these two models stack up on the IFEval task?
What about other evaluations?
Both models are close in performance on many other tasks, showing minimal differences. Surprisingly, the 1.5B Qwen model performs just as well as the 2.5B Gemma in many areas, even though it's smaller in size! ๐
This is a great example of how parameter size isnโt everything. With efficient design and training, a smaller model like Qwen2.5-1.5B can match or even surpass larger models in certain tasks.
Looking for other comparisons? Drop your model suggestions below! ๐
Need an LLM assistant but unsure which hashtag#smolLM to run locally? With so many models available, how can you decide which one suits your needs best? ๐ค
If the model youโre interested in is evaluated on the Hugging Face Open LLM Leaderboard, thereโs an easy way to compare them: use the model Comparator tool: open-llm-leaderboard/comparator
Letโs walk through an example๐
Letโs compare two solid options:
- Qwen2.5-1.5B-Instruct from Alibaba Cloud Qwen (1.5B params)
- gemma-2-2b-it from Google (2.5B params)
For an assistant, you want a model thatโs great at instruction following. So, how do these two models stack up on the IFEval task?
What about other evaluations?
Both models are close in performance on many other tasks, showing minimal differences. Surprisingly, the 1.5B Qwen model performs just as well as the 2.5B Gemma in many areas, even though it's smaller in size! ๐
This is a great example of how parameter size isnโt everything. With efficient design and training, a smaller model like Qwen2.5-1.5B can match or even surpass larger models in certain tasks.
Looking for other comparisons? Drop your model suggestions below! ๐
albertvillanovaย
posted
an
update
3 months ago
Post
1959
๐จ Weโve just released a new tool to compare the performance of models in the ๐ค Open LLM Leaderboard: the Comparator ๐
open-llm-leaderboard/comparator
Want to see how two different versions of LLaMA stack up? Letโs walk through a step-by-step comparison of LLaMA-3.1 and LLaMA-3.2. ๐ฆ๐งต๐
1/ Load the Models' Results
- Go to the ๐ค Open LLM Leaderboard Comparator: open-llm-leaderboard/comparator
- Search for "LLaMA-3.1" and "LLaMA-3.2" in the model dropdowns.
- Press the Load button. Ready to dive into the results!
2/ Compare Metric Results in the Results Tab ๐
- Head over to the Results tab.
- Here, youโll see the performance metrics for each model, beautifully color-coded using a gradient to highlight performance differences: greener is better! ๐
- Want to focus on a specific task? Use the Task filter to hone in on comparisons for tasks like BBH or MMLU-Pro.
3/ Check Config Alignment in the Configs Tab โ๏ธ
- To ensure youโre comparing apples to apples, head to the Configs tab.
- Review both modelsโ evaluation configurations, such as metrics, datasets, prompts, few-shot configs...
- If something looks off, itโs good to know before drawing conclusions! โ
4/ Compare Predictions by Sample in the Details Tab ๐
- Curious about how each model responds to specific inputs? The Details tab is your go-to!
- Select a Task (e.g., MuSR) and then a Subtask (e.g., Murder Mystery) and then press the Load Details button.
- Check out the side-by-side predictions and dive into the nuances of each modelโs outputs.
5/ With this tool, itโs never been easier to explore how small changes between model versions affect performance on a wide range of tasks. Whether youโre a researcher or enthusiast, you can instantly visualize improvements and dive into detailed comparisons.
๐ Try the ๐ค Open LLM Leaderboard Comparator now and take your model evaluations to the next level!
open-llm-leaderboard/comparator
Want to see how two different versions of LLaMA stack up? Letโs walk through a step-by-step comparison of LLaMA-3.1 and LLaMA-3.2. ๐ฆ๐งต๐
1/ Load the Models' Results
- Go to the ๐ค Open LLM Leaderboard Comparator: open-llm-leaderboard/comparator
- Search for "LLaMA-3.1" and "LLaMA-3.2" in the model dropdowns.
- Press the Load button. Ready to dive into the results!
2/ Compare Metric Results in the Results Tab ๐
- Head over to the Results tab.
- Here, youโll see the performance metrics for each model, beautifully color-coded using a gradient to highlight performance differences: greener is better! ๐
- Want to focus on a specific task? Use the Task filter to hone in on comparisons for tasks like BBH or MMLU-Pro.
3/ Check Config Alignment in the Configs Tab โ๏ธ
- To ensure youโre comparing apples to apples, head to the Configs tab.
- Review both modelsโ evaluation configurations, such as metrics, datasets, prompts, few-shot configs...
- If something looks off, itโs good to know before drawing conclusions! โ
4/ Compare Predictions by Sample in the Details Tab ๐
- Curious about how each model responds to specific inputs? The Details tab is your go-to!
- Select a Task (e.g., MuSR) and then a Subtask (e.g., Murder Mystery) and then press the Load Details button.
- Check out the side-by-side predictions and dive into the nuances of each modelโs outputs.
5/ With this tool, itโs never been easier to explore how small changes between model versions affect performance on a wide range of tasks. Whether youโre a researcher or enthusiast, you can instantly visualize improvements and dive into detailed comparisons.
๐ Try the ๐ค Open LLM Leaderboard Comparator now and take your model evaluations to the next level!
albertvillanovaย
posted
an
update
4 months ago
Post
1527
Check out the new Structured #Wikipedia dataset by Wikimedia Enterprise: abstract, infobox, structured sections, main image,...
Currently in early beta (English & French). Explore it and give feedback: wikimedia/structured-wikipedia
More info: https://enterprise.wikimedia.com/blog/hugging-face-dataset/
@sdelbecque @resquito-wmf
Currently in early beta (English & French). Explore it and give feedback: wikimedia/structured-wikipedia
More info: https://enterprise.wikimedia.com/blog/hugging-face-dataset/
@sdelbecque @resquito-wmf
albertvillanovaย
posted
an
update
8 months ago
Post
2702
Easily convert your script-based datasets to Parquet and explore them in the dataset viewer. ๐
๐ ๏ธ Use @huggingface Datasets CLI:
$ ๐๐๐๐๐๐๐๐-๐๐๐ ๐๐๐๐๐๐๐_๐๐_๐๐๐๐๐๐๐ ๐๐๐ด๐๐ฝ๐ฐ๐ผ๐ด/๐ณ๐ฐ๐๐ฐ๐๐ด๐_๐ฝ๐ฐ๐ผ๐ด
Learn more: https://huggingface.co/docs/datasets/main/en/cli#convert-to-parquet
#Data #AI
๐ ๏ธ Use @huggingface Datasets CLI:
$ ๐๐๐๐๐๐๐๐-๐๐๐ ๐๐๐๐๐๐๐_๐๐_๐๐๐๐๐๐๐ ๐๐๐ด๐๐ฝ๐ฐ๐ผ๐ด/๐ณ๐ฐ๐๐ฐ๐๐ด๐_๐ฝ๐ฐ๐ผ๐ด
Learn more: https://huggingface.co/docs/datasets/main/en/cli#convert-to-parquet
#Data #AI
albertvillanovaย
posted
an
update
8 months ago
Post
4061
Recently, the Hugging Face ๐ค datasets team met with the Language Technologies team led by Marta Villegas (
@mvillegas
) at Barcelona Supercomputing Center
@BSC-LT
. Eager to collaborate to promote AI across Catalan, Spanish, Basque, and Galician languages and share open-source datasets/models. ๐ค #AI #LanguageTech #OpenSource
- 1 reply
albertvillanovaย
posted
an
update
9 months ago
Post
1663
๐ We recently released datasets 2.19.0! ๐ฆ
๐ฅ What's New:
- Polars integration ๐ปโโ๏ธ
- fsspec support for conversion to JSON, CSV, and Parquet
- Mode parameter for Image feature
- CLI function to convert script-datasets to Parquet
- Dataset.take and Dataset.skip
Plus, a bunch of general improvements & bug fixes!
Check out the release notes: https://github.com/huggingface/datasets/releases/tag/2.19.0
Upgrade now and power up your data workflows! ๐ฅ
๐ฅ What's New:
- Polars integration ๐ปโโ๏ธ
- fsspec support for conversion to JSON, CSV, and Parquet
- Mode parameter for Image feature
- CLI function to convert script-datasets to Parquet
- Dataset.take and Dataset.skip
Plus, a bunch of general improvements & bug fixes!
Check out the release notes: https://github.com/huggingface/datasets/releases/tag/2.19.0
Upgrade now and power up your data workflows! ๐ฅ
- 2 replies
albertvillanovaย
authored
3
papers
almost 2 years ago
Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements
Paper
โข
2210.01970
โข
Published
โข
11
Datasets: A Community Library for Natural Language Processing
Paper
โข
2109.02846
โข
Published
โข
11
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Paper
โข
2303.03915
โข
Published
โข
6
albertvillanovaย
updated
a
dataset
about 2 years ago