Introducing smolagents: simple agents that write actions in code.
โข
410
Aymeric Roucher
m-ric
AI & ML interests
Leading Agents at Hugging Face ๐ค
Recent Activity
commented
a paper
about 11 hours ago
OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse
Task Synthesis
posted
an
update
about 11 hours ago
๐ข๐ฆ-๐๐ฒ๐ป๐ฒ๐๐ถ๐: ๐ป๐ฒ๐ ๐ฟ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐ฝ๐ฎ๐ฝ๐ฒ๐ฟ ๐ฝ๐ฟ๐ผ๐ฝ๐ผ๐๐ฒ๐ ๐ฎ ๐ป๐ผ๐๐ฒ๐น ๐๐ฟ๐ฎ๐ถ๐ป๐ถ๐ป๐ด ๐ฑ๐ฎ๐๐ฎ ๐ด๐ฒ๐ป๐ฒ๐ฟ๐ฎ๐๐ถ๐ผ๐ป ๐บ๐ฒ๐๐ต๐ผ๐ฑ ๐ณ๐ผ๐ฟ ๐๐น๐ฎ๐๐ฑ๐ฒ-๐๐ผ๐บ๐ฝ๐๐๐ฒ๐ฟ-๐จ๐๐ฒ-๐น๐ถ๐ธ๐ฒ ๐ฎ๐ด๐ฒ๐ป๐๐, ๐๐ถ๐๐ต ๐ถ๐บ๐ฝ๐ฟ๐ฒ๐๐๐ถ๐๐ฒ ๐ฟ๐ฒ๐๐๐น๐๐! ๐ฅ
The main bottleneck in building GUI agents it to find training data.
GUI Agent trajectories are not easy to get by. Crowdsourcing trajectories, then manually annotating them, could be an option, but at scale, it's hard to do
You could use synthetic data generation (ask 1000s small existing GUI agents to solve tasks, keep only successful runs). But then it's hard to come up with many high level-tasks.
โก๏ธ Well, a novel technique was just published that creates a new promising paradigm for synthetic data generation: Shanghai AI Lab researchers propose OS-Genesis, a novel way to create training data for GUI agents that flips the traditional approach on its head. Instead of starting with predefined tasks and having humans or machines execute them, OS-Genesis first explores the interface naturally, then derives meaningful tasks from those interactions.
๐ Exploration-driven vs task-driven approach:
โฃ Instead of starting with tasks, OS-Genesis first explores GUIs by clicking and interacting
โฃ It then reverse-engineers high-level tasks from successful interaction patterns
โฃ This leads to more natural and diverse training data than predefined tasks
๐ฏ Novel reward model for trajectory quality:
โฃ Rather than discarding incomplete trajectories, OS-Genesis scores them based on coherence and completion
โฃ This preserves valuable partial successes that would otherwise be wasted
๐ Superior results across environments:
โฃ Nearly doubles performance on AndroidWorld (9.8% โ 17.4%)
By the way, this field of GUI agents is still in infancy, so you can still make a difference with "low-cost" setups: their paper gets SOTA results with only 8xA100!
Read the paper here ๐ https://huggingface.co/papers/2412.19723
upvoted
a
paper
about 13 hours ago
Agent Laboratory: Using LLM Agents as Research Assistants
Articles
Organizations
m-ric's activity
commented
a
paper
about 11 hours ago
Upload _versions.yml
1
#32 opened 14 days ago
by
m-ric
Upload processor
3
#14 opened about 1 month ago
by
m-ric
Upload processor
#13 opened about 1 month ago
by
m-ric
Upload processor
#3 opened about 1 month ago
by
m-ric
Upload AriaForConditionalGeneration
#2 opened about 1 month ago
by
m-ric
commented
a
paper
about 1 month ago
Upload processor
1
#12 opened about 1 month ago
by
m-ric
Upload AriaForConditionalGeneration
1
#11 opened about 1 month ago
by
m-ric
What are Meta-Llama-3.1-Instruct "Turbo" models?
3
#4 opened about 2 months ago
by
m-ric
commented
a
paper
2 months ago
is there a plan to support FLUX.1-schnell?
12
#28 opened 5 months ago
by
michaelj
commented
a
paper
2 months ago
commented
5
papers
3 months ago
Differential Transformer
Paper
โข
2410.05258
โข
Published
โข
169
โข
33
Aria: An Open Multimodal Native Mixture-of-Experts Model
Paper
โข
2410.05993
โข
Published
โข
108
โข
7
Addition is All You Need for Energy-efficient Language Models
Paper
โข
2410.00907
โข
Published
โข
145
โข
17
Were RNNs All We Needed?
Paper
โข
2410.01201
โข
Published
โข
51
โข
5
Emu3: Next-Token Prediction is All You Need
Paper
โข
2409.18869
โข
Published
โข
94
โข
9
commented
2
papers
4 months ago