--- language: - en tags: - falcon3 - falcon3_mamba - falcon_mamba base_model: - tiiuae/Falcon3-Mamba-7B-Base --- # Falcon3-Mamba-7B-Instruct **Falcon3** family of Open Foundation Models is a set of pretrained and instruct LLMs ranging from 1B to 10B. This repository contains the **Falcon3-Mamba-7B-Instruct**. It achieves, compared to similar SSM-based models of the same size, state of art results (at release's time) on reasoning, language understanding, instruction following, code and mathematics tasks. Falcon3-Mamba-7B-Instruct supports a context length up to 32K and was mainly trained on english corpus. ## Model Details - Architecture (same as [Falcon-Mamba-7b](https://huggingface.co/tiiuae/falcon-mamba-7b)) - Mamba1 based causal decoder only architecture trained on a causal language modeling task (i.e., predict the next token). - 64 decoder blocks - width: 4096 - state_size: 16 - 32k context length - 65k vocab size - Continue Pretrained from [Falcon Mamba 7B](https://huggingface.co/tiiuae/falcon-mamba-7b), with another 1500 Gigatokens of data comprising of web, code, STEM and high quality data. - Postrained on 1.2 million samples of STEM, conversations, code, and safety. - Developed by [Technology Innovation Institute](https://www.tii.ae) - License: TII Falcon-LLM License 2.0 - Model Release Date: December 2024 ## Getting started <details> <summary> Click to expand </summary> ```python from transformers import AutoTokenizer, AutoModelForCausalLM from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "tiiuae/Falcon3-Mamba-7B-Instruct" model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype="auto", device_map="auto" ) tokenizer = AutoTokenizer.from_pretrained(model_name) prompt = "How many hours in one day?" messages = [ {"role": "system", "content": "You are a helpful friendly assistant Falcon3 from TII, try to follow instructions as much as possible."}, {"role": "user", "content": prompt} ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) model_inputs = tokenizer([text], return_tensors="pt").to(model.device) generated_ids = model.generate( **model_inputs, max_new_tokens=1024 ) generated_ids = [ output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids) ] response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0] print(response) ``` </details> <br> # Benchmarks We report in the following table our internal pipeline benchmarks. For the benchmarks marked by star, we normalize the results with HuggingFace score normalization: <table border="1" style="width: 100%; text-align: center; border-collapse: collapse;"> <colgroup> <col style="width: 10%;"> <col style="width: 10%;"> <col style="width: 7%;"> <col style="width: 7%;"> <col style="width: 7%;"> <col style="background-color: rgba(80, 15, 213, 0.5); width: 7%;"> </colgroup> <thead> <tr> <th>Category</th> <th>Benchmark</th> <th>Zamba2-7B-instruct</th> <th>Jamba-1.5-Mini</th> <th>Llama-3.1-8B-Instruct</th> <th>Falcon3-Mamba-7B-Instruct</th> </tr> </thead> <tbody> <tr> <td rowspan="3">General</td> <td>MMLU (5-shot)</td> <td>30.6%</td> <td>68.7%</td> <td>55.9%</td> <td>65.3%</td> </tr> <tr> <td>MMLU-PRO (5-shot)*</td> <td>32.4%</td> <td>31.6%</td> <td>21.8%</td> <td>26.3%</td> </tr> <tr> <td>IFEval</td> <td>69.9%</td> <td>65.7%</td> <td>78.8%</td> <td>71.7%</td> </tr> <tr> <td rowspan="2">Math</td> <td>GSM8K (5-shot)</td> <td>0%</td> <td>74.9%</td> <td>19.2%</td> <td>65.2%</td> </tr> <tr> <td>MATH Lvl-5 (4-shot)</td> <td>13.6%</td> <td>6.9%</td> <td>10.4%</td> <td>27.3%</td> </tr> <tr> <td rowspan="4">Reasoning</td> <td>Arc Challenge (25-shot)</td> <td>54%</td> <td>54.3%</td> <td>46.6%</td> <td>53.7%</td> </tr> <tr> <td>GPQA (0-shot)*</td> <td>10.3%</td> <td>11.1%</td> <td>33.6%</td> <td>7.2%</td> </tr> <tr> <td>MUSR (0-shot)*</td> <td>8.2%</td> <td>12.2%</td> <td>38.6%</td> <td>8.3%</td> </tr> <tr> <td>BBH (3-shot)*</td> <td>33.3%</td> <td>35.3%</td> <td>43.7%</td> <td>25.2%</td> </tr> <tr> <td rowspan="4">CommonSense Understanding</td> <td>PIQA (0-shot)</td> <td>75.6%</td> <td>82.3%</td> <td>78.9%</td> <td>80.9%</td> </tr> <tr> <td>SciQ (0-shot)</td> <td>29.2%</td> <td>94.9%</td> <td>80.2%</td> <td>93.6%</td> </tr> <tr> <td>Winogrande (0-shot)</td> <td>75.9%</td> <td>64.5%</td> <td>-</td> <td>-</td> </tr> <tr> <td>OpenbookQA (0-shot)</td> <td>45.6%</td> <td>34.6%</td> <td>46.2%</td> <td>47.2%</td> </tr> </tbody> </table> ## Useful links - View our [release blogpost](https://huggingface.co/blog/falcon3). - Feel free to join [our discord server](https://discord.gg/fwXpMyGc) if you have any questions or to interact with our researchers and developers. ## Citation If the Falcon3 family of models were helpful to your work, feel free to give us a cite. ``` @misc{Falcon3, title = {The Falcon 3 Family of Open Models}, author = {Falcon-LLM Team}, month = {December}, year = {2024} } ```