Create README.md

README.md

@@ -0,0 +1,76 @@
+---
+license: llama3.1
+datasets:
+- allenai/c4
+language:
+- en
+metrics:
+- accuracy
+base_model:
+- meta-llama/Llama-3.1-8B-Instruct
+---
+# Language Model Evaluation Results
+
+## Overview
+This document presents the evaluation results of `Llama-3.1-8B-Instruct-gptq-4bit` using the **Language Model Evaluation Harness** on the **ARC-Challenge** benchmark.
+
+---
+
+## 📊 Evaluation Summary
+
+| **Metric**            | **Value**  | **Description**  |
+|----------------------|-----------|-----------------|
+| **Accuracy (acc,none)** | `47.1%`  | Raw accuracy - percentage of correct answers. |
+| **Standard Error (acc_stderr,none)** | `1.46%` | Uncertainty in the accuracy estimate. |
+| **Normalized Accuracy (acc_norm,none)** | `49.9%`  | Accuracy after dataset-specific normalization. |
+| **Standard Error (acc_norm_stderr,none)** | `1.46%` | Uncertainty for normalized accuracy. |
+
+📌 **Interpretation:**
+- The model correctly answered **47.1% of the questions**.
+- After **normalization**, the accuracy slightly improves to **49.9%**.
+- The **standard error (~1.46%)** indicates a small margin of uncertainty.
+
+---
+
+## ⚙️ Model Configuration
+
+- **Model:** `Llama-3.1-8B-Instruct-gptq-4bit`
+- **Parameters:** `1.05 billion` (Quantized 4-bit model)
+- **Source:** Hugging Face (`hf`)
+- **Precision:** `torch.float16`
+- **Hardware:** `NVIDIA A100 80GB PCIe`
+- **CUDA Version:** `12.4`
+- **PyTorch Version:** `2.6.0+cu124`
+- **Batch Size:** `1`
+- **Evaluation Time:** `365.89 seconds (~6 minutes)`
+
+📌 **Interpretation:**
+- The evaluation was performed on a **high-performance GPU (A100 80GB)**.
+- The model is **4-bit quantized**, reducing memory usage but possibly affecting accuracy.
+- A **single-sample batch size** was used, which might slow evaluation speed.
+
+---
+
+## 📂 Dataset Information
+
+- **Dataset:** `AI2 ARC-Challenge`
+- **Task Type:** `Multiple Choice`
+- **Number of Samples Evaluated:** `1,172`
+- **Few-shot Examples Used:** `0` (Zero-shot setting)
+
+📌 **Interpretation:**
+- This benchmark assesses **grade-school-level scientific reasoning**.
+- Since **no few-shot examples** were provided, the model was evaluated in a **pure zero-shot setting**.
+
+---
+
+## 📈 Performance Insights
+
+- The `"higher_is_better"` flag confirms that **higher accuracy is preferred**.
+- The model's **raw accuracy (47.1%)** is moderate compared to state-of-the-art models (**60–80%** on ARC-Challenge).
+- **Quantization Impact:** The **4-bit quantized model** might perform slightly worse than a full-precision version.
+- **Zero-shot Limitation:** Performance could improve with **few-shot prompting** (providing examples before testing).
+
+---
+
+📌 Let us know if you need further analysis or model tuning! 🚀