github/toon
1
0
Fork 0
mirror of https://github.com/voson-wang/toon.git synced 2026-01-29 23:34:10 +08:00
Code Issues Packages Projects Releases Wiki Activity

test: update retrieval accuracy benchmarks

Browse Source
This commit is contained in:
Johann Schopplich
2025-10-27 13:45:48 +01:00
parent b2c58d2b97
commit 1a5e6199ac
10 changed files with 5686 additions and 5709 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10976
benchmarks/results/accuracy/raw-results.json
View File

File diff suppressed because it is too large Load Diff

77
benchmarks/results/accuracy/report.md
View File

@@ -3,19 +3,22 @@
Tested across **2 LLMs** with data retrieval tasks:
```
gpt-4o-mini ██████████████░░░░░░ 72.3% accuracy
claude-haiku-4-5 ███████████████░░░░░ 76.7% accuracy
gpt-5-nano
toon ███████████████████░ 97.5% (155/159)
markdown-kv ███████████████████░ 95.6% (152/159)
yaml ███████████████████░ 94.3% (150/159)
json ███████████████████░ 93.7% (149/159)
csv ███████████████████░ 93.7% (149/159)
claude-haiku-4-5
markdown-kv ███████████████░░░░░ 76.7% (122/159)
toon ███████████████░░░░░ 75.5% (120/159)
json ███████████████░░░░░ 75.5% (120/159)
csv ███████████████░░░░░ 75.5% (120/159)
yaml ███████████████░░░░░ 74.8% (119/159)
```
**TOON achieves 73.9% accuracy (vs JSON's 73.6%) while using 46.3% fewer tokens.**
| Format | Accuracy | Average Tokens |
| ------ | -------- | -------------- |
| `toon` | 73.9% | 4.678 |
| `json` | 73.6% | 8.713 |
| `markdown-kv` | 73.6% | 8.649 |
| `csv` | 72.3% | 4.745 |
| `yaml` | 71.7% | 7.091 |
**Tradeoff:** TOON achieves 86.5% accuracy (vs JSON's 84.6%) while using 46.3% fewer tokens.
<details>
<summary><strong>View detailed breakdown by dataset and model</strong></summary>
@@ -26,53 +29,53 @@ claude-haiku-4-5 ███████████████░░░░
| Format | Accuracy | Tokens | Correct/Total |
|--------|----------|--------|---------------|
| `toon` | 72.4% | 2.483 | 84/116 |
| `csv` | 69.0% | 2.337 | 80/116 |
| `yaml` | 68.1% | 4.969 | 79/116 |
| `markdown-kv` | 68.1% | 6.270 | 79/116 |
| `json` | 68.1% | 6.347 | 79/116 |
| `toon` | 86.2% | 2.483 | 100/116 |
| `csv` | 80.2% | 2.337 | 93/116 |
| `yaml` | 82.8% | 4.969 | 96/116 |
| `markdown-kv` | 84.5% | 6.270 | 98/116 |
| `json` | 84.5% | 6.347 | 98/116 |
##### E-commerce orders with nested structures
| Format | Accuracy | Tokens | Correct/Total |
|--------|----------|--------|---------------|
| `toon` | 84.1% | 5.967 | 74/88 |
| `csv` | 83.0% | 6.735 | 73/88 |
| `yaml` | 81.8% | 7.328 | 72/88 |
| `markdown-kv` | 86.4% | 9.110 | 76/88 |
| `json` | 84.1% | 9.694 | 74/88 |
| `toon` | 90.9% | 5.967 | 80/88 |
| `csv` | 90.9% | 6.735 | 80/88 |
| `yaml` | 89.8% | 7.328 | 79/88 |
| `markdown-kv` | 90.9% | 9.110 | 80/88 |
| `json` | 89.8% | 9.694 | 79/88 |
##### Time-series analytics data
| Format | Accuracy | Tokens | Correct/Total |
|--------|----------|--------|---------------|
| `csv` | 72.4% | 1.393 | 42/58 |
| `toon` | 70.7% | 1.515 | 41/58 |
| `yaml` | 72.4% | 2.938 | 42/58 |
| `json` | 74.1% | 3.665 | 43/58 |
| `markdown-kv` | 70.7% | 3.779 | 41/58 |
| `csv` | 87.9% | 1.393 | 51/58 |
| `toon` | 86.2% | 1.515 | 50/58 |
| `yaml` | 86.2% | 2.938 | 50/58 |
| `json` | 87.9% | 3.665 | 51/58 |
| `markdown-kv` | 86.2% | 3.779 | 50/58 |
##### Popular GitHub repositories
| Format | Accuracy | Tokens | Correct/Total |
|--------|----------|--------|---------------|
| `toon` | 64.3% | 8.745 | 36/56 |
| `csv` | 62.5% | 8.513 | 35/56 |
| `json` | 67.9% | 15.145 | 38/56 |
| `markdown-kv` | 67.9% | 15.436 | 38/56 |
| `yaml` | 62.5% | 13.129 | 35/56 |
| `csv` | 80.4% | 8.513 | 45/56 |
| `toon` | 80.4% | 8.745 | 45/56 |
| `yaml` | 78.6% | 13.129 | 44/56 |
| `markdown-kv` | 82.1% | 15.436 | 46/56 |
| `json` | 73.2% | 15.145 | 41/56 |
#### Performance by Model
##### gpt-4o-mini
##### gpt-5-nano
| Format | Accuracy | Correct/Total |
|--------|----------|---------------|
| `toon` | 72.3% | 115/159 |
| `json` | 71.7% | 114/159 |
| `markdown-kv` | 70.4% | 112/159 |
| `csv` | 69.2% | 110/159 |
| `yaml` | 68.6% | 109/159 |
| `toon` | 97.5% | 155/159 |
| `markdown-kv` | 95.6% | 152/159 |
| `yaml` | 94.3% | 150/159 |
| `json` | 93.7% | 149/159 |
| `csv` | 93.7% | 149/159 |
##### claude-haiku-4-5

49
benchmarks/results/accuracy/summary.json
View File

@@ -2,53 +2,48 @@
"formatResults": [
{
"format": "toon",
"accuracy": 0.7389937106918238,
"accuracy": 0.8647798742138365,
"totalTokens": 4678,
"avgInputTokens": 4675,
"avgLatency": 1424,
"correctCount": 235,
"totalCount": 318
},
{
"format": "json",
"accuracy": 0.7358490566037735,
"totalTokens": 8713,
"avgInputTokens": 9177,
"avgLatency": 1678,
"correctCount": 234,
"averageLatency": 5016,
"correctCount": 275,
"totalCount": 318
},
{
"format": "markdown-kv",
"accuracy": 0.7358490566037735,
"accuracy": 0.8616352201257862,
"totalTokens": 8649,
"avgInputTokens": 8242,
"avgLatency": 1724,
"correctCount": 234,
"averageLatency": 4628,
"correctCount": 274,
"totalCount": 318
},
{
"format": "json",
"accuracy": 0.8459119496855346,
"totalTokens": 8713,
"averageLatency": 5369,
"correctCount": 269,
"totalCount": 318
},
{
"format": "csv",
"accuracy": 0.7232704402515723,
"accuracy": 0.8459119496855346,
"totalTokens": 4745,
"avgInputTokens": 4878,
"avgLatency": 1573,
"correctCount": 230,
"averageLatency": 5168,
"correctCount": 269,
"totalCount": 318
},
{
"format": "yaml",
"accuracy": 0.7169811320754716,
"accuracy": 0.8459119496855346,
"totalTokens": 7091,
"avgInputTokens": 7136,
"avgLatency": 1602,
"correctCount": 228,
"averageLatency": 4299,
"correctCount": 269,
"totalCount": 318
}
],
"questions": 159,
"models": [
"gpt-4o-mini",
"gpt-5-nano",
"claude-haiku-4-5"
],
"datasets": [
@@ -91,5 +86,5 @@
"yaml-analytics": 2938,
"yaml-github": 13129
},
"timestamp": "2025-10-27T10:46:35.127Z"
"timestamp": "2025-10-27T12:43:38.288Z"
}