mirror of
https://github.com/voson-wang/toon.git
synced 2026-01-29 15:24:10 +08:00
docs: add benchmarks for gemini-2.5-flash
This commit is contained in:
Johann Schopplich
File diff suppressed because it is too large
Load Diff
@@ -1,24 +1,31 @@
|
||||
### Retrieval Accuracy
|
||||
|
||||
Tested across **2 LLMs** with data retrieval tasks:
|
||||
Tested across **3 LLMs** with data retrieval tasks:
|
||||
|
||||
```
|
||||
gpt-5-nano
|
||||
toon ███████████████████░ 97.5% (155/159)
|
||||
markdown-kv ███████████████████░ 95.6% (152/159)
|
||||
yaml ███████████████████░ 94.3% (150/159)
|
||||
json ███████████████████░ 93.7% (149/159)
|
||||
csv ███████████████████░ 93.7% (149/159)
|
||||
toon ████████████████████ 99.4% (158/159)
|
||||
yaml ███████████████████░ 95.0% (151/159)
|
||||
csv ██████████████████░░ 92.5% (147/159)
|
||||
json ██████████████████░░ 92.5% (147/159)
|
||||
xml ██████████████████░░ 91.2% (145/159)
|
||||
|
||||
claude-haiku-4-5
|
||||
markdown-kv ███████████████░░░░░ 76.7% (122/159)
|
||||
toon ███████████████░░░░░ 75.5% (120/159)
|
||||
json ███████████████░░░░░ 75.5% (120/159)
|
||||
xml ███████████████░░░░░ 75.5% (120/159)
|
||||
csv ███████████████░░░░░ 75.5% (120/159)
|
||||
yaml ███████████████░░░░░ 74.8% (119/159)
|
||||
json ███████████████░░░░░ 75.5% (120/159)
|
||||
yaml ███████████████░░░░░ 74.2% (118/159)
|
||||
|
||||
gemini-2.5-flash
|
||||
xml ██████████████████░░ 91.8% (146/159)
|
||||
csv █████████████████░░░ 86.2% (137/159)
|
||||
toon █████████████████░░░ 84.9% (135/159)
|
||||
json ████████████████░░░░ 81.8% (130/159)
|
||||
yaml ████████████████░░░░ 78.6% (125/159)
|
||||
```
|
||||
|
||||
**Advantage:** TOON achieves **86.5% accuracy** (vs JSON's 84.6%) while using **46.3% fewer tokens**.
|
||||
**Advantage:** TOON achieves **86.6% accuracy** (vs JSON's 83.2%) while using **46.3% fewer tokens**.
|
||||
|
||||
<details>
|
||||
<summary><strong>View detailed breakdown by dataset and model</strong></summary>
|
||||
@@ -29,41 +36,41 @@ claude-haiku-4-5
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
| ------ | -------- | ------ | ------------- |
|
||||
| `toon` | 86.2% | 2.483 | 100/116 |
|
||||
| `csv` | 80.2% | 2.337 | 93/116 |
|
||||
| `yaml` | 82.8% | 4.969 | 96/116 |
|
||||
| `markdown-kv` | 84.5% | 6.270 | 98/116 |
|
||||
| `json` | 84.5% | 6.347 | 98/116 |
|
||||
| `toon` | 87.4% | 2.483 | 152/174 |
|
||||
| `csv` | 82.8% | 2.337 | 144/174 |
|
||||
| `yaml` | 83.9% | 4.969 | 146/174 |
|
||||
| `json` | 83.9% | 6.347 | 146/174 |
|
||||
| `xml` | 88.5% | 7.314 | 154/174 |
|
||||
|
||||
##### E-commerce orders with nested structures
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
| ------ | -------- | ------ | ------------- |
|
||||
| `toon` | 90.9% | 5.967 | 80/88 |
|
||||
| `csv` | 90.9% | 6.735 | 80/88 |
|
||||
| `yaml` | 89.8% | 7.328 | 79/88 |
|
||||
| `markdown-kv` | 90.9% | 9.110 | 80/88 |
|
||||
| `json` | 89.8% | 9.694 | 79/88 |
|
||||
| `toon` | 90.9% | 5.967 | 120/132 |
|
||||
| `csv` | 93.9% | 6.735 | 124/132 |
|
||||
| `yaml` | 87.1% | 7.328 | 115/132 |
|
||||
| `json` | 87.9% | 9.694 | 116/132 |
|
||||
| `xml` | 93.2% | 10.992 | 123/132 |
|
||||
|
||||
##### Time-series analytics data
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
| ------ | -------- | ------ | ------------- |
|
||||
| `csv` | 87.9% | 1.393 | 51/58 |
|
||||
| `toon` | 86.2% | 1.515 | 50/58 |
|
||||
| `yaml` | 86.2% | 2.938 | 50/58 |
|
||||
| `json` | 87.9% | 3.665 | 51/58 |
|
||||
| `markdown-kv` | 86.2% | 3.779 | 50/58 |
|
||||
| `csv` | 89.7% | 1.393 | 78/87 |
|
||||
| `toon` | 88.5% | 1.515 | 77/87 |
|
||||
| `yaml` | 83.9% | 2.938 | 73/87 |
|
||||
| `json` | 88.5% | 3.665 | 77/87 |
|
||||
| `xml` | 85.1% | 4.376 | 74/87 |
|
||||
|
||||
##### Top 100 GitHub repositories
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
| ------ | -------- | ------ | ------------- |
|
||||
| `csv` | 80.4% | 8.513 | 45/56 |
|
||||
| `toon` | 80.4% | 8.745 | 45/56 |
|
||||
| `yaml` | 78.6% | 13.129 | 44/56 |
|
||||
| `markdown-kv` | 82.1% | 15.436 | 46/56 |
|
||||
| `json` | 73.2% | 15.145 | 41/56 |
|
||||
| `toon` | 76.2% | 8.745 | 64/84 |
|
||||
| `csv` | 69.0% | 8.513 | 58/84 |
|
||||
| `yaml` | 71.4% | 13.129 | 60/84 |
|
||||
| `json` | 69.0% | 15.145 | 58/84 |
|
||||
| `xml` | 71.4% | 17.095 | 60/84 |
|
||||
|
||||
#### Performance by Model
|
||||
|
||||
@@ -71,27 +78,37 @@ claude-haiku-4-5
|
||||
|
||||
| Format | Accuracy | Correct/Total |
|
||||
| ------ | -------- | ------------- |
|
||||
| `toon` | 97.5% | 155/159 |
|
||||
| `markdown-kv` | 95.6% | 152/159 |
|
||||
| `yaml` | 94.3% | 150/159 |
|
||||
| `json` | 93.7% | 149/159 |
|
||||
| `csv` | 93.7% | 149/159 |
|
||||
| `toon` | 99.4% | 158/159 |
|
||||
| `yaml` | 95.0% | 151/159 |
|
||||
| `csv` | 92.5% | 147/159 |
|
||||
| `json` | 92.5% | 147/159 |
|
||||
| `xml` | 91.2% | 145/159 |
|
||||
|
||||
##### claude-haiku-4-5
|
||||
|
||||
| Format | Accuracy | Correct/Total |
|
||||
| ------ | -------- | ------------- |
|
||||
| `markdown-kv` | 76.7% | 122/159 |
|
||||
| `toon` | 75.5% | 120/159 |
|
||||
| `json` | 75.5% | 120/159 |
|
||||
| `xml` | 75.5% | 120/159 |
|
||||
| `csv` | 75.5% | 120/159 |
|
||||
| `yaml` | 74.8% | 119/159 |
|
||||
| `json` | 75.5% | 120/159 |
|
||||
| `yaml` | 74.2% | 118/159 |
|
||||
|
||||
##### gemini-2.5-flash
|
||||
|
||||
| Format | Accuracy | Correct/Total |
|
||||
| ------ | -------- | ------------- |
|
||||
| `xml` | 91.8% | 146/159 |
|
||||
| `csv` | 86.2% | 137/159 |
|
||||
| `toon` | 84.9% | 135/159 |
|
||||
| `json` | 81.8% | 130/159 |
|
||||
| `yaml` | 78.6% | 125/159 |
|
||||
|
||||
#### Methodology
|
||||
|
||||
- **Semantic validation**: LLM-as-judge validates responses semantically (not exact string matching).
|
||||
- **Token counting**: Using `gpt-tokenizer` with `o200k_base` encoding.
|
||||
- **Question types**: Field retrieval, aggregation, and filtering tasks.
|
||||
- **Real data**: Faker.js-generated datasets + GitHub repositories.
|
||||
- **Question types**: ~160 questions across field retrieval, aggregation, and filtering tasks.
|
||||
- **Datasets**: Faker.js-generated datasets (seeded) + GitHub repositories.
|
||||
|
||||
</details>
|
||||
|
||||
@@ -2,49 +2,50 @@
|
||||
"formatResults": [
|
||||
{
|
||||
"format": "toon",
|
||||
"accuracy": 0.8647798742138365,
|
||||
"accuracy": 0.8658280922431866,
|
||||
"totalTokens": 4678,
|
||||
"averageLatency": 5016,
|
||||
"correctCount": 275,
|
||||
"totalCount": 318
|
||||
"averageLatency": 5321,
|
||||
"correctCount": 413,
|
||||
"totalCount": 477
|
||||
},
|
||||
{
|
||||
"format": "markdown-kv",
|
||||
"format": "xml",
|
||||
"accuracy": 0.8616352201257862,
|
||||
"totalTokens": 8649,
|
||||
"averageLatency": 4628,
|
||||
"correctCount": 274,
|
||||
"totalCount": 318
|
||||
},
|
||||
{
|
||||
"format": "json",
|
||||
"accuracy": 0.8459119496855346,
|
||||
"totalTokens": 8713,
|
||||
"averageLatency": 5369,
|
||||
"correctCount": 269,
|
||||
"totalCount": 318
|
||||
"totalTokens": 9944,
|
||||
"averageLatency": 6035,
|
||||
"correctCount": 411,
|
||||
"totalCount": 477
|
||||
},
|
||||
{
|
||||
"format": "csv",
|
||||
"accuracy": 0.8459119496855346,
|
||||
"accuracy": 0.8469601677148847,
|
||||
"totalTokens": 4745,
|
||||
"averageLatency": 5168,
|
||||
"correctCount": 269,
|
||||
"totalCount": 318
|
||||
"averageLatency": 6551,
|
||||
"correctCount": 404,
|
||||
"totalCount": 477
|
||||
},
|
||||
{
|
||||
"format": "json",
|
||||
"accuracy": 0.8322851153039832,
|
||||
"totalTokens": 8713,
|
||||
"averageLatency": 7981,
|
||||
"correctCount": 397,
|
||||
"totalCount": 477
|
||||
},
|
||||
{
|
||||
"format": "yaml",
|
||||
"accuracy": 0.8459119496855346,
|
||||
"accuracy": 0.8259958071278826,
|
||||
"totalTokens": 7091,
|
||||
"averageLatency": 4299,
|
||||
"correctCount": 269,
|
||||
"totalCount": 318
|
||||
"averageLatency": 5561,
|
||||
"correctCount": 394,
|
||||
"totalCount": 477
|
||||
}
|
||||
],
|
||||
"questions": 159,
|
||||
"models": [
|
||||
"gpt-5-nano",
|
||||
"claude-haiku-4-5"
|
||||
"claude-haiku-4-5",
|
||||
"gemini-2.5-flash"
|
||||
],
|
||||
"datasets": [
|
||||
{
|
||||
@@ -77,14 +78,14 @@
|
||||
"csv-nested": 6735,
|
||||
"csv-analytics": 1393,
|
||||
"csv-github": 8513,
|
||||
"markdown-kv-tabular": 6270,
|
||||
"markdown-kv-nested": 9110,
|
||||
"markdown-kv-analytics": 3779,
|
||||
"markdown-kv-github": 15436,
|
||||
"xml-tabular": 7314,
|
||||
"xml-nested": 10992,
|
||||
"xml-analytics": 4376,
|
||||
"xml-github": 17095,
|
||||
"yaml-tabular": 4969,
|
||||
"yaml-nested": 7328,
|
||||
"yaml-analytics": 2938,
|
||||
"yaml-github": 13129
|
||||
},
|
||||
"timestamp": "2025-10-27T13:17:28.071Z"
|
||||
"timestamp": "2025-10-27T15:01:57.523Z"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user