github/toon
1
0
Fork 0
mirror of https://github.com/voson-wang/toon.git synced 2026-01-29 23:34:10 +08:00
Code Issues Packages Projects Releases Wiki Activity

docs: add benchmarks for gemini-2.5-flash

Browse Source
This commit is contained in:
Johann Schopplich
2025-10-27 16:02:51 +01:00
parent 77696ce932
commit 7b76acde31
10 changed files with 15837 additions and 7011 deletions
Download Patch File Download Diff File Expand all files Collapse all files

194
README.md
View File

@@ -55,13 +55,23 @@ I built TOON to save tokens when sending large datasets to LLMs at work, where I
### Token Efficiency ### Token Efficiency
``` ```
⭐ GitHub Repositories ██████████████░░░░░░░░░░░ 8,745 tokens (JSON: 15,145) 💰 42.3% saved ⭐ GitHub Repositories ██████████████░░░░░░░░░░░ 8,745 tokens
📈 Daily Analytics ██████████░░░░░░░░░░░░░░░ 3,630 tokens (JSON: 9,023) 💰 59.8% saved vs JSON: 15,145 💰 42.3% saved
👥 API Response ██████████████░░░░░░░░░░░ 2,597 tokens (JSON: 4,589) 💰 43.4% saved vs XML: 17,095 💰 48.8% saved
🛒 E-Commerce Order ████████████████░░░░░░░░░ 164 tokens (JSON: 256) 💰 35.9% saved
```
**Total:** 15,136 tokens (TOON) vs 29,013 tokens (JSON) → 47.8% savings 📈 Daily Analytics ██████████░░░░░░░░░░░░░░░ 4,507 tokens
vs JSON: 10,977 💰 58.9% saved
vs XML: 13,128 💰 65.7% saved
🛒 E-Commerce Order ████████████████░░░░░░░░░ 166 tokens
vs JSON: 257 💰 35.4% saved
vs XML: 271 💰 38.7% saved
─────────────────────────────────────────────────────────────────────
Total ████████████░░░░░░░░░░░░░ 13,418 tokens
vs JSON: 26,379 💰 49.1% saved
vs XML: 30,494 💰 56.0% saved
```
<details> <details>
<summary><strong>View detailed examples</strong></summary> <summary><strong>View detailed examples</strong></summary>
@@ -70,7 +80,7 @@ I built TOON to save tokens when sending large datasets to LLMs at work, where I
**Configuration:** Top 100 GitHub repositories with stars, forks, and metadata **Configuration:** Top 100 GitHub repositories with stars, forks, and metadata
**Savings:** 6,400 tokens (42.3% reduction) **Savings:** 6,400 tokens (42.3% reduction vs JSON)
**JSON** (15,145 tokens): **JSON** (15,145 tokens):
@@ -81,7 +91,7 @@ I built TOON to save tokens when sending large datasets to LLMs at work, where I
"id": 28457823, "id": 28457823,
"name": "freeCodeCamp", "name": "freeCodeCamp",
"repo": "freeCodeCamp/freeCodeCamp", "repo": "freeCodeCamp/freeCodeCamp",
"description": "freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,...", "description": "freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…",
"createdAt": "2014-12-24T17:49:19Z", "createdAt": "2014-12-24T17:49:19Z",
"updatedAt": "2025-10-27T07:40:58Z", "updatedAt": "2025-10-27T07:40:58Z",
"pushedAt": "2025-10-26T11:31:08Z", "pushedAt": "2025-10-26T11:31:08Z",
@@ -124,7 +134,7 @@ I built TOON to save tokens when sending large datasets to LLMs at work, where I
``` ```
repositories[3]{id,name,repo,description,createdAt,updatedAt,pushedAt,stars,watchers,forks,defaultBranch}: repositories[3]{id,name,repo,description,createdAt,updatedAt,pushedAt,stars,watchers,forks,defaultBranch}:
28457823,freeCodeCamp,freeCodeCamp/freeCodeCamp,"freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,...","2014-12-24T17:49:19Z","2025-10-27T07:40:58Z","2025-10-26T11:31:08Z",430828,8582,42136,main 28457823,freeCodeCamp,freeCodeCamp/freeCodeCamp,"freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…","2014-12-24T17:49:19Z","2025-10-27T07:40:58Z","2025-10-26T11:31:08Z",430828,8582,42136,main
132750724,build-your-own-x,codecrafters-io/build-your-own-x,Master programming by recreating your favorite technologies from scratch.,"2018-05-09T12:03:18Z","2025-10-27T07:43:25Z","2025-10-10T18:45:01Z",430102,6322,40388,master 132750724,build-your-own-x,codecrafters-io/build-your-own-x,Master programming by recreating your favorite technologies from scratch.,"2018-05-09T12:03:18Z","2025-10-27T07:43:25Z","2025-10-10T18:45:01Z",430102,6322,40388,master
21737465,awesome,sindresorhus/awesome,😎 Awesome lists about all kinds of interesting topics,"2014-07-11T13:42:37Z","2025-10-27T07:44:27Z","2025-10-23T17:26:53Z",409760,8016,32015,main 21737465,awesome,sindresorhus/awesome,😎 Awesome lists about all kinds of interesting topics,"2014-07-11T13:42:37Z","2025-10-27T07:44:27Z","2025-10-23T17:26:53Z",409760,8016,32015,main
``` ```
@@ -135,61 +145,66 @@ repositories[3]{id,name,repo,description,createdAt,updatedAt,pushedAt,stars,watc
**Configuration:** 180 days of web metrics (views, clicks, conversions, revenue) **Configuration:** 180 days of web metrics (views, clicks, conversions, revenue)
**Savings:** 5,393 tokens (59.8% reduction) **Savings:** 6,470 tokens (58.9% reduction vs JSON)
**JSON** (9,023 tokens): **JSON** (10,977 tokens):
```json ```json
{ {
"metrics": [ "metrics": [
{
"date": "2024-12-31",
"views": 1953,
"clicks": 224,
"conversions": 60,
"revenue": 409.79
},
{ {
"date": "2025-01-01", "date": "2025-01-01",
"views": 2981, "views": 6890,
"clicks": 242, "clicks": 401,
"conversions": 109, "conversions": 23,
"revenue": 467.73 "revenue": 6015.59,
"bounceRate": 0.63
}, },
{ {
"date": "2025-01-02", "date": "2025-01-02",
"views": 3842, "views": 6940,
"clicks": 100, "clicks": 323,
"conversions": 15, "conversions": 37,
"revenue": 569.44 "revenue": 9086.44,
"bounceRate": 0.36
}, },
{ {
"date": "2025-01-03", "date": "2025-01-03",
"views": 4083, "views": 4390,
"clicks": 161, "clicks": 346,
"conversions": 73, "conversions": 26,
"revenue": 444.75 "revenue": 6360.75,
"bounceRate": 0.48
}, },
{ {
"date": "2025-01-04", "date": "2025-01-04",
"views": 5382, "views": 3429,
"clicks": 257, "clicks": 231,
"conversions": 63, "conversions": 13,
"revenue": 457.28 "revenue": 2360.96,
"bounceRate": 0.65
},
{
"date": "2025-01-05",
"views": 5804,
"clicks": 186,
"conversions": 22,
"revenue": 2535.96,
"bounceRate": 0.37
} }
] ]
} }
``` ```
**TOON** (3,630 tokens): **TOON** (4,507 tokens):
``` ```
metrics[5]{date,views,clicks,conversions,revenue}: metrics[5]{date,views,clicks,conversions,revenue,bounceRate}:
2024-12-31,1953,224,60,409.79 2025-01-01,6890,401,23,6015.59,0.63
2025-01-01,2981,242,109,467.73 2025-01-02,6940,323,37,9086.44,0.36
2025-01-02,3842,100,15,569.44 2025-01-03,4390,346,26,6360.75,0.48
2025-01-03,4083,161,73,444.75 2025-01-04,3429,231,13,2360.96,0.65
2025-01-04,5382,257,63,457.28 2025-01-05,5804,186,22,2535.96,0.37
``` ```
</details> </details>
@@ -203,25 +218,32 @@ metrics[5]{date,views,clicks,conversions,revenue}:
### Retrieval Accuracy ### Retrieval Accuracy
Tested across **2 LLMs** with data retrieval tasks: Tested across **3 LLMs** with data retrieval tasks:
``` ```
gpt-5-nano gpt-5-nano
toon ███████████████████░ 97.5% (155/159) toon ████████████████████ 99.4% (158/159)
markdown-kv ███████████████████░ 95.6% (152/159) yaml ███████████████████░ 95.0% (151/159)
yaml ███████████████████░ 94.3% (150/159) csv ██████████████████░░ 92.5% (147/159)
json ███████████████████░ 93.7% (149/159) json ██████████████████░░ 92.5% (147/159)
csv ███████████████████░ 93.7% (149/159) xml ██████████████████░░ 91.2% (145/159)
claude-haiku-4-5 claude-haiku-4-5
markdown-kv ███████████████░░░░░ 76.7% (122/159)
toon ███████████████░░░░░ 75.5% (120/159) toon ███████████████░░░░░ 75.5% (120/159)
json ███████████████░░░░░ 75.5% (120/159) xml ███████████████░░░░░ 75.5% (120/159)
csv ███████████████░░░░░ 75.5% (120/159) csv ███████████████░░░░░ 75.5% (120/159)
yaml ███████████████░░░░░ 74.8% (119/159) json ███████████████░░░░░ 75.5% (120/159)
yaml ███████████████░░░░░ 74.2% (118/159)
gemini-2.5-flash
xml ██████████████████░░ 91.8% (146/159)
csv █████████████████░░░ 86.2% (137/159)
toon █████████████████░░░ 84.9% (135/159)
json ████████████████░░░░ 81.8% (130/159)
yaml ████████████████░░░░ 78.6% (125/159)
``` ```
**Advantage:** TOON achieves **86.5% accuracy** (vs JSON's 84.6%) while using **46.3% fewer tokens**. **Advantage:** TOON achieves **86.6% accuracy** (vs JSON's 83.2%) while using **46.3% fewer tokens**.
<details> <details>
<summary><strong>View detailed breakdown by dataset and model</strong></summary> <summary><strong>View detailed breakdown by dataset and model</strong></summary>
@@ -232,41 +254,41 @@ claude-haiku-4-5
| Format | Accuracy | Tokens | Correct/Total | | Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- | | ------ | -------- | ------ | ------------- |
| `toon` | 86.2% | 2.483 | 100/116 | | `toon` | 87.4% | 2.483 | 152/174 |
| `csv` | 80.2% | 2.337 | 93/116 | | `csv` | 82.8% | 2.337 | 144/174 |
| `yaml` | 82.8% | 4.969 | 96/116 | | `yaml` | 83.9% | 4.969 | 146/174 |
| `markdown-kv` | 84.5% | 6.270 | 98/116 | | `json` | 83.9% | 6.347 | 146/174 |
| `json` | 84.5% | 6.347 | 98/116 | | `xml` | 88.5% | 7.314 | 154/174 |
##### E-commerce orders with nested structures ##### E-commerce orders with nested structures
| Format | Accuracy | Tokens | Correct/Total | | Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- | | ------ | -------- | ------ | ------------- |
| `toon` | 90.9% | 5.967 | 80/88 | | `toon` | 90.9% | 5.967 | 120/132 |
| `csv` | 90.9% | 6.735 | 80/88 | | `csv` | 93.9% | 6.735 | 124/132 |
| `yaml` | 89.8% | 7.328 | 79/88 | | `yaml` | 87.1% | 7.328 | 115/132 |
| `markdown-kv` | 90.9% | 9.110 | 80/88 | | `json` | 87.9% | 9.694 | 116/132 |
| `json` | 89.8% | 9.694 | 79/88 | | `xml` | 93.2% | 10.992 | 123/132 |
##### Time-series analytics data ##### Time-series analytics data
| Format | Accuracy | Tokens | Correct/Total | | Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- | | ------ | -------- | ------ | ------------- |
| `csv` | 87.9% | 1.393 | 51/58 | | `csv` | 89.7% | 1.393 | 78/87 |
| `toon` | 86.2% | 1.515 | 50/58 | | `toon` | 88.5% | 1.515 | 77/87 |
| `yaml` | 86.2% | 2.938 | 50/58 | | `yaml` | 83.9% | 2.938 | 73/87 |
| `json` | 87.9% | 3.665 | 51/58 | | `json` | 88.5% | 3.665 | 77/87 |
| `markdown-kv` | 86.2% | 3.779 | 50/58 | | `xml` | 85.1% | 4.376 | 74/87 |
##### Top 100 GitHub repositories ##### Top 100 GitHub repositories
| Format | Accuracy | Tokens | Correct/Total | | Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- | | ------ | -------- | ------ | ------------- |
| `csv` | 80.4% | 8.513 | 45/56 | | `toon` | 76.2% | 8.745 | 64/84 |
| `toon` | 80.4% | 8.745 | 45/56 | | `csv` | 69.0% | 8.513 | 58/84 |
| `yaml` | 78.6% | 13.129 | 44/56 | | `yaml` | 71.4% | 13.129 | 60/84 |
| `markdown-kv` | 82.1% | 15.436 | 46/56 | | `json` | 69.0% | 15.145 | 58/84 |
| `json` | 73.2% | 15.145 | 41/56 | | `xml` | 71.4% | 17.095 | 60/84 |
#### Performance by Model #### Performance by Model
@@ -274,28 +296,38 @@ claude-haiku-4-5
| Format | Accuracy | Correct/Total | | Format | Accuracy | Correct/Total |
| ------ | -------- | ------------- | | ------ | -------- | ------------- |
| `toon` | 97.5% | 155/159 | | `toon` | 99.4% | 158/159 |
| `markdown-kv` | 95.6% | 152/159 | | `yaml` | 95.0% | 151/159 |
| `yaml` | 94.3% | 150/159 | | `csv` | 92.5% | 147/159 |
| `json` | 93.7% | 149/159 | | `json` | 92.5% | 147/159 |
| `csv` | 93.7% | 149/159 | | `xml` | 91.2% | 145/159 |
##### claude-haiku-4-5 ##### claude-haiku-4-5
| Format | Accuracy | Correct/Total | | Format | Accuracy | Correct/Total |
| ------ | -------- | ------------- | | ------ | -------- | ------------- |
| `markdown-kv` | 76.7% | 122/159 |
| `toon` | 75.5% | 120/159 | | `toon` | 75.5% | 120/159 |
| `json` | 75.5% | 120/159 | | `xml` | 75.5% | 120/159 |
| `csv` | 75.5% | 120/159 | | `csv` | 75.5% | 120/159 |
| `yaml` | 74.8% | 119/159 | | `json` | 75.5% | 120/159 |
| `yaml` | 74.2% | 118/159 |
##### gemini-2.5-flash
| Format | Accuracy | Correct/Total |
| ------ | -------- | ------------- |
| `xml` | 91.8% | 146/159 |
| `csv` | 86.2% | 137/159 |
| `toon` | 84.9% | 135/159 |
| `json` | 81.8% | 130/159 |
| `yaml` | 78.6% | 125/159 |
#### Methodology #### Methodology
- **Semantic validation**: LLM-as-judge validates responses semantically (not exact string matching). - **Semantic validation**: LLM-as-judge validates responses semantically (not exact string matching).
- **Token counting**: Using `gpt-tokenizer` with `o200k_base` encoding. - **Token counting**: Using `gpt-tokenizer` with `o200k_base` encoding.
- **Question types**: Field retrieval, aggregation, and filtering tasks. - **Question types**: ~160 questions across field retrieval, aggregation, and filtering tasks.
- **Real data**: Faker.js-generated datasets + GitHub repositories. - **Datasets**: Faker.js-generated datasets (seeded) + GitHub repositories.
</details> </details>

22271
benchmarks/results/accuracy/raw-results.json
View File

File diff suppressed because it is too large Load Diff

97
benchmarks/results/accuracy/report.md
View File

@@ -1,24 +1,31 @@
### Retrieval Accuracy ### Retrieval Accuracy
Tested across **2 LLMs** with data retrieval tasks: Tested across **3 LLMs** with data retrieval tasks:
``` ```
gpt-5-nano gpt-5-nano
toon ███████████████████░ 97.5% (155/159) toon ████████████████████ 99.4% (158/159)
markdown-kv ███████████████████░ 95.6% (152/159) yaml ███████████████████░ 95.0% (151/159)
yaml ███████████████████░ 94.3% (150/159) csv ██████████████████░░ 92.5% (147/159)
json ███████████████████░ 93.7% (149/159) json ██████████████████░░ 92.5% (147/159)
csv ███████████████████░ 93.7% (149/159) xml ██████████████████░░ 91.2% (145/159)
claude-haiku-4-5 claude-haiku-4-5
markdown-kv ███████████████░░░░░ 76.7% (122/159)
toon ███████████████░░░░░ 75.5% (120/159) toon ███████████████░░░░░ 75.5% (120/159)
json ███████████████░░░░░ 75.5% (120/159) xml ███████████████░░░░░ 75.5% (120/159)
csv ███████████████░░░░░ 75.5% (120/159) csv ███████████████░░░░░ 75.5% (120/159)
yaml ███████████████░░░░░ 74.8% (119/159) json ███████████████░░░░░ 75.5% (120/159)
yaml ███████████████░░░░░ 74.2% (118/159)
gemini-2.5-flash
xml ██████████████████░░ 91.8% (146/159)
csv █████████████████░░░ 86.2% (137/159)
toon █████████████████░░░ 84.9% (135/159)
json ████████████████░░░░ 81.8% (130/159)
yaml ████████████████░░░░ 78.6% (125/159)
``` ```
**Advantage:** TOON achieves **86.5% accuracy** (vs JSON's 84.6%) while using **46.3% fewer tokens**. **Advantage:** TOON achieves **86.6% accuracy** (vs JSON's 83.2%) while using **46.3% fewer tokens**.
<details> <details>
<summary><strong>View detailed breakdown by dataset and model</strong></summary> <summary><strong>View detailed breakdown by dataset and model</strong></summary>
@@ -29,41 +36,41 @@ claude-haiku-4-5
| Format | Accuracy | Tokens | Correct/Total | | Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- | | ------ | -------- | ------ | ------------- |
| `toon` | 86.2% | 2.483 | 100/116 | | `toon` | 87.4% | 2.483 | 152/174 |
| `csv` | 80.2% | 2.337 | 93/116 | | `csv` | 82.8% | 2.337 | 144/174 |
| `yaml` | 82.8% | 4.969 | 96/116 | | `yaml` | 83.9% | 4.969 | 146/174 |
| `markdown-kv` | 84.5% | 6.270 | 98/116 | | `json` | 83.9% | 6.347 | 146/174 |
| `json` | 84.5% | 6.347 | 98/116 | | `xml` | 88.5% | 7.314 | 154/174 |
##### E-commerce orders with nested structures ##### E-commerce orders with nested structures
| Format | Accuracy | Tokens | Correct/Total | | Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- | | ------ | -------- | ------ | ------------- |
| `toon` | 90.9% | 5.967 | 80/88 | | `toon` | 90.9% | 5.967 | 120/132 |
| `csv` | 90.9% | 6.735 | 80/88 | | `csv` | 93.9% | 6.735 | 124/132 |
| `yaml` | 89.8% | 7.328 | 79/88 | | `yaml` | 87.1% | 7.328 | 115/132 |
| `markdown-kv` | 90.9% | 9.110 | 80/88 | | `json` | 87.9% | 9.694 | 116/132 |
| `json` | 89.8% | 9.694 | 79/88 | | `xml` | 93.2% | 10.992 | 123/132 |
##### Time-series analytics data ##### Time-series analytics data
| Format | Accuracy | Tokens | Correct/Total | | Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- | | ------ | -------- | ------ | ------------- |
| `csv` | 87.9% | 1.393 | 51/58 | | `csv` | 89.7% | 1.393 | 78/87 |
| `toon` | 86.2% | 1.515 | 50/58 | | `toon` | 88.5% | 1.515 | 77/87 |
| `yaml` | 86.2% | 2.938 | 50/58 | | `yaml` | 83.9% | 2.938 | 73/87 |
| `json` | 87.9% | 3.665 | 51/58 | | `json` | 88.5% | 3.665 | 77/87 |
| `markdown-kv` | 86.2% | 3.779 | 50/58 | | `xml` | 85.1% | 4.376 | 74/87 |
##### Top 100 GitHub repositories ##### Top 100 GitHub repositories
| Format | Accuracy | Tokens | Correct/Total | | Format | Accuracy | Tokens | Correct/Total |
| ------ | -------- | ------ | ------------- | | ------ | -------- | ------ | ------------- |
| `csv` | 80.4% | 8.513 | 45/56 | | `toon` | 76.2% | 8.745 | 64/84 |
| `toon` | 80.4% | 8.745 | 45/56 | | `csv` | 69.0% | 8.513 | 58/84 |
| `yaml` | 78.6% | 13.129 | 44/56 | | `yaml` | 71.4% | 13.129 | 60/84 |
| `markdown-kv` | 82.1% | 15.436 | 46/56 | | `json` | 69.0% | 15.145 | 58/84 |
| `json` | 73.2% | 15.145 | 41/56 | | `xml` | 71.4% | 17.095 | 60/84 |
#### Performance by Model #### Performance by Model
@@ -71,27 +78,37 @@ claude-haiku-4-5
| Format | Accuracy | Correct/Total | | Format | Accuracy | Correct/Total |
| ------ | -------- | ------------- | | ------ | -------- | ------------- |
| `toon` | 97.5% | 155/159 | | `toon` | 99.4% | 158/159 |
| `markdown-kv` | 95.6% | 152/159 | | `yaml` | 95.0% | 151/159 |
| `yaml` | 94.3% | 150/159 | | `csv` | 92.5% | 147/159 |
| `json` | 93.7% | 149/159 | | `json` | 92.5% | 147/159 |
| `csv` | 93.7% | 149/159 | | `xml` | 91.2% | 145/159 |
##### claude-haiku-4-5 ##### claude-haiku-4-5
| Format | Accuracy | Correct/Total | | Format | Accuracy | Correct/Total |
| ------ | -------- | ------------- | | ------ | -------- | ------------- |
| `markdown-kv` | 76.7% | 122/159 |
| `toon` | 75.5% | 120/159 | | `toon` | 75.5% | 120/159 |
| `json` | 75.5% | 120/159 | | `xml` | 75.5% | 120/159 |
| `csv` | 75.5% | 120/159 | | `csv` | 75.5% | 120/159 |
| `yaml` | 74.8% | 119/159 | | `json` | 75.5% | 120/159 |
| `yaml` | 74.2% | 118/159 |
##### gemini-2.5-flash
| Format | Accuracy | Correct/Total |
| ------ | -------- | ------------- |
| `xml` | 91.8% | 146/159 |
| `csv` | 86.2% | 137/159 |
| `toon` | 84.9% | 135/159 |
| `json` | 81.8% | 130/159 |
| `yaml` | 78.6% | 125/159 |
#### Methodology #### Methodology
- **Semantic validation**: LLM-as-judge validates responses semantically (not exact string matching). - **Semantic validation**: LLM-as-judge validates responses semantically (not exact string matching).
- **Token counting**: Using `gpt-tokenizer` with `o200k_base` encoding. - **Token counting**: Using `gpt-tokenizer` with `o200k_base` encoding.
- **Question types**: Field retrieval, aggregation, and filtering tasks. - **Question types**: ~160 questions across field retrieval, aggregation, and filtering tasks.
- **Real data**: Faker.js-generated datasets + GitHub repositories. - **Datasets**: Faker.js-generated datasets (seeded) + GitHub repositories.
</details> </details>

63
benchmarks/results/accuracy/summary.json
View File

@@ -2,49 +2,50 @@
"formatResults": [ "formatResults": [
{ {
"format": "toon", "format": "toon",
"accuracy": 0.8647798742138365, "accuracy": 0.8658280922431866,
"totalTokens": 4678, "totalTokens": 4678,
"averageLatency": 5016, "averageLatency": 5321,
"correctCount": 275, "correctCount": 413,
"totalCount": 318 "totalCount": 477
}, },
{ {
"format": "markdown-kv", "format": "xml",
"accuracy": 0.8616352201257862, "accuracy": 0.8616352201257862,
"totalTokens": 8649, "totalTokens": 9944,
"averageLatency": 4628, "averageLatency": 6035,
"correctCount": 274, "correctCount": 411,
"totalCount": 318 "totalCount": 477
},
{
"format": "json",
"accuracy": 0.8459119496855346,
"totalTokens": 8713,
"averageLatency": 5369,
"correctCount": 269,
"totalCount": 318
}, },
{ {
"format": "csv", "format": "csv",
"accuracy": 0.8459119496855346, "accuracy": 0.8469601677148847,
"totalTokens": 4745, "totalTokens": 4745,
"averageLatency": 5168, "averageLatency": 6551,
"correctCount": 269, "correctCount": 404,
"totalCount": 318 "totalCount": 477
},
{
"format": "json",
"accuracy": 0.8322851153039832,
"totalTokens": 8713,
"averageLatency": 7981,
"correctCount": 397,
"totalCount": 477
}, },
{ {
"format": "yaml", "format": "yaml",
"accuracy": 0.8459119496855346, "accuracy": 0.8259958071278826,
"totalTokens": 7091, "totalTokens": 7091,
"averageLatency": 4299, "averageLatency": 5561,
"correctCount": 269, "correctCount": 394,
"totalCount": 318 "totalCount": 477
} }
], ],
"questions": 159, "questions": 159,
"models": [ "models": [
"gpt-5-nano", "gpt-5-nano",
"claude-haiku-4-5" "claude-haiku-4-5",
"gemini-2.5-flash"
], ],
"datasets": [ "datasets": [
{ {
@@ -77,14 +78,14 @@
"csv-nested": 6735, "csv-nested": 6735,
"csv-analytics": 1393, "csv-analytics": 1393,
"csv-github": 8513, "csv-github": 8513,
"markdown-kv-tabular": 6270, "xml-tabular": 7314,
"markdown-kv-nested": 9110, "xml-nested": 10992,
"markdown-kv-analytics": 3779, "xml-analytics": 4376,
"markdown-kv-github": 15436, "xml-github": 17095,
"yaml-tabular": 4969, "yaml-tabular": 4969,
"yaml-nested": 7328, "yaml-nested": 7328,
"yaml-analytics": 2938, "yaml-analytics": 2938,
"yaml-github": 13129 "yaml-github": 13129
}, },
"timestamp": "2025-10-27T13:17:28.071Z" "timestamp": "2025-10-27T15:01:57.523Z"
} }

97
benchmarks/results/token-efficiency.md
View File

@@ -1,13 +1,23 @@
### Token Efficiency ### Token Efficiency
``` ```
⭐ GitHub Repositories ██████████████░░░░░░░░░░░ 8,745 tokens (JSON: 15,145) 💰 42.3% saved ⭐ GitHub Repositories ██████████████░░░░░░░░░░░ 8,745 tokens
📈 Daily Analytics ██████████░░░░░░░░░░░░░░░ 3,630 tokens (JSON: 9,023) 💰 59.8% saved vs JSON: 15,145 💰 42.3% saved
👥 API Response ██████████████░░░░░░░░░░░ 2,597 tokens (JSON: 4,589) 💰 43.4% saved vs XML: 17,095 💰 48.8% saved
🛒 E-Commerce Order ████████████████░░░░░░░░░ 164 tokens (JSON: 256) 💰 35.9% saved
```
**Total:** 15,136 tokens (TOON) vs 29,013 tokens (JSON) → 47.8% savings 📈 Daily Analytics ██████████░░░░░░░░░░░░░░░ 4,507 tokens
vs JSON: 10,977 💰 58.9% saved
vs XML: 13,128 💰 65.7% saved
🛒 E-Commerce Order ████████████████░░░░░░░░░ 166 tokens
vs JSON: 257 💰 35.4% saved
vs XML: 271 💰 38.7% saved
─────────────────────────────────────────────────────────────────────
Total ████████████░░░░░░░░░░░░░ 13,418 tokens
vs JSON: 26,379 💰 49.1% saved
vs XML: 30,494 💰 56.0% saved
```
<details> <details>
<summary><strong>View detailed examples</strong></summary> <summary><strong>View detailed examples</strong></summary>
@@ -16,7 +26,7 @@
**Configuration:** Top 100 GitHub repositories with stars, forks, and metadata **Configuration:** Top 100 GitHub repositories with stars, forks, and metadata
**Savings:** 6,400 tokens (42.3% reduction) **Savings:** 6,400 tokens (42.3% reduction vs JSON)
**JSON** (15,145 tokens): **JSON** (15,145 tokens):
@@ -27,7 +37,7 @@
"id": 28457823, "id": 28457823,
"name": "freeCodeCamp", "name": "freeCodeCamp",
"repo": "freeCodeCamp/freeCodeCamp", "repo": "freeCodeCamp/freeCodeCamp",
"description": "freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,...", "description": "freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…",
"createdAt": "2014-12-24T17:49:19Z", "createdAt": "2014-12-24T17:49:19Z",
"updatedAt": "2025-10-27T07:40:58Z", "updatedAt": "2025-10-27T07:40:58Z",
"pushedAt": "2025-10-26T11:31:08Z", "pushedAt": "2025-10-26T11:31:08Z",
@@ -70,7 +80,7 @@
``` ```
repositories[3]{id,name,repo,description,createdAt,updatedAt,pushedAt,stars,watchers,forks,defaultBranch}: repositories[3]{id,name,repo,description,createdAt,updatedAt,pushedAt,stars,watchers,forks,defaultBranch}:
28457823,freeCodeCamp,freeCodeCamp/freeCodeCamp,"freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,...","2014-12-24T17:49:19Z","2025-10-27T07:40:58Z","2025-10-26T11:31:08Z",430828,8582,42136,main 28457823,freeCodeCamp,freeCodeCamp/freeCodeCamp,"freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…","2014-12-24T17:49:19Z","2025-10-27T07:40:58Z","2025-10-26T11:31:08Z",430828,8582,42136,main
132750724,build-your-own-x,codecrafters-io/build-your-own-x,Master programming by recreating your favorite technologies from scratch.,"2018-05-09T12:03:18Z","2025-10-27T07:43:25Z","2025-10-10T18:45:01Z",430102,6322,40388,master 132750724,build-your-own-x,codecrafters-io/build-your-own-x,Master programming by recreating your favorite technologies from scratch.,"2018-05-09T12:03:18Z","2025-10-27T07:43:25Z","2025-10-10T18:45:01Z",430102,6322,40388,master
21737465,awesome,sindresorhus/awesome,😎 Awesome lists about all kinds of interesting topics,"2014-07-11T13:42:37Z","2025-10-27T07:44:27Z","2025-10-23T17:26:53Z",409760,8016,32015,main 21737465,awesome,sindresorhus/awesome,😎 Awesome lists about all kinds of interesting topics,"2014-07-11T13:42:37Z","2025-10-27T07:44:27Z","2025-10-23T17:26:53Z",409760,8016,32015,main
``` ```
@@ -81,61 +91,66 @@ repositories[3]{id,name,repo,description,createdAt,updatedAt,pushedAt,stars,watc
**Configuration:** 180 days of web metrics (views, clicks, conversions, revenue) **Configuration:** 180 days of web metrics (views, clicks, conversions, revenue)
**Savings:** 5,393 tokens (59.8% reduction) **Savings:** 6,470 tokens (58.9% reduction vs JSON)
**JSON** (9,023 tokens): **JSON** (10,977 tokens):
```json ```json
{ {
"metrics": [ "metrics": [
{
"date": "2024-12-31",
"views": 1953,
"clicks": 224,
"conversions": 60,
"revenue": 409.79
},
{ {
"date": "2025-01-01", "date": "2025-01-01",
"views": 2981, "views": 6890,
"clicks": 242, "clicks": 401,
"conversions": 109, "conversions": 23,
"revenue": 467.73 "revenue": 6015.59,
"bounceRate": 0.63
}, },
{ {
"date": "2025-01-02", "date": "2025-01-02",
"views": 3842, "views": 6940,
"clicks": 100, "clicks": 323,
"conversions": 15, "conversions": 37,
"revenue": 569.44 "revenue": 9086.44,
"bounceRate": 0.36
}, },
{ {
"date": "2025-01-03", "date": "2025-01-03",
"views": 4083, "views": 4390,
"clicks": 161, "clicks": 346,
"conversions": 73, "conversions": 26,
"revenue": 444.75 "revenue": 6360.75,
"bounceRate": 0.48
}, },
{ {
"date": "2025-01-04", "date": "2025-01-04",
"views": 5382, "views": 3429,
"clicks": 257, "clicks": 231,
"conversions": 63, "conversions": 13,
"revenue": 457.28 "revenue": 2360.96,
"bounceRate": 0.65
},
{
"date": "2025-01-05",
"views": 5804,
"clicks": 186,
"conversions": 22,
"revenue": 2535.96,
"bounceRate": 0.37
} }
] ]
} }
``` ```
**TOON** (3,630 tokens): **TOON** (4,507 tokens):
``` ```
metrics[5]{date,views,clicks,conversions,revenue}: metrics[5]{date,views,clicks,conversions,revenue,bounceRate}:
2024-12-31,1953,224,60,409.79 2025-01-01,6890,401,23,6015.59,0.63
2025-01-01,2981,242,109,467.73 2025-01-02,6940,323,37,9086.44,0.36
2025-01-02,3842,100,15,569.44 2025-01-03,4390,346,26,6360.75,0.48
2025-01-03,4083,161,73,444.75 2025-01-04,3429,231,13,2360.96,0.65
2025-01-04,5382,257,63,457.28 2025-01-05,5804,186,22,2535.96,0.37
``` ```
</details> </details>

1
benchmarks/scripts/accuracy-benchmark.ts
View File

@@ -116,6 +116,7 @@ else {
formatName: task.formatName, formatName: task.formatName,
formattedData, formattedData,
model, model,
modelName: task.modelName,
}) })
// Progress update after task completes // Progress update after task completes

100
benchmarks/scripts/token-efficiency-benchmark.ts
View File

@@ -7,16 +7,20 @@ import { encode } from '../../src/index'
import githubRepos from '../data/github-repos.json' with { type: 'json' } import githubRepos from '../data/github-repos.json' with { type: 'json' }
import { BENCHMARKS_DIR, ROOT_DIR } from '../src/constants' import { BENCHMARKS_DIR, ROOT_DIR } from '../src/constants'
import { generateAnalyticsData } from '../src/datasets' import { generateAnalyticsData } from '../src/datasets'
import { formatters } from '../src/formatters'
interface BenchmarkResult { interface BenchmarkResult {
name: string name: string
emoji: string emoji: string
description: string description: string
data: any data: Record<string, any>
jsonTokens: number jsonTokens: number
toonTokens: number toonTokens: number
savings: number xmlTokens: number
savingsPercent: string jsonSavings: number
jsonSavingsPercent: string
xmlSavings: number
xmlSavingsPercent: string
showDetailed: boolean showDetailed: boolean
} }
@@ -37,13 +41,6 @@ const BENCHMARK_EXAMPLES = [
getData: () => generateAnalyticsData(180), getData: () => generateAnalyticsData(180),
showDetailed: true, showDetailed: true,
}, },
{
name: 'API Response',
emoji: '👥',
description: '50 user records with metadata and timestamps',
getData: () => generateUsers(50),
showDetailed: false,
},
{ {
name: 'E-Commerce Order', name: 'E-Commerce Order',
emoji: '🛒', emoji: '🛒',
@@ -56,6 +53,7 @@ const BENCHMARK_EXAMPLES = [
// Calculate total savings // Calculate total savings
let totalJsonTokens = 0 let totalJsonTokens = 0
let totalToonTokens = 0 let totalToonTokens = 0
let totalXmlTokens = 0
const results: BenchmarkResult[] = [] const results: BenchmarkResult[] = []
@@ -64,14 +62,21 @@ for (const example of BENCHMARK_EXAMPLES) {
const jsonString = JSON.stringify(data, undefined, 2) const jsonString = JSON.stringify(data, undefined, 2)
const toonString = encode(data) const toonString = encode(data)
const xmlString = formatters.xml(data)
const jsonTokens = encodeTokens(jsonString).length const jsonTokens = encodeTokens(jsonString).length
const toonTokens = encodeTokens(toonString).length const toonTokens = encodeTokens(toonString).length
const savings = jsonTokens - toonTokens const xmlTokens = encodeTokens(xmlString).length
const savingsPercent = ((savings / jsonTokens) * 100).toFixed(1)
const jsonSavings = jsonTokens - toonTokens
const jsonSavingsPercent = ((jsonSavings / jsonTokens) * 100).toFixed(1)
const xmlSavings = xmlTokens - toonTokens
const xmlSavingsPercent = ((xmlSavings / xmlTokens) * 100).toFixed(1)
totalJsonTokens += jsonTokens totalJsonTokens += jsonTokens
totalToonTokens += toonTokens totalToonTokens += toonTokens
totalXmlTokens += xmlTokens
results.push({ results.push({
name: example.name, name: example.name,
@@ -80,25 +85,51 @@ for (const example of BENCHMARK_EXAMPLES) {
data, data,
jsonTokens, jsonTokens,
toonTokens, toonTokens,
savings, xmlTokens,
savingsPercent, jsonSavings,
jsonSavingsPercent,
xmlSavings,
xmlSavingsPercent,
showDetailed: example.showDetailed, showDetailed: example.showDetailed,
}) })
} }
const totalSavings = totalJsonTokens - totalToonTokens const totalJsonSavings = totalJsonTokens - totalToonTokens
const totalSavingsPercent = ((totalSavings / totalJsonTokens) * 100).toFixed(1) const totalJsonSavingsPercent = ((totalJsonSavings / totalJsonTokens) * 100).toFixed(1)
// Generate ASCII bar chart visualization const totalXmlSavings = totalXmlTokens - totalToonTokens
const barChartSection = results const totalXmlSavingsPercent = ((totalXmlSavings / totalXmlTokens) * 100).toFixed(1)
// Generate ASCII bar chart visualization (stacked compact format)
const datasetRows = results
.map((result) => { .map((result) => {
const percentage = Number.parseFloat(result.savingsPercent) const percentage = Number.parseFloat(result.jsonSavingsPercent)
const bar = generateBarChart(100 - percentage) // Invert to show TOON tokens const bar = generateBarChart(100 - percentage) // Invert to show TOON tokens
const jsonStr = result.jsonTokens.toLocaleString('en-US')
const toonStr = result.toonTokens.toLocaleString('en-US') const toonStr = result.toonTokens.toLocaleString('en-US')
return `${result.emoji} ${result.name.padEnd(25)} ${bar} ${toonStr.padStart(6)} tokens (JSON: ${jsonStr.padStart(6)}) 💰 ${result.savingsPercent}% saved` const jsonStr = result.jsonTokens.toLocaleString('en-US')
const xmlStr = result.xmlTokens.toLocaleString('en-US')
const line1 = `${result.emoji} ${result.name.padEnd(25)} ${bar} ${toonStr.padStart(6)} tokens`
const line2 = ` vs JSON: ${jsonStr.padStart(6)} 💰 ${result.jsonSavingsPercent}% saved`
const line3 = ` vs XML: ${xmlStr.padStart(6)} 💰 ${result.xmlSavingsPercent}% saved`
return `${line1}\n${line2}\n${line3}`
}) })
.join('\n') .join('\n\n')
// Add separator and totals row
const separator = '─────────────────────────────────────────────────────────────────────'
// Calculate bar for totals (TOON vs average of JSON+XML)
const averageComparisonTokens = (totalJsonTokens + totalXmlTokens) / 2
const totalPercentage = (totalToonTokens / averageComparisonTokens) * 100
const totalBar = generateBarChart(totalPercentage)
const totalLine1 = `Total ${totalBar} ${totalToonTokens.toLocaleString('en-US').padStart(6)} tokens`
const totalLine2 = ` vs JSON: ${totalJsonTokens.toLocaleString('en-US').padStart(6)} 💰 ${totalJsonSavingsPercent}% saved`
const totalLine3 = ` vs XML: ${totalXmlTokens.toLocaleString('en-US').padStart(6)} 💰 ${totalXmlSavingsPercent}% saved`
const barChartSection = `${datasetRows}\n\n${separator}\n${totalLine1}\n${totalLine2}\n${totalLine3}`
// Generate detailed examples (only for selected examples) // Generate detailed examples (only for selected examples)
const detailedExamples = results const detailedExamples = results
@@ -108,9 +139,9 @@ const detailedExamples = results
let displayData = result.data let displayData = result.data
if (result.name === 'GitHub Repositories') { if (result.name === 'GitHub Repositories') {
displayData = { displayData = {
repositories: result.data.repositories.slice(0, 3).map((repo: any) => ({ repositories: result.data.repositories.slice(0, 3).map((repo: Record<string, any>) => ({
...repo, ...repo,
description: repo.description?.slice(0, 80) + (repo.description?.length > 80 ? '...' : ''), description: repo.description?.slice(0, 80) + (repo.description?.length > 80 ? '…' : ''),
})), })),
} }
} }
@@ -124,7 +155,7 @@ const detailedExamples = results
**Configuration:** ${result.description} **Configuration:** ${result.description}
**Savings:** ${result.savings.toLocaleString('en-US')} tokens (${result.savingsPercent}% reduction) **Savings:** ${result.jsonSavings.toLocaleString('en-US')} tokens (${result.jsonSavingsPercent}% reduction vs JSON)
**JSON** (${result.jsonTokens.toLocaleString('en-US')} tokens): **JSON** (${result.jsonTokens.toLocaleString('en-US')} tokens):
@@ -146,8 +177,6 @@ const markdown = `### Token Efficiency
${barChartSection} ${barChartSection}
\`\`\` \`\`\`
**Total:** ${totalToonTokens.toLocaleString('en-US')} tokens (TOON) vs ${totalJsonTokens.toLocaleString('en-US')} tokens (JSON) → ${totalSavingsPercent}% savings
<details> <details>
<summary><strong>View detailed examples</strong></summary> <summary><strong>View detailed examples</strong></summary>
@@ -170,23 +199,6 @@ function generateBarChart(percentage: number, maxWidth: number = 25): string {
return '█'.repeat(filled) + '░'.repeat(empty) return '█'.repeat(filled) + '░'.repeat(empty)
} }
// Generate user API response
function generateUsers(count: number) {
return {
users: Array.from({ length: count }, (_, i) => ({
id: i + 1,
name: faker.person.fullName(),
email: faker.internet.email(),
role: faker.helpers.arrayElement(['admin', 'user', 'moderator']),
active: faker.datatype.boolean(),
createdAt: faker.date.past({ years: 2 }).toISOString(),
lastLogin: faker.date.recent({ days: 30 }).toISOString(),
})),
total: count,
page: 1,
}
}
// Generate nested e-commerce order // Generate nested e-commerce order
function generateOrder() { function generateOrder() {
return { return {

19
benchmarks/src/evaluate.ts
View File

@@ -10,6 +10,7 @@
import type { LanguageModelV2 } from '@ai-sdk/provider' import type { LanguageModelV2 } from '@ai-sdk/provider'
import type { EvaluationResult, Question } from './types' import type { EvaluationResult, Question } from './types'
import { anthropic } from '@ai-sdk/anthropic' import { anthropic } from '@ai-sdk/anthropic'
import { google } from '@ai-sdk/google'
import { openai } from '@ai-sdk/openai' import { openai } from '@ai-sdk/openai'
import { generateText } from 'ai' import { generateText } from 'ai'
import { consola } from 'consola' import { consola } from 'consola'
@@ -20,16 +21,18 @@ import { consola } from 'consola'
export const models: Record<string, LanguageModelV2> = { export const models: Record<string, LanguageModelV2> = {
'gpt-5-nano': openai('gpt-5-nano'), 'gpt-5-nano': openai('gpt-5-nano'),
'claude-haiku-4-5': anthropic('claude-haiku-4-5-20251001'), 'claude-haiku-4-5': anthropic('claude-haiku-4-5-20251001'),
'gemini-2.5-flash': google('gemini-2.5-flash'),
} }
/** /**
* Evaluate a single question with a specific format and model * Evaluate a single question with a specific format and model
*/ */
export async function evaluateQuestion( export async function evaluateQuestion(
{ question, formatName, formattedData, model}: { question, formatName, formattedData, model, modelName}:
{ question: Question, formatName: string, formattedData: string, model: LanguageModelV2 }, { question: Question, formatName: string, formattedData: string, model: LanguageModelV2, modelName: string },
): Promise<EvaluationResult> { ): Promise<EvaluationResult> {
const prompt = `Given the following data in ${formatName} format: const prompt = `
Given the following data in ${formatName} format:
\`\`\` \`\`\`
${formattedData} ${formattedData}
@@ -37,13 +40,14 @@ ${formattedData}
Question: ${question.prompt} Question: ${question.prompt}
Provide only the direct answer, without any additional explanation or formatting.` Provide only the direct answer, without any additional explanation or formatting.
`.trim()
const startTime = performance.now() const startTime = performance.now()
const { text, usage } = await generateText({ const { text, usage } = await generateText({
model, model,
prompt, prompt,
temperature: model.modelId.startsWith('gpt-') ? undefined : 0, temperature: !model.modelId.startsWith('gpt-') ? 0 : undefined,
}) })
const latencyMs = performance.now() - startTime const latencyMs = performance.now() - startTime
@@ -56,7 +60,7 @@ Provide only the direct answer, without any additional explanation or formatting
return { return {
questionId: question.id, questionId: question.id,
format: formatName, format: formatName,
model: model.modelId, model: modelName,
expected: question.groundTruth, expected: question.groundTruth,
actual: text.trim(), actual: text.trim(),
isCorrect, isCorrect,
@@ -93,9 +97,8 @@ Respond with only "YES" or "NO".`
try { try {
const { text } = await generateText({ const { text } = await generateText({
model: models['claude-haiku-4-5']!, model: models['gpt-5-nano']!,
prompt, prompt,
temperature: 0,
}) })
return text.trim().toUpperCase() === 'YES' return text.trim().toUpperCase() === 'YES'

4
benchmarks/src/report.ts
View File

@@ -201,8 +201,8 @@ ${modelPerformance}
- **Semantic validation**: LLM-as-judge validates responses semantically (not exact string matching). - **Semantic validation**: LLM-as-judge validates responses semantically (not exact string matching).
- **Token counting**: Using \`gpt-tokenizer\` with \`o200k_base\` encoding. - **Token counting**: Using \`gpt-tokenizer\` with \`o200k_base\` encoding.
- **Question types**: Field retrieval, aggregation, and filtering tasks. - **Question types**: ~160 questions across field retrieval, aggregation, and filtering tasks.
- **Real data**: Faker.js-generated datasets + GitHub repositories. - **Datasets**: Faker.js-generated datasets (seeded) + GitHub repositories.
</details> </details>
`.trimStart() `.trimStart()

2
benchmarks/src/types.ts
View File

@@ -1,7 +1,7 @@
export interface Dataset { export interface Dataset {
name: string name: string
description: string description: string
data: any data: Record<string, any>
} }
export interface Question { export interface Question {