github/toon
1
0
Fork 0
mirror of https://github.com/voson-wang/toon.git synced 2026-01-29 23:34:10 +08:00
Code Issues Packages Projects Releases Wiki Activity

test: add LLM retrieval accuracy tests

Browse Source
This commit is contained in:
Johann Schopplich
2025-10-27 11:48:33 +01:00
parent eb8f7e28e1
commit 3c840259fe
25 changed files with 21404 additions and 723 deletions
Download Patch File Download Diff File Expand all files Collapse all files

96
benchmarks/results/accuracy/accuracy.md Normal file
View File

@@ -0,0 +1,96 @@
### Retrieval Accuracy
Tested across **2 LLMs** with data retrieval tasks:
```
gpt-4o-mini ██████████████░░░░░░ 72.3% accuracy
claude-haiku-4-5 ███████████████░░░░░ 76.7% accuracy
```
**TOON achieves 73.9% accuracy (vs JSON's 73.6%) while using 46.3% fewer tokens.**
| Format | Accuracy | Average Tokens |
| ------ | -------- | -------------- |
| `toon` | 73.9% | 4.678 |
| `json` | 73.6% | 8.713 |
| `markdown-kv` | 73.6% | 8.649 |
| `csv` | 72.3% | 4.745 |
| `yaml` | 71.7% | 7.091 |
<details>
<summary><strong>View detailed breakdown by dataset and model</strong></summary>
#### Performance by Dataset
##### Uniform employee records (TOON optimal format)
| Format | Accuracy | Tokens | Correct/Total |
|--------|----------|--------|---------------|
| `toon` | 72.4% | 2.483 | 84/116 |
| `csv` | 69.0% | 2.337 | 80/116 |
| `yaml` | 68.1% | 4.969 | 79/116 |
| `markdown-kv` | 68.1% | 6.270 | 79/116 |
| `json` | 68.1% | 6.347 | 79/116 |
##### E-commerce orders with nested structures
| Format | Accuracy | Tokens | Correct/Total |
|--------|----------|--------|---------------|
| `toon` | 84.1% | 5.967 | 74/88 |
| `csv` | 83.0% | 6.735 | 73/88 |
| `yaml` | 81.8% | 7.328 | 72/88 |
| `markdown-kv` | 86.4% | 9.110 | 76/88 |
| `json` | 84.1% | 9.694 | 74/88 |
##### Time-series analytics data
| Format | Accuracy | Tokens | Correct/Total |
|--------|----------|--------|---------------|
| `csv` | 72.4% | 1.393 | 42/58 |
| `toon` | 70.7% | 1.515 | 41/58 |
| `yaml` | 72.4% | 2.938 | 42/58 |
| `json` | 74.1% | 3.665 | 43/58 |
| `markdown-kv` | 70.7% | 3.779 | 41/58 |
##### Popular GitHub repositories
| Format | Accuracy | Tokens | Correct/Total |
|--------|----------|--------|---------------|
| `toon` | 64.3% | 8.745 | 36/56 |
| `csv` | 62.5% | 8.513 | 35/56 |
| `json` | 67.9% | 15.145 | 38/56 |
| `markdown-kv` | 67.9% | 15.436 | 38/56 |
| `yaml` | 62.5% | 13.129 | 35/56 |
#### Performance by Model
##### gpt-4o-mini
| Format | Accuracy | Correct/Total |
|--------|----------|---------------|
| `toon` | 72.3% | 115/159 |
| `json` | 71.7% | 114/159 |
| `markdown-kv` | 70.4% | 112/159 |
| `csv` | 69.2% | 110/159 |
| `yaml` | 68.6% | 109/159 |
##### claude-haiku-4-5
| Format | Accuracy | Correct/Total |
|--------|----------|---------------|
| `markdown-kv` | 76.7% | 122/159 |
| `toon` | 75.5% | 120/159 |
| `json` | 75.5% | 120/159 |
| `csv` | 75.5% | 120/159 |
| `yaml` | 74.8% | 119/159 |
#### Methodology
- **Semantic validation**: LLM-as-judge validates responses semantically (not exact string matching)
- **Token counting**: Using `gpt-tokenizer` with `o200k_base` encoding
- **Question types**: Field retrieval, aggregation, and filtering tasks
- **Real data**: faker.js-generated datasets + real GitHub repository data
</details>

17492
benchmarks/results/accuracy/raw-results.json Normal file
View File

File diff suppressed because it is too large Load Diff

96
benchmarks/results/accuracy/report.md Normal file
View File

@@ -0,0 +1,96 @@
### Retrieval Accuracy
Tested across **2 LLMs** with data retrieval tasks:
```
gpt-4o-mini ██████████████░░░░░░ 72.3% accuracy
claude-haiku-4-5 ███████████████░░░░░ 76.7% accuracy
```
**TOON achieves 73.9% accuracy (vs JSON's 73.6%) while using 46.3% fewer tokens.**
| Format | Accuracy | Average Tokens |
| ------ | -------- | -------------- |
| `toon` | 73.9% | 4.678 |
| `json` | 73.6% | 8.713 |
| `markdown-kv` | 73.6% | 8.649 |
| `csv` | 72.3% | 4.745 |
| `yaml` | 71.7% | 7.091 |
<details>
<summary><strong>View detailed breakdown by dataset and model</strong></summary>
#### Performance by Dataset
##### Uniform employee records (TOON optimal format)
| Format | Accuracy | Tokens | Correct/Total |
|--------|----------|--------|---------------|
| `toon` | 72.4% | 2.483 | 84/116 |
| `csv` | 69.0% | 2.337 | 80/116 |
| `yaml` | 68.1% | 4.969 | 79/116 |
| `markdown-kv` | 68.1% | 6.270 | 79/116 |
| `json` | 68.1% | 6.347 | 79/116 |
##### E-commerce orders with nested structures
| Format | Accuracy | Tokens | Correct/Total |
|--------|----------|--------|---------------|
| `toon` | 84.1% | 5.967 | 74/88 |
| `csv` | 83.0% | 6.735 | 73/88 |
| `yaml` | 81.8% | 7.328 | 72/88 |
| `markdown-kv` | 86.4% | 9.110 | 76/88 |
| `json` | 84.1% | 9.694 | 74/88 |
##### Time-series analytics data
| Format | Accuracy | Tokens | Correct/Total |
|--------|----------|--------|---------------|
| `csv` | 72.4% | 1.393 | 42/58 |
| `toon` | 70.7% | 1.515 | 41/58 |
| `yaml` | 72.4% | 2.938 | 42/58 |
| `json` | 74.1% | 3.665 | 43/58 |
| `markdown-kv` | 70.7% | 3.779 | 41/58 |
##### Popular GitHub repositories
| Format | Accuracy | Tokens | Correct/Total |
|--------|----------|--------|---------------|
| `toon` | 64.3% | 8.745 | 36/56 |
| `csv` | 62.5% | 8.513 | 35/56 |
| `json` | 67.9% | 15.145 | 38/56 |
| `markdown-kv` | 67.9% | 15.436 | 38/56 |
| `yaml` | 62.5% | 13.129 | 35/56 |
#### Performance by Model
##### gpt-4o-mini
| Format | Accuracy | Correct/Total |
|--------|----------|---------------|
| `toon` | 72.3% | 115/159 |
| `json` | 71.7% | 114/159 |
| `markdown-kv` | 70.4% | 112/159 |
| `csv` | 69.2% | 110/159 |
| `yaml` | 68.6% | 109/159 |
##### claude-haiku-4-5
| Format | Accuracy | Correct/Total |
|--------|----------|---------------|
| `markdown-kv` | 76.7% | 122/159 |
| `toon` | 75.5% | 120/159 |
| `json` | 75.5% | 120/159 |
| `csv` | 75.5% | 120/159 |
| `yaml` | 74.8% | 119/159 |
#### Methodology
- **Semantic validation**: LLM-as-judge validates responses semantically (not exact string matching).
- **Token counting**: Using `gpt-tokenizer` with `o200k_base` encoding.
- **Question types**: Field retrieval, aggregation, and filtering tasks.
- **Real data**: Faker.js-generated datasets + GitHub repositories.
</details>

95
benchmarks/results/accuracy/summary.json Normal file
View File

@@ -0,0 +1,95 @@
{
"formatResults": [
{
"format": "toon",
"accuracy": 0.7389937106918238,
"totalTokens": 4678,
"avgInputTokens": 4675,
"avgLatency": 1424,
"correctCount": 235,
"totalCount": 318
},
{
"format": "json",
"accuracy": 0.7358490566037735,
"totalTokens": 8713,
"avgInputTokens": 9177,
"avgLatency": 1678,
"correctCount": 234,
"totalCount": 318
},
{
"format": "markdown-kv",
"accuracy": 0.7358490566037735,
"totalTokens": 8649,
"avgInputTokens": 8242,
"avgLatency": 1724,
"correctCount": 234,
"totalCount": 318
},
{
"format": "csv",
"accuracy": 0.7232704402515723,
"totalTokens": 4745,
"avgInputTokens": 4878,
"avgLatency": 1573,
"correctCount": 230,
"totalCount": 318
},
{
"format": "yaml",
"accuracy": 0.7169811320754716,
"totalTokens": 7091,
"avgInputTokens": 7136,
"avgLatency": 1602,
"correctCount": 228,
"totalCount": 318
}
],
"questions": 159,
"models": [
"gpt-4o-mini",
"claude-haiku-4-5"
],
"datasets": [
{
"name": "tabular",
"description": "Uniform employee records (TOON optimal format)"
},
{
"name": "nested",
"description": "E-commerce orders with nested structures"
},
{
"name": "analytics",
"description": "Time-series analytics data"
},
{
"name": "github",
"description": "Popular GitHub repositories"
}
],
"tokenCounts": {
"json-tabular": 6347,
"json-nested": 9694,
"json-analytics": 3665,
"json-github": 15145,
"toon-tabular": 2483,
"toon-nested": 5967,
"toon-analytics": 1515,
"toon-github": 8745,
"csv-tabular": 2337,
"csv-nested": 6735,
"csv-analytics": 1393,
"csv-github": 8513,
"markdown-kv-tabular": 6270,
"markdown-kv-nested": 9110,
"markdown-kv-analytics": 3779,
"markdown-kv-github": 15436,
"yaml-tabular": 4969,
"yaml-nested": 7328,
"yaml-analytics": 2938,
"yaml-github": 13129
},
"timestamp": "2025-10-27T10:46:35.127Z"
}

141
benchmarks/results/token-efficiency.md Normal file
View File

@@ -0,0 +1,141 @@
### Token Efficiency
```
⭐ GitHub Repositories ██████████████░░░░░░░░░░░ 8,745 tokens (JSON: 15,145) 💰 42.3% saved
📈 Analytics Time Series ██████████░░░░░░░░░░░░░░░ 3,631 tokens (JSON: 9,024) 💰 59.8% saved
👥 API Response ██████████████░░░░░░░░░░░ 2,593 tokens (JSON: 4,589) 💰 43.5% saved
🛒 E-commerce Order ███████████████░░░░░░░░░░ 203 tokens (JSON: 338) 💰 39.9% saved
```
**Total:** 15,172 tokens (TOON) vs 29,096 tokens (JSON) → 47.9% savings
<details>
<summary><strong>View detailed examples</strong></summary>
#### ⭐ GitHub Repositories
**Configuration:** Top 100 GitHub repositories with stars, forks, and metadata
**Savings:** 6,400 tokens (42.3% reduction)
**JSON** (15,145 tokens):
```json
{
"repositories": [
{
"id": 28457823,
"name": "freeCodeCamp",
"repo": "freeCodeCamp/freeCodeCamp",
"description": "freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,...",
"createdAt": "2014-12-24T17:49:19Z",
"updatedAt": "2025-10-27T07:40:58Z",
"pushedAt": "2025-10-26T11:31:08Z",
"stars": 430828,
"watchers": 8582,
"forks": 42136,
"defaultBranch": "main"
},
{
"id": 132750724,
"name": "build-your-own-x",
"repo": "codecrafters-io/build-your-own-x",
"description": "Master programming by recreating your favorite technologies from scratch.",
"createdAt": "2018-05-09T12:03:18Z",
"updatedAt": "2025-10-27T07:43:25Z",
"pushedAt": "2025-10-10T18:45:01Z",
"stars": 430102,
"watchers": 6322,
"forks": 40388,
"defaultBranch": "master"
},
{
"id": 21737465,
"name": "awesome",
"repo": "sindresorhus/awesome",
"description": "😎 Awesome lists about all kinds of interesting topics",
"createdAt": "2014-07-11T13:42:37Z",
"updatedAt": "2025-10-27T07:44:27Z",
"pushedAt": "2025-10-23T17:26:53Z",
"stars": 409760,
"watchers": 8016,
"forks": 32015,
"defaultBranch": "main"
}
]
}
```
**TOON** (8,745 tokens):
```
repositories[3]{id,name,repo,description,createdAt,updatedAt,pushedAt,stars,watchers,forks,defaultBranch}:
28457823,freeCodeCamp,freeCodeCamp/freeCodeCamp,"freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,...","2014-12-24T17:49:19Z","2025-10-27T07:40:58Z","2025-10-26T11:31:08Z",430828,8582,42136,main
132750724,build-your-own-x,codecrafters-io/build-your-own-x,Master programming by recreating your favorite technologies from scratch.,"2018-05-09T12:03:18Z","2025-10-27T07:43:25Z","2025-10-10T18:45:01Z",430102,6322,40388,master
21737465,awesome,sindresorhus/awesome,😎 Awesome lists about all kinds of interesting topics,"2014-07-11T13:42:37Z","2025-10-27T07:44:27Z","2025-10-23T17:26:53Z",409760,8016,32015,main
```
---
#### 📈 Analytics Time Series
**Configuration:** 180 days of web metrics (views, clicks, conversions, revenue)
**Savings:** 5,393 tokens (59.8% reduction)
**JSON** (9,024 tokens):
```json
{
"metrics": [
{
"date": "2024-12-31",
"views": 3769,
"clicks": 400,
"conversions": 59,
"revenue": 198.98
},
{
"date": "2025-01-01",
"views": 5742,
"clicks": 463,
"conversions": 28,
"revenue": 295.77
},
{
"date": "2025-01-02",
"views": 3669,
"clicks": 336,
"conversions": 102,
"revenue": 624.23
},
{
"date": "2025-01-03",
"views": 1332,
"clicks": 304,
"conversions": 99,
"revenue": 113.06
},
{
"date": "2025-01-04",
"views": 1444,
"clicks": 222,
"conversions": 88,
"revenue": 986.69
}
]
}
```
**TOON** (3,631 tokens):
```
metrics[5]{date,views,clicks,conversions,revenue}:
2024-12-31,3769,400,59,198.98
2025-01-01,5742,463,28,295.77
2025-01-02,3669,336,102,624.23
2025-01-03,1332,304,99,113.06
2025-01-04,1444,222,88,986.69
```
</details>