mirror of
https://github.com/voson-wang/toon.git
synced 2026-01-29 23:34:10 +08:00
test: add LLM retrieval accuracy tests
This commit is contained in:
Johann Schopplich
96
benchmarks/results/accuracy/accuracy.md
Normal file
96
benchmarks/results/accuracy/accuracy.md
Normal file
@@ -0,0 +1,96 @@
|
||||
### Retrieval Accuracy
|
||||
|
||||
Tested across **2 LLMs** with data retrieval tasks:
|
||||
|
||||
```
|
||||
gpt-4o-mini ██████████████░░░░░░ 72.3% accuracy
|
||||
claude-haiku-4-5 ███████████████░░░░░ 76.7% accuracy
|
||||
```
|
||||
|
||||
**TOON achieves 73.9% accuracy (vs JSON's 73.6%) while using 46.3% fewer tokens.**
|
||||
|
||||
| Format | Accuracy | Average Tokens |
|
||||
| ------ | -------- | -------------- |
|
||||
| `toon` | 73.9% | 4.678 |
|
||||
| `json` | 73.6% | 8.713 |
|
||||
| `markdown-kv` | 73.6% | 8.649 |
|
||||
| `csv` | 72.3% | 4.745 |
|
||||
| `yaml` | 71.7% | 7.091 |
|
||||
|
||||
<details>
|
||||
<summary><strong>View detailed breakdown by dataset and model</strong></summary>
|
||||
|
||||
#### Performance by Dataset
|
||||
|
||||
##### Uniform employee records (TOON optimal format)
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
|--------|----------|--------|---------------|
|
||||
| `toon` | 72.4% | 2.483 | 84/116 |
|
||||
| `csv` | 69.0% | 2.337 | 80/116 |
|
||||
| `yaml` | 68.1% | 4.969 | 79/116 |
|
||||
| `markdown-kv` | 68.1% | 6.270 | 79/116 |
|
||||
| `json` | 68.1% | 6.347 | 79/116 |
|
||||
|
||||
##### E-commerce orders with nested structures
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
|--------|----------|--------|---------------|
|
||||
| `toon` | 84.1% | 5.967 | 74/88 |
|
||||
| `csv` | 83.0% | 6.735 | 73/88 |
|
||||
| `yaml` | 81.8% | 7.328 | 72/88 |
|
||||
| `markdown-kv` | 86.4% | 9.110 | 76/88 |
|
||||
| `json` | 84.1% | 9.694 | 74/88 |
|
||||
|
||||
##### Time-series analytics data
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
|--------|----------|--------|---------------|
|
||||
| `csv` | 72.4% | 1.393 | 42/58 |
|
||||
| `toon` | 70.7% | 1.515 | 41/58 |
|
||||
| `yaml` | 72.4% | 2.938 | 42/58 |
|
||||
| `json` | 74.1% | 3.665 | 43/58 |
|
||||
| `markdown-kv` | 70.7% | 3.779 | 41/58 |
|
||||
|
||||
##### Popular GitHub repositories
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
|--------|----------|--------|---------------|
|
||||
| `toon` | 64.3% | 8.745 | 36/56 |
|
||||
| `csv` | 62.5% | 8.513 | 35/56 |
|
||||
| `json` | 67.9% | 15.145 | 38/56 |
|
||||
| `markdown-kv` | 67.9% | 15.436 | 38/56 |
|
||||
| `yaml` | 62.5% | 13.129 | 35/56 |
|
||||
|
||||
|
||||
#### Performance by Model
|
||||
|
||||
##### gpt-4o-mini
|
||||
|
||||
| Format | Accuracy | Correct/Total |
|
||||
|--------|----------|---------------|
|
||||
| `toon` | 72.3% | 115/159 |
|
||||
| `json` | 71.7% | 114/159 |
|
||||
| `markdown-kv` | 70.4% | 112/159 |
|
||||
| `csv` | 69.2% | 110/159 |
|
||||
| `yaml` | 68.6% | 109/159 |
|
||||
|
||||
##### claude-haiku-4-5
|
||||
|
||||
| Format | Accuracy | Correct/Total |
|
||||
|--------|----------|---------------|
|
||||
| `markdown-kv` | 76.7% | 122/159 |
|
||||
| `toon` | 75.5% | 120/159 |
|
||||
| `json` | 75.5% | 120/159 |
|
||||
| `csv` | 75.5% | 120/159 |
|
||||
| `yaml` | 74.8% | 119/159 |
|
||||
|
||||
|
||||
#### Methodology
|
||||
|
||||
- **Semantic validation**: LLM-as-judge validates responses semantically (not exact string matching)
|
||||
- **Token counting**: Using `gpt-tokenizer` with `o200k_base` encoding
|
||||
- **Question types**: Field retrieval, aggregation, and filtering tasks
|
||||
- **Real data**: faker.js-generated datasets + real GitHub repository data
|
||||
|
||||
</details>
|
||||
17492
benchmarks/results/accuracy/raw-results.json
Normal file
17492
benchmarks/results/accuracy/raw-results.json
Normal file
File diff suppressed because it is too large
Load Diff
96
benchmarks/results/accuracy/report.md
Normal file
96
benchmarks/results/accuracy/report.md
Normal file
@@ -0,0 +1,96 @@
|
||||
### Retrieval Accuracy
|
||||
|
||||
Tested across **2 LLMs** with data retrieval tasks:
|
||||
|
||||
```
|
||||
gpt-4o-mini ██████████████░░░░░░ 72.3% accuracy
|
||||
claude-haiku-4-5 ███████████████░░░░░ 76.7% accuracy
|
||||
```
|
||||
|
||||
**TOON achieves 73.9% accuracy (vs JSON's 73.6%) while using 46.3% fewer tokens.**
|
||||
|
||||
| Format | Accuracy | Average Tokens |
|
||||
| ------ | -------- | -------------- |
|
||||
| `toon` | 73.9% | 4.678 |
|
||||
| `json` | 73.6% | 8.713 |
|
||||
| `markdown-kv` | 73.6% | 8.649 |
|
||||
| `csv` | 72.3% | 4.745 |
|
||||
| `yaml` | 71.7% | 7.091 |
|
||||
|
||||
<details>
|
||||
<summary><strong>View detailed breakdown by dataset and model</strong></summary>
|
||||
|
||||
#### Performance by Dataset
|
||||
|
||||
##### Uniform employee records (TOON optimal format)
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
|--------|----------|--------|---------------|
|
||||
| `toon` | 72.4% | 2.483 | 84/116 |
|
||||
| `csv` | 69.0% | 2.337 | 80/116 |
|
||||
| `yaml` | 68.1% | 4.969 | 79/116 |
|
||||
| `markdown-kv` | 68.1% | 6.270 | 79/116 |
|
||||
| `json` | 68.1% | 6.347 | 79/116 |
|
||||
|
||||
##### E-commerce orders with nested structures
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
|--------|----------|--------|---------------|
|
||||
| `toon` | 84.1% | 5.967 | 74/88 |
|
||||
| `csv` | 83.0% | 6.735 | 73/88 |
|
||||
| `yaml` | 81.8% | 7.328 | 72/88 |
|
||||
| `markdown-kv` | 86.4% | 9.110 | 76/88 |
|
||||
| `json` | 84.1% | 9.694 | 74/88 |
|
||||
|
||||
##### Time-series analytics data
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
|--------|----------|--------|---------------|
|
||||
| `csv` | 72.4% | 1.393 | 42/58 |
|
||||
| `toon` | 70.7% | 1.515 | 41/58 |
|
||||
| `yaml` | 72.4% | 2.938 | 42/58 |
|
||||
| `json` | 74.1% | 3.665 | 43/58 |
|
||||
| `markdown-kv` | 70.7% | 3.779 | 41/58 |
|
||||
|
||||
##### Popular GitHub repositories
|
||||
|
||||
| Format | Accuracy | Tokens | Correct/Total |
|
||||
|--------|----------|--------|---------------|
|
||||
| `toon` | 64.3% | 8.745 | 36/56 |
|
||||
| `csv` | 62.5% | 8.513 | 35/56 |
|
||||
| `json` | 67.9% | 15.145 | 38/56 |
|
||||
| `markdown-kv` | 67.9% | 15.436 | 38/56 |
|
||||
| `yaml` | 62.5% | 13.129 | 35/56 |
|
||||
|
||||
|
||||
#### Performance by Model
|
||||
|
||||
##### gpt-4o-mini
|
||||
|
||||
| Format | Accuracy | Correct/Total |
|
||||
|--------|----------|---------------|
|
||||
| `toon` | 72.3% | 115/159 |
|
||||
| `json` | 71.7% | 114/159 |
|
||||
| `markdown-kv` | 70.4% | 112/159 |
|
||||
| `csv` | 69.2% | 110/159 |
|
||||
| `yaml` | 68.6% | 109/159 |
|
||||
|
||||
##### claude-haiku-4-5
|
||||
|
||||
| Format | Accuracy | Correct/Total |
|
||||
|--------|----------|---------------|
|
||||
| `markdown-kv` | 76.7% | 122/159 |
|
||||
| `toon` | 75.5% | 120/159 |
|
||||
| `json` | 75.5% | 120/159 |
|
||||
| `csv` | 75.5% | 120/159 |
|
||||
| `yaml` | 74.8% | 119/159 |
|
||||
|
||||
|
||||
#### Methodology
|
||||
|
||||
- **Semantic validation**: LLM-as-judge validates responses semantically (not exact string matching).
|
||||
- **Token counting**: Using `gpt-tokenizer` with `o200k_base` encoding.
|
||||
- **Question types**: Field retrieval, aggregation, and filtering tasks.
|
||||
- **Real data**: Faker.js-generated datasets + GitHub repositories.
|
||||
|
||||
</details>
|
||||
95
benchmarks/results/accuracy/summary.json
Normal file
95
benchmarks/results/accuracy/summary.json
Normal file
@@ -0,0 +1,95 @@
|
||||
{
|
||||
"formatResults": [
|
||||
{
|
||||
"format": "toon",
|
||||
"accuracy": 0.7389937106918238,
|
||||
"totalTokens": 4678,
|
||||
"avgInputTokens": 4675,
|
||||
"avgLatency": 1424,
|
||||
"correctCount": 235,
|
||||
"totalCount": 318
|
||||
},
|
||||
{
|
||||
"format": "json",
|
||||
"accuracy": 0.7358490566037735,
|
||||
"totalTokens": 8713,
|
||||
"avgInputTokens": 9177,
|
||||
"avgLatency": 1678,
|
||||
"correctCount": 234,
|
||||
"totalCount": 318
|
||||
},
|
||||
{
|
||||
"format": "markdown-kv",
|
||||
"accuracy": 0.7358490566037735,
|
||||
"totalTokens": 8649,
|
||||
"avgInputTokens": 8242,
|
||||
"avgLatency": 1724,
|
||||
"correctCount": 234,
|
||||
"totalCount": 318
|
||||
},
|
||||
{
|
||||
"format": "csv",
|
||||
"accuracy": 0.7232704402515723,
|
||||
"totalTokens": 4745,
|
||||
"avgInputTokens": 4878,
|
||||
"avgLatency": 1573,
|
||||
"correctCount": 230,
|
||||
"totalCount": 318
|
||||
},
|
||||
{
|
||||
"format": "yaml",
|
||||
"accuracy": 0.7169811320754716,
|
||||
"totalTokens": 7091,
|
||||
"avgInputTokens": 7136,
|
||||
"avgLatency": 1602,
|
||||
"correctCount": 228,
|
||||
"totalCount": 318
|
||||
}
|
||||
],
|
||||
"questions": 159,
|
||||
"models": [
|
||||
"gpt-4o-mini",
|
||||
"claude-haiku-4-5"
|
||||
],
|
||||
"datasets": [
|
||||
{
|
||||
"name": "tabular",
|
||||
"description": "Uniform employee records (TOON optimal format)"
|
||||
},
|
||||
{
|
||||
"name": "nested",
|
||||
"description": "E-commerce orders with nested structures"
|
||||
},
|
||||
{
|
||||
"name": "analytics",
|
||||
"description": "Time-series analytics data"
|
||||
},
|
||||
{
|
||||
"name": "github",
|
||||
"description": "Popular GitHub repositories"
|
||||
}
|
||||
],
|
||||
"tokenCounts": {
|
||||
"json-tabular": 6347,
|
||||
"json-nested": 9694,
|
||||
"json-analytics": 3665,
|
||||
"json-github": 15145,
|
||||
"toon-tabular": 2483,
|
||||
"toon-nested": 5967,
|
||||
"toon-analytics": 1515,
|
||||
"toon-github": 8745,
|
||||
"csv-tabular": 2337,
|
||||
"csv-nested": 6735,
|
||||
"csv-analytics": 1393,
|
||||
"csv-github": 8513,
|
||||
"markdown-kv-tabular": 6270,
|
||||
"markdown-kv-nested": 9110,
|
||||
"markdown-kv-analytics": 3779,
|
||||
"markdown-kv-github": 15436,
|
||||
"yaml-tabular": 4969,
|
||||
"yaml-nested": 7328,
|
||||
"yaml-analytics": 2938,
|
||||
"yaml-github": 13129
|
||||
},
|
||||
"timestamp": "2025-10-27T10:46:35.127Z"
|
||||
}
|
||||
Reference in New Issue
Block a user