docs: add benchmarks for gemini-2.5-flash

2026-01-29 15:24:10 +08:00 · 2025-10-27 16:02:51 +01:00
parent 77696ce932
commit 7b76acde31
10 changed files with 15837 additions and 7011 deletions
--- a/benchmarks/results/accuracy/raw-results.json
+++ b/benchmarks/results/accuracy/raw-results.json
--- a/benchmarks/results/accuracy/report.md
+++ b/benchmarks/results/accuracy/report.md
@@ -1,24 +1,31 @@
 ### Retrieval Accuracy

-Tested across **2 LLMs** with data retrieval tasks:
+Tested across **3 LLMs** with data retrieval tasks:

 ```
 gpt-5-nano
-  toon         ███████████████████░  97.5% (155/159)
-  markdown-kv  ███████████████████░  95.6% (152/159)
-  yaml         ███████████████████░  94.3% (150/159)
-  json         ███████████████████░  93.7% (149/159)
-  csv          ███████████████████░  93.7% (149/159)
+  toon         ████████████████████  99.4% (158/159)
+  yaml         ███████████████████░  95.0% (151/159)
+  csv          ██████████████████░░  92.5% (147/159)
+  json         ██████████████████░░  92.5% (147/159)
+  xml          ██████████████████░░  91.2% (145/159)

 claude-haiku-4-5
-  markdown-kv  ███████████████░░░░░  76.7% (122/159)
  toon         ███████████████░░░░░  75.5% (120/159)
-  json         ███████████████░░░░░  75.5% (120/159)
+  xml          ███████████████░░░░░  75.5% (120/159)
  csv          ███████████████░░░░░  75.5% (120/159)
-  yaml         ███████████████░░░░░  74.8% (119/159)
+  json         ███████████████░░░░░  75.5% (120/159)
+  yaml         ███████████████░░░░░  74.2% (118/159)
+
+gemini-2.5-flash
+  xml          ██████████████████░░  91.8% (146/159)
+  csv          █████████████████░░░  86.2% (137/159)
+  toon         █████████████████░░░  84.9% (135/159)
+  json         ████████████████░░░░  81.8% (130/159)
+  yaml         ████████████████░░░░  78.6% (125/159)
 ```

-**Advantage:** TOON achieves **86.5% accuracy** (vs JSON's 84.6%) while using **46.3% fewer tokens**.
+**Advantage:** TOON achieves **86.6% accuracy** (vs JSON's 83.2%) while using **46.3% fewer tokens**.

 <details>
 <summary><strong>View detailed breakdown by dataset and model</strong></summary>
@@ -29,41 +36,41 @@ claude-haiku-4-5

 | Format | Accuracy | Tokens | Correct/Total |
 | ------ | -------- | ------ | ------------- |
-| `toon` | 86.2% | 2.483 | 100/116 |
-| `csv` | 80.2% | 2.337 | 93/116 |
-| `yaml` | 82.8% | 4.969 | 96/116 |
-| `markdown-kv` | 84.5% | 6.270 | 98/116 |
-| `json` | 84.5% | 6.347 | 98/116 |
+| `toon` | 87.4% | 2.483 | 152/174 |
+| `csv` | 82.8% | 2.337 | 144/174 |
+| `yaml` | 83.9% | 4.969 | 146/174 |
+| `json` | 83.9% | 6.347 | 146/174 |
+| `xml` | 88.5% | 7.314 | 154/174 |

 ##### E-commerce orders with nested structures

 | Format | Accuracy | Tokens | Correct/Total |
 | ------ | -------- | ------ | ------------- |
-| `toon` | 90.9% | 5.967 | 80/88 |
-| `csv` | 90.9% | 6.735 | 80/88 |
-| `yaml` | 89.8% | 7.328 | 79/88 |
-| `markdown-kv` | 90.9% | 9.110 | 80/88 |
-| `json` | 89.8% | 9.694 | 79/88 |
+| `toon` | 90.9% | 5.967 | 120/132 |
+| `csv` | 93.9% | 6.735 | 124/132 |
+| `yaml` | 87.1% | 7.328 | 115/132 |
+| `json` | 87.9% | 9.694 | 116/132 |
+| `xml` | 93.2% | 10.992 | 123/132 |

 ##### Time-series analytics data

 | Format | Accuracy | Tokens | Correct/Total |
 | ------ | -------- | ------ | ------------- |
-| `csv` | 87.9% | 1.393 | 51/58 |
-| `toon` | 86.2% | 1.515 | 50/58 |
-| `yaml` | 86.2% | 2.938 | 50/58 |
-| `json` | 87.9% | 3.665 | 51/58 |
-| `markdown-kv` | 86.2% | 3.779 | 50/58 |
+| `csv` | 89.7% | 1.393 | 78/87 |
+| `toon` | 88.5% | 1.515 | 77/87 |
+| `yaml` | 83.9% | 2.938 | 73/87 |
+| `json` | 88.5% | 3.665 | 77/87 |
+| `xml` | 85.1% | 4.376 | 74/87 |

 ##### Top 100 GitHub repositories

 | Format | Accuracy | Tokens | Correct/Total |
 | ------ | -------- | ------ | ------------- |
-| `csv` | 80.4% | 8.513 | 45/56 |
-| `toon` | 80.4% | 8.745 | 45/56 |
-| `yaml` | 78.6% | 13.129 | 44/56 |
-| `markdown-kv` | 82.1% | 15.436 | 46/56 |
-| `json` | 73.2% | 15.145 | 41/56 |
+| `toon` | 76.2% | 8.745 | 64/84 |
+| `csv` | 69.0% | 8.513 | 58/84 |
+| `yaml` | 71.4% | 13.129 | 60/84 |
+| `json` | 69.0% | 15.145 | 58/84 |
+| `xml` | 71.4% | 17.095 | 60/84 |

 #### Performance by Model

@@ -71,27 +78,37 @@ claude-haiku-4-5

 | Format | Accuracy | Correct/Total |
 | ------ | -------- | ------------- |
-| `toon` | 97.5% | 155/159 |
-| `markdown-kv` | 95.6% | 152/159 |
-| `yaml` | 94.3% | 150/159 |
-| `json` | 93.7% | 149/159 |
-| `csv` | 93.7% | 149/159 |
+| `toon` | 99.4% | 158/159 |
+| `yaml` | 95.0% | 151/159 |
+| `csv` | 92.5% | 147/159 |
+| `json` | 92.5% | 147/159 |
+| `xml` | 91.2% | 145/159 |

 ##### claude-haiku-4-5

 | Format | Accuracy | Correct/Total |
 | ------ | -------- | ------------- |
-| `markdown-kv` | 76.7% | 122/159 |
 | `toon` | 75.5% | 120/159 |
-| `json` | 75.5% | 120/159 |
+| `xml` | 75.5% | 120/159 |
 | `csv` | 75.5% | 120/159 |
-| `yaml` | 74.8% | 119/159 |
+| `json` | 75.5% | 120/159 |
+| `yaml` | 74.2% | 118/159 |
+
+##### gemini-2.5-flash
+
+| Format | Accuracy | Correct/Total |
+| ------ | -------- | ------------- |
+| `xml` | 91.8% | 146/159 |
+| `csv` | 86.2% | 137/159 |
+| `toon` | 84.9% | 135/159 |
+| `json` | 81.8% | 130/159 |
+| `yaml` | 78.6% | 125/159 |

 #### Methodology

 - **Semantic validation**: LLM-as-judge validates responses semantically (not exact string matching).
 - **Token counting**: Using `gpt-tokenizer` with `o200k_base` encoding.
- **Question types**: Field retrieval, aggregation, and filtering tasks.
- **Real data**: Faker.js-generated datasets + GitHub repositories.
+- **Question types**: ~160 questions across field retrieval, aggregation, and filtering tasks.
+- **Datasets**: Faker.js-generated datasets (seeded) + GitHub repositories.

 </details>
--- a/benchmarks/results/accuracy/summary.json
+++ b/benchmarks/results/accuracy/summary.json
@@ -2,49 +2,50 @@
  "formatResults": [
    {
      "format": "toon",
-      "accuracy": 0.8647798742138365,
+      "accuracy": 0.8658280922431866,
      "totalTokens": 4678,
-      "averageLatency": 5016,
-      "correctCount": 275,
-      "totalCount": 318
+      "averageLatency": 5321,
+      "correctCount": 413,
+      "totalCount": 477
    },
    {
-      "format": "markdown-kv",
+      "format": "xml",
      "accuracy": 0.8616352201257862,
-      "totalTokens": 8649,
-      "averageLatency": 4628,
-      "correctCount": 274,
-      "totalCount": 318
-    },
-    {
-      "format": "json",
-      "accuracy": 0.8459119496855346,
-      "totalTokens": 8713,
-      "averageLatency": 5369,
-      "correctCount": 269,
-      "totalCount": 318
+      "totalTokens": 9944,
+      "averageLatency": 6035,
+      "correctCount": 411,
+      "totalCount": 477
    },
    {
      "format": "csv",
-      "accuracy": 0.8459119496855346,
+      "accuracy": 0.8469601677148847,
      "totalTokens": 4745,
-      "averageLatency": 5168,
-      "correctCount": 269,
-      "totalCount": 318
+      "averageLatency": 6551,
+      "correctCount": 404,
+      "totalCount": 477
+    },
+    {
+      "format": "json",
+      "accuracy": 0.8322851153039832,
+      "totalTokens": 8713,
+      "averageLatency": 7981,
+      "correctCount": 397,
+      "totalCount": 477
    },
    {
      "format": "yaml",
-      "accuracy": 0.8459119496855346,
+      "accuracy": 0.8259958071278826,
      "totalTokens": 7091,
-      "averageLatency": 4299,
-      "correctCount": 269,
-      "totalCount": 318
+      "averageLatency": 5561,
+      "correctCount": 394,
+      "totalCount": 477
    }
  ],
  "questions": 159,
  "models": [
    "gpt-5-nano",
-    "claude-haiku-4-5"
+    "claude-haiku-4-5",
+    "gemini-2.5-flash"
  ],
  "datasets": [
    {
@@ -77,14 +78,14 @@
    "csv-nested": 6735,
    "csv-analytics": 1393,
    "csv-github": 8513,
-    "markdown-kv-tabular": 6270,
-    "markdown-kv-nested": 9110,
-    "markdown-kv-analytics": 3779,
-    "markdown-kv-github": 15436,
+    "xml-tabular": 7314,
+    "xml-nested": 10992,
+    "xml-analytics": 4376,
+    "xml-github": 17095,
    "yaml-tabular": 4969,
    "yaml-nested": 7328,
    "yaml-analytics": 2938,
    "yaml-github": 13129
  },
-  "timestamp": "2025-10-27T13:17:28.071Z"
+  "timestamp": "2025-10-27T15:01:57.523Z"
 }
--- a/benchmarks/results/token-efficiency.md
+++ b/benchmarks/results/token-efficiency.md
@@ -1,13 +1,23 @@
 ### Token Efficiency

 ```
-⭐ GitHub Repositories       ██████████████░░░░░░░░░░░   8,745 tokens  (JSON: 15,145)  💰 42.3% saved
-📈 Daily Analytics           ██████████░░░░░░░░░░░░░░░   3,630 tokens  (JSON:  9,023)  💰 59.8% saved
-👥 API Response              ██████████████░░░░░░░░░░░   2,597 tokens  (JSON:  4,589)  💰 43.4% saved
-🛒 E-Commerce Order          ████████████████░░░░░░░░░     164 tokens  (JSON:    256)  💰 35.9% saved
-```
+⭐ GitHub Repositories       ██████████████░░░░░░░░░░░   8,745 tokens
+                             vs JSON: 15,145  💰 42.3% saved
+                             vs XML:  17,095  💰 48.8% saved

-**Total:** 15,136 tokens (TOON) vs 29,013 tokens (JSON) → 47.8% savings
+📈 Daily Analytics           ██████████░░░░░░░░░░░░░░░   4,507 tokens
+                             vs JSON: 10,977  💰 58.9% saved
+                             vs XML:  13,128  💰 65.7% saved
+
+🛒 E-Commerce Order          ████████████████░░░░░░░░░     166 tokens
+                             vs JSON:    257  💰 35.4% saved
+                             vs XML:     271  💰 38.7% saved
+
+─────────────────────────────────────────────────────────────────────
+Total                        ████████████░░░░░░░░░░░░░  13,418 tokens
+                             vs JSON: 26,379  💰 49.1% saved
+                             vs XML:  30,494  💰 56.0% saved
+```

 <details>
 <summary><strong>View detailed examples</strong></summary>
@@ -16,7 +26,7 @@

 **Configuration:** Top 100 GitHub repositories with stars, forks, and metadata

-**Savings:** 6,400 tokens (42.3% reduction)
+**Savings:** 6,400 tokens (42.3% reduction vs JSON)

 **JSON** (15,145 tokens):

@@ -27,7 +37,7 @@
      "id": 28457823,
      "name": "freeCodeCamp",
      "repo": "freeCodeCamp/freeCodeCamp",
-      "description": "freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,...",
+      "description": "freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…",
      "createdAt": "2014-12-24T17:49:19Z",
      "updatedAt": "2025-10-27T07:40:58Z",
      "pushedAt": "2025-10-26T11:31:08Z",
@@ -70,7 +80,7 @@

 ```
 repositories[3]{id,name,repo,description,createdAt,updatedAt,pushedAt,stars,watchers,forks,defaultBranch}:
-  28457823,freeCodeCamp,freeCodeCamp/freeCodeCamp,"freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,...","2014-12-24T17:49:19Z","2025-10-27T07:40:58Z","2025-10-26T11:31:08Z",430828,8582,42136,main
+  28457823,freeCodeCamp,freeCodeCamp/freeCodeCamp,"freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,…","2014-12-24T17:49:19Z","2025-10-27T07:40:58Z","2025-10-26T11:31:08Z",430828,8582,42136,main
  132750724,build-your-own-x,codecrafters-io/build-your-own-x,Master programming by recreating your favorite technologies from scratch.,"2018-05-09T12:03:18Z","2025-10-27T07:43:25Z","2025-10-10T18:45:01Z",430102,6322,40388,master
  21737465,awesome,sindresorhus/awesome,😎 Awesome lists about all kinds of interesting topics,"2014-07-11T13:42:37Z","2025-10-27T07:44:27Z","2025-10-23T17:26:53Z",409760,8016,32015,main
 ```
@@ -81,61 +91,66 @@ repositories[3]{id,name,repo,description,createdAt,updatedAt,pushedAt,stars,watc

 **Configuration:** 180 days of web metrics (views, clicks, conversions, revenue)

-**Savings:** 5,393 tokens (59.8% reduction)
+**Savings:** 6,470 tokens (58.9% reduction vs JSON)

-**JSON** (9,023 tokens):
+**JSON** (10,977 tokens):

 ```json
 {
  "metrics": [
-    {
-      "date": "2024-12-31",
-      "views": 1953,
-      "clicks": 224,
-      "conversions": 60,
-      "revenue": 409.79
-    },
    {
      "date": "2025-01-01",
-      "views": 2981,
-      "clicks": 242,
-      "conversions": 109,
-      "revenue": 467.73
+      "views": 6890,
+      "clicks": 401,
+      "conversions": 23,
+      "revenue": 6015.59,
+      "bounceRate": 0.63
    },
    {
      "date": "2025-01-02",
-      "views": 3842,
-      "clicks": 100,
-      "conversions": 15,
-      "revenue": 569.44
+      "views": 6940,
+      "clicks": 323,
+      "conversions": 37,
+      "revenue": 9086.44,
+      "bounceRate": 0.36
    },
    {
      "date": "2025-01-03",
-      "views": 4083,
-      "clicks": 161,
-      "conversions": 73,
-      "revenue": 444.75
+      "views": 4390,
+      "clicks": 346,
+      "conversions": 26,
+      "revenue": 6360.75,
+      "bounceRate": 0.48
    },
    {
      "date": "2025-01-04",
-      "views": 5382,
-      "clicks": 257,
-      "conversions": 63,
-      "revenue": 457.28
+      "views": 3429,
+      "clicks": 231,
+      "conversions": 13,
+      "revenue": 2360.96,
+      "bounceRate": 0.65
+    },
+    {
+      "date": "2025-01-05",
+      "views": 5804,
+      "clicks": 186,
+      "conversions": 22,
+      "revenue": 2535.96,
+      "bounceRate": 0.37
    }
  ]
 }
 ```

-**TOON** (3,630 tokens):
+**TOON** (4,507 tokens):

 ```
-metrics[5]{date,views,clicks,conversions,revenue}:
-  2024-12-31,1953,224,60,409.79
-  2025-01-01,2981,242,109,467.73
-  2025-01-02,3842,100,15,569.44
-  2025-01-03,4083,161,73,444.75
-  2025-01-04,5382,257,63,457.28
+metrics[5]{date,views,clicks,conversions,revenue,bounceRate}:
+  2025-01-01,6890,401,23,6015.59,0.63
+  2025-01-02,6940,323,37,9086.44,0.36
+  2025-01-03,4390,346,26,6360.75,0.48
+  2025-01-04,3429,231,13,2360.96,0.65
+  2025-01-05,5804,186,22,2535.96,0.37
 ```

 </details>
--- a/benchmarks/scripts/accuracy-benchmark.ts
+++ b/benchmarks/scripts/accuracy-benchmark.ts
@@ -116,6 +116,7 @@ else {
        formatName: task.formatName,
        formattedData,
        model,
+        modelName: task.modelName,
      })

      // Progress update after task completes
--- a/benchmarks/scripts/token-efficiency-benchmark.ts
+++ b/benchmarks/scripts/token-efficiency-benchmark.ts
@@ -7,16 +7,20 @@ import { encode } from '../../src/index'
 import githubRepos from '../data/github-repos.json' with { type: 'json' }
 import { BENCHMARKS_DIR, ROOT_DIR } from '../src/constants'
 import { generateAnalyticsData } from '../src/datasets'
+import { formatters } from '../src/formatters'

 interface BenchmarkResult {
  name: string
  emoji: string
  description: string
-  data: any
+  data: Record<string, any>
  jsonTokens: number
  toonTokens: number
-  savings: number
-  savingsPercent: string
+  xmlTokens: number
+  jsonSavings: number
+  jsonSavingsPercent: string
+  xmlSavings: number
+  xmlSavingsPercent: string
  showDetailed: boolean
 }

@@ -37,13 +41,6 @@ const BENCHMARK_EXAMPLES = [
    getData: () => generateAnalyticsData(180),
    showDetailed: true,
  },
-  {
-    name: 'API Response',
-    emoji: '👥',
-    description: '50 user records with metadata and timestamps',
-    getData: () => generateUsers(50),
-    showDetailed: false,
-  },
  {
    name: 'E-Commerce Order',
    emoji: '🛒',
@@ -56,6 +53,7 @@ const BENCHMARK_EXAMPLES = [
 // Calculate total savings
 let totalJsonTokens = 0
 let totalToonTokens = 0
+let totalXmlTokens = 0

 const results: BenchmarkResult[] = []

@@ -64,14 +62,21 @@ for (const example of BENCHMARK_EXAMPLES) {

  const jsonString = JSON.stringify(data, undefined, 2)
  const toonString = encode(data)
+  const xmlString = formatters.xml(data)

  const jsonTokens = encodeTokens(jsonString).length
  const toonTokens = encodeTokens(toonString).length
-  const savings = jsonTokens - toonTokens
-  const savingsPercent = ((savings / jsonTokens) * 100).toFixed(1)
+  const xmlTokens = encodeTokens(xmlString).length
+
+  const jsonSavings = jsonTokens - toonTokens
+  const jsonSavingsPercent = ((jsonSavings / jsonTokens) * 100).toFixed(1)
+
+  const xmlSavings = xmlTokens - toonTokens
+  const xmlSavingsPercent = ((xmlSavings / xmlTokens) * 100).toFixed(1)

  totalJsonTokens += jsonTokens
  totalToonTokens += toonTokens
+  totalXmlTokens += xmlTokens

  results.push({
    name: example.name,
@@ -80,25 +85,51 @@ for (const example of BENCHMARK_EXAMPLES) {
    data,
    jsonTokens,
    toonTokens,
-    savings,
-    savingsPercent,
+    xmlTokens,
+    jsonSavings,
+    jsonSavingsPercent,
+    xmlSavings,
+    xmlSavingsPercent,
    showDetailed: example.showDetailed,
  })
 }

-const totalSavings = totalJsonTokens - totalToonTokens
-const totalSavingsPercent = ((totalSavings / totalJsonTokens) * 100).toFixed(1)
+const totalJsonSavings = totalJsonTokens - totalToonTokens
+const totalJsonSavingsPercent = ((totalJsonSavings / totalJsonTokens) * 100).toFixed(1)

-// Generate ASCII bar chart visualization
-const barChartSection = results
+const totalXmlSavings = totalXmlTokens - totalToonTokens
+const totalXmlSavingsPercent = ((totalXmlSavings / totalXmlTokens) * 100).toFixed(1)
+
+// Generate ASCII bar chart visualization (stacked compact format)
+const datasetRows = results
  .map((result) => {
-    const percentage = Number.parseFloat(result.savingsPercent)
+    const percentage = Number.parseFloat(result.jsonSavingsPercent)
    const bar = generateBarChart(100 - percentage) // Invert to show TOON tokens
-    const jsonStr = result.jsonTokens.toLocaleString('en-US')
    const toonStr = result.toonTokens.toLocaleString('en-US')
-    return `${result.emoji} ${result.name.padEnd(25)} ${bar}  ${toonStr.padStart(6)} tokens  (JSON: ${jsonStr.padStart(6)})  💰 ${result.savingsPercent}% saved`
+    const jsonStr = result.jsonTokens.toLocaleString('en-US')
+    const xmlStr = result.xmlTokens.toLocaleString('en-US')
+
+    const line1 = `${result.emoji} ${result.name.padEnd(25)} ${bar}  ${toonStr.padStart(6)} tokens`
+    const line2 = `                             vs JSON: ${jsonStr.padStart(6)}  💰 ${result.jsonSavingsPercent}% saved`
+    const line3 = `                             vs XML:  ${xmlStr.padStart(6)}  💰 ${result.xmlSavingsPercent}% saved`
+
+    return `${line1}\n${line2}\n${line3}`
  })
-  .join('\n')
+  .join('\n\n')
+
+// Add separator and totals row
+const separator = '─────────────────────────────────────────────────────────────────────'
+
+// Calculate bar for totals (TOON vs average of JSON+XML)
+const averageComparisonTokens = (totalJsonTokens + totalXmlTokens) / 2
+const totalPercentage = (totalToonTokens / averageComparisonTokens) * 100
+const totalBar = generateBarChart(totalPercentage)
+
+const totalLine1 = `Total                        ${totalBar}  ${totalToonTokens.toLocaleString('en-US').padStart(6)} tokens`
+const totalLine2 = `                             vs JSON: ${totalJsonTokens.toLocaleString('en-US').padStart(6)}  💰 ${totalJsonSavingsPercent}% saved`
+const totalLine3 = `                             vs XML:  ${totalXmlTokens.toLocaleString('en-US').padStart(6)}  💰 ${totalXmlSavingsPercent}% saved`
+
+const barChartSection = `${datasetRows}\n\n${separator}\n${totalLine1}\n${totalLine2}\n${totalLine3}`

 // Generate detailed examples (only for selected examples)
 const detailedExamples = results
@@ -108,9 +139,9 @@ const detailedExamples = results
    let displayData = result.data
    if (result.name === 'GitHub Repositories') {
      displayData = {
-        repositories: result.data.repositories.slice(0, 3).map((repo: any) => ({
+        repositories: result.data.repositories.slice(0, 3).map((repo: Record<string, any>) => ({
          ...repo,
-          description: repo.description?.slice(0, 80) + (repo.description?.length > 80 ? '...' : ''),
+          description: repo.description?.slice(0, 80) + (repo.description?.length > 80 ? '…' : ''),
        })),
      }
    }
@@ -124,7 +155,7 @@ const detailedExamples = results

 **Configuration:** ${result.description}

-**Savings:** ${result.savings.toLocaleString('en-US')} tokens (${result.savingsPercent}% reduction)
+**Savings:** ${result.jsonSavings.toLocaleString('en-US')} tokens (${result.jsonSavingsPercent}% reduction vs JSON)

 **JSON** (${result.jsonTokens.toLocaleString('en-US')} tokens):

@@ -146,8 +177,6 @@ const markdown = `### Token Efficiency
 ${barChartSection}
 \`\`\`

-**Total:** ${totalToonTokens.toLocaleString('en-US')} tokens (TOON) vs ${totalJsonTokens.toLocaleString('en-US')} tokens (JSON) → ${totalSavingsPercent}% savings
-
 <details>
 <summary><strong>View detailed examples</strong></summary>

@@ -170,23 +199,6 @@ function generateBarChart(percentage: number, maxWidth: number = 25): string {
  return '█'.repeat(filled) + '░'.repeat(empty)
 }

-// Generate user API response
-function generateUsers(count: number) {
-  return {
-    users: Array.from({ length: count }, (_, i) => ({
-      id: i + 1,
-      name: faker.person.fullName(),
-      email: faker.internet.email(),
-      role: faker.helpers.arrayElement(['admin', 'user', 'moderator']),
-      active: faker.datatype.boolean(),
-      createdAt: faker.date.past({ years: 2 }).toISOString(),
-      lastLogin: faker.date.recent({ days: 30 }).toISOString(),
-    })),
-    total: count,
-    page: 1,
-  }
-}
-
 // Generate nested e-commerce order
 function generateOrder() {
  return {
--- a/benchmarks/src/evaluate.ts
+++ b/benchmarks/src/evaluate.ts
@@ -10,6 +10,7 @@
 import type { LanguageModelV2 } from '@ai-sdk/provider'
 import type { EvaluationResult, Question } from './types'
 import { anthropic } from '@ai-sdk/anthropic'
+import { google } from '@ai-sdk/google'
 import { openai } from '@ai-sdk/openai'
 import { generateText } from 'ai'
 import { consola } from 'consola'
@@ -20,16 +21,18 @@ import { consola } from 'consola'
 export const models: Record<string, LanguageModelV2> = {
  'gpt-5-nano': openai('gpt-5-nano'),
  'claude-haiku-4-5': anthropic('claude-haiku-4-5-20251001'),
+  'gemini-2.5-flash': google('gemini-2.5-flash'),
 }

 /**
 * Evaluate a single question with a specific format and model
 */
 export async function evaluateQuestion(
-  { question, formatName, formattedData, model}:
-  { question: Question, formatName: string, formattedData: string, model: LanguageModelV2 },
+  { question, formatName, formattedData, model, modelName}:
+  { question: Question, formatName: string, formattedData: string, model: LanguageModelV2, modelName: string },
 ): Promise<EvaluationResult> {
-  const prompt = `Given the following data in ${formatName} format:
+  const prompt = `
+Given the following data in ${formatName} format:

 \`\`\`
 ${formattedData}
@@ -37,13 +40,14 @@ ${formattedData}

 Question: ${question.prompt}

-Provide only the direct answer, without any additional explanation or formatting.`
+Provide only the direct answer, without any additional explanation or formatting.
+`.trim()

  const startTime = performance.now()
  const { text, usage } = await generateText({
    model,
    prompt,
-    temperature: model.modelId.startsWith('gpt-') ? undefined : 0,
+    temperature: !model.modelId.startsWith('gpt-') ? 0 : undefined,
  })

  const latencyMs = performance.now() - startTime
@@ -56,7 +60,7 @@ Provide only the direct answer, without any additional explanation or formatting
  return {
    questionId: question.id,
    format: formatName,
-    model: model.modelId,
+    model: modelName,
    expected: question.groundTruth,
    actual: text.trim(),
    isCorrect,
@@ -93,9 +97,8 @@ Respond with only "YES" or "NO".`

  try {
    const { text } = await generateText({
-      model: models['claude-haiku-4-5']!,
+      model: models['gpt-5-nano']!,
      prompt,
-      temperature: 0,
    })

    return text.trim().toUpperCase() === 'YES'
--- a/benchmarks/src/report.ts
+++ b/benchmarks/src/report.ts
@@ -201,8 +201,8 @@ ${modelPerformance}

 - **Semantic validation**: LLM-as-judge validates responses semantically (not exact string matching).
 - **Token counting**: Using \`gpt-tokenizer\` with \`o200k_base\` encoding.
- **Question types**: Field retrieval, aggregation, and filtering tasks.
- **Real data**: Faker.js-generated datasets + GitHub repositories.
+- **Question types**: ~160 questions across field retrieval, aggregation, and filtering tasks.
+- **Datasets**: Faker.js-generated datasets (seeded) + GitHub repositories.

 </details>
 `.trimStart()
--- a/benchmarks/src/types.ts
+++ b/benchmarks/src/types.ts
@@ -1,7 +1,7 @@
 export interface Dataset {
  name: string
  description: string
-  data: any
+  data: Record<string, any>
 }

 export interface Question {