toon/README.md

Token-Oriented Object Notation (TOON)

Token-Oriented Object Notation is a compact, human-readable format designed for passing structured data to Large Language Models with significantly reduced token usage.

In other words, if YAML and CSV had a baby, optimized for LLM contexts. TOON borrows YAML's indentation-based structure for nested objects and CSV's tabular format for uniform data rows, then optimizes both for token efficiency in LLM contexts.

Tip

Wrap your JSON in encode() before sending it to LLMs and save ~1/2 of the token cost for structured data!

Why TOON?

AI is becoming cheaper and more accessible, but larger context windows allow for larger data inputs as well. LLM tokens still cost money – and standard JSON is verbose and token-expensive:

{
  "users": [
    { "id": 1, "name": "Alice", "role": "admin" },
    { "id": 2, "name": "Bob", "role": "user" }
  ]
}

TOON conveys the same information with fewer tokens:

users[2]{id,name,role}:
  1,Alice,admin
  2,Bob,user

Note

I built TOON to save tokens when sending large datasets to LLMs at work, where I tend to have uniform arrays of objects that benefit from the tabular format.

Key Features

• 💸 Token-efficient: typically 30–60% fewer tokens than JSON
• 🤿 LLM-friendly guardrails: explicit lengths and field lists help models validate output
• 🍱 Minimal syntax: removes redundant punctuation (braces, brackets, most quotes)
• 📐 Indentation-based structure: replaces braces with whitespace for better readability
• 🧺 Tabular arrays: declare keys once, then stream rows without repetition

Benchmarks

Token Efficiency

⭐ GitHub Repositories       ██████████████░░░░░░░░░░░   8,745 tokens  (JSON: 15,145)  💰 42.3% saved
📈 Analytics Time Series     ██████████░░░░░░░░░░░░░░░   3,631 tokens  (JSON:  9,024)  💰 59.8% saved
👥 API Response              ██████████████░░░░░░░░░░░   2,593 tokens  (JSON:  4,589)  💰 43.5% saved
🛒 E-commerce Order          ███████████████░░░░░░░░░░     203 tokens  (JSON:    338)  💰 39.9% saved

Total: 15,172 tokens (TOON) vs 29,096 tokens (JSON) → 47.9% savings

View detailed examples

⭐ GitHub Repositories

Configuration: Top 100 GitHub repositories with stars, forks, and metadata

Savings: 6,400 tokens (42.3% reduction)

JSON (15,145 tokens):

{
  "repositories": [
    {
      "id": 28457823,
      "name": "freeCodeCamp",
      "repo": "freeCodeCamp/freeCodeCamp",
      "description": "freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,...",
      "createdAt": "2014-12-24T17:49:19Z",
      "updatedAt": "2025-10-27T07:40:58Z",
      "pushedAt": "2025-10-26T11:31:08Z",
      "stars": 430828,
      "watchers": 8582,
      "forks": 42136,
      "defaultBranch": "main"
    },
    {
      "id": 132750724,
      "name": "build-your-own-x",
      "repo": "codecrafters-io/build-your-own-x",
      "description": "Master programming by recreating your favorite technologies from scratch.",
      "createdAt": "2018-05-09T12:03:18Z",
      "updatedAt": "2025-10-27T07:43:25Z",
      "pushedAt": "2025-10-10T18:45:01Z",
      "stars": 430102,
      "watchers": 6322,
      "forks": 40388,
      "defaultBranch": "master"
    },
    {
      "id": 21737465,
      "name": "awesome",
      "repo": "sindresorhus/awesome",
      "description": "😎 Awesome lists about all kinds of interesting topics",
      "createdAt": "2014-07-11T13:42:37Z",
      "updatedAt": "2025-10-27T07:44:27Z",
      "pushedAt": "2025-10-23T17:26:53Z",
      "stars": 409760,
      "watchers": 8016,
      "forks": 32015,
      "defaultBranch": "main"
    }
  ]
}

TOON (8,745 tokens):

repositories[3]{id,name,repo,description,createdAt,updatedAt,pushedAt,stars,watchers,forks,defaultBranch}:
  28457823,freeCodeCamp,freeCodeCamp/freeCodeCamp,"freeCodeCamp.org's open-source codebase and curriculum. Learn math, programming,...","2014-12-24T17:49:19Z","2025-10-27T07:40:58Z","2025-10-26T11:31:08Z",430828,8582,42136,main
  132750724,build-your-own-x,codecrafters-io/build-your-own-x,Master programming by recreating your favorite technologies from scratch.,"2018-05-09T12:03:18Z","2025-10-27T07:43:25Z","2025-10-10T18:45:01Z",430102,6322,40388,master
  21737465,awesome,sindresorhus/awesome,😎 Awesome lists about all kinds of interesting topics,"2014-07-11T13:42:37Z","2025-10-27T07:44:27Z","2025-10-23T17:26:53Z",409760,8016,32015,main

---

📈 Analytics Time Series

Configuration: 180 days of web metrics (views, clicks, conversions, revenue)

Savings: 5,393 tokens (59.8% reduction)

JSON (9,024 tokens):

{
  "metrics": [
    {
      "date": "2024-12-31",
      "views": 3769,
      "clicks": 400,
      "conversions": 59,
      "revenue": 198.98
    },
    {
      "date": "2025-01-01",
      "views": 5742,
      "clicks": 463,
      "conversions": 28,
      "revenue": 295.77
    },
    {
      "date": "2025-01-02",
      "views": 3669,
      "clicks": 336,
      "conversions": 102,
      "revenue": 624.23
    },
    {
      "date": "2025-01-03",
      "views": 1332,
      "clicks": 304,
      "conversions": 99,
      "revenue": 113.06
    },
    {
      "date": "2025-01-04",
      "views": 1444,
      "clicks": 222,
      "conversions": 88,
      "revenue": 986.69
    }
  ]
}

TOON (3,631 tokens):

metrics[5]{date,views,clicks,conversions,revenue}:
  2024-12-31,3769,400,59,198.98
  2025-01-01,5742,463,28,295.77
  2025-01-02,3669,336,102,624.23
  2025-01-03,1332,304,99,113.06
  2025-01-04,1444,222,88,986.69

Note

Measured with gpt-tokenizer using o200k_base encoding (used by GPT-5 and other modern models). Savings will vary across models and tokenizers.

Retrieval Accuracy

Tested across 2 LLMs with data retrieval tasks:

gpt-4o-mini          ██████████████░░░░░░ 72.3% accuracy
claude-haiku-4-5     ███████████████░░░░░ 76.7% accuracy

TOON achieves 73.9% accuracy (vs JSON's 73.6%) while using 46.3% fewer tokens.

Format	Accuracy	Average Tokens
toon	73.9%	4.678
json	73.6%	8.713
markdown-kv	73.6%	8.649
csv	72.3%	4.745
yaml	71.7%	7.091

View detailed breakdown by dataset and model

Performance by Dataset

Uniform employee records (TOON optimal format)

Format	Accuracy	Tokens	Correct/Total
toon	72.4%	2.483	84/116
csv	69.0%	2.337	80/116
yaml	68.1%	4.969	79/116
markdown-kv	68.1%	6.270	79/116
json	68.1%	6.347	79/116

E-commerce orders with nested structures

Format	Accuracy	Tokens	Correct/Total
toon	84.1%	5.967	74/88
csv	83.0%	6.735	73/88
yaml	81.8%	7.328	72/88
markdown-kv	86.4%	9.110	76/88
json	84.1%	9.694	74/88

Time-series analytics data

Format	Accuracy	Tokens	Correct/Total
csv	72.4%	1.393	42/58
toon	70.7%	1.515	41/58
yaml	72.4%	2.938	42/58
json	74.1%	3.665	43/58
markdown-kv	70.7%	3.779	41/58

Popular GitHub repositories

Format	Accuracy	Tokens	Correct/Total
toon	64.3%	8.745	36/56
csv	62.5%	8.513	35/56
json	67.9%	15.145	38/56
markdown-kv	67.9%	15.436	38/56
yaml	62.5%	13.129	35/56

Performance by Model

gpt-4o-mini

Format	Accuracy	Correct/Total
toon	72.3%	115/159
json	71.7%	114/159
markdown-kv	70.4%	112/159
csv	69.2%	110/159
yaml	68.6%	109/159

claude-haiku-4-5

Format	Accuracy	Correct/Total
markdown-kv	76.7%	122/159
toon	75.5%	120/159
json	75.5%	120/159
csv	75.5%	120/159
yaml	74.8%	119/159

Methodology

• Semantic validation: LLM-as-judge validates responses semantically (not exact string matching).
• Token counting: Using gpt-tokenizer with o200k_base encoding.
• Question types: Field retrieval, aggregation, and filtering tasks.
• Real data: Faker.js-generated datasets + GitHub repositories.

Installation

# npm
npm install @byjohann/toon

# pnpm
pnpm add @byjohann/toon

# yarn
yarn add @byjohann/toon

Quick Start

import { encode } from '@byjohann/toon'

const data = {
  user: {
    id: 123,
    name: 'Ada',
    tags: ['reading', 'gaming'],
    active: true,
    preferences: []
  }
}

console.log(encode(data))

Output:

user:
  id: 123
  name: Ada
  tags[2]: reading,gaming
  active: true
  preferences[0]:

Canonical Formatting Rules

TOON formatting is deterministic and minimal:

• Indentation: 2 spaces per nesting level.
• Lines:
  • key: value for primitives (single space after colon).
  • key: for nested/empty objects (no trailing space on that line).
• Arrays:
  • Delimiter encoding: Comma delimiters are implicit in array headers (e.g., tags[3]:, items[2]{id,name}:). Tab and pipe delimiters are explicitly shown in array headers (e.g., tags[3|]:, items[2 ]{id name}:).
  • Primitive arrays inline: key[N]: v1,v2 (comma) or key[N<delim>]: v1<delim>v2 (tab/pipe).
  • Tabular arrays: key[N]{f1,f2}: … (comma) or key[N<delim>]{f1<delim>f2}: … (tab/pipe).
  • List items: two spaces, hyphen, space (" - …").
• Whitespace invariants:
  • No trailing spaces at end of any line.
  • No trailing newline at end of output.

Format Overview

Objects

Simple objects with primitive values:

encode({
  id: 123,
  name: 'Ada',
  active: true
})

id: 123
name: Ada
active: true

Nested objects:

encode({
  user: {
    id: 123,
    name: 'Ada'
  }
})

user:
  id: 123
  name: Ada

Arrays

Tip

TOON includes the array length in brackets (e.g., items[3]). When using comma delimiters (default), the delimiter is implicit. When using tab or pipe delimiters, the delimiter is explicitly shown in the header (e.g., tags[2|] or [2 ]). This encoding helps LLMs identify the delimiter and track the number of elements, reducing errors when generating or validating structured output.

Primitive Arrays (Inline)

encode({
  tags: ['admin', 'ops', 'dev']
})

tags[3]: admin,ops,dev

Arrays of Objects (Tabular)

When all objects share the same primitive fields, TOON uses an efficient tabular format:

encode({
  items: [
    { sku: 'A1', qty: 2, price: 9.99 },
    { sku: 'B2', qty: 1, price: 14.5 }
  ]
})

items[2]{sku,qty,price}:
  A1,2,9.99
  B2,1,14.5

Tabular formatting applies recursively: nested arrays of objects (whether as object properties or inside list items) also use tabular format if they meet the same requirements.

encode({
  items: [
    {
      users: [
        { id: 1, name: 'Ada' },
        { id: 2, name: 'Bob' }
      ],
      status: 'active'
    }
  ]
})

items[1]:
  - users[2]{id,name}:
    1,Ada
    2,Bob
    status: active

Mixed and Non-Uniform Arrays

Arrays that don't meet the tabular requirements use list format:

items[3]:
  - 1
  - a: 1
  - text

When objects appear in list format, the first field is placed on the hyphen line:

items[2]:
  - id: 1
    name: First
  - id: 2
    name: Second
    extra: true

Note

Nested array indentation: When the first field of a list item is an array (primitive, tabular, or nested), its contents are indented two spaces under the header line, and subsequent fields of the same object appear at that same indentation level. This remains unambiguous because list items begin with "- ", tabular arrays declare a fixed row count in their header, and object fields contain ":".

Arrays of Arrays

When you have arrays containing primitive inner arrays:

encode({
  pairs: [
    [1, 2],
    [3, 4]
  ]
})

pairs[2]:
  - [2]: 1,2
  - [2]: 3,4

Empty Arrays and Objects

Empty containers have special representations:

encode({ items: [] }) // items[0]:
encode([]) // [0]:
encode({}) // (empty output)
encode({ config: {} }) // config:

Quoting Rules

TOON quotes strings only when necessary to maximize token efficiency. Inner spaces are allowed; leading or trailing spaces force quotes. Unicode and emoji are safe unquoted.

Note

When using alternative delimiters (tab or pipe), the quoting rules adapt automatically. Strings containing the active delimiter will be quoted, while other delimiters remain safe.

Keys

Keys are quoted when any of the following is true:

Condition	Examples
Contains spaces, commas, colons, quotes, control chars	"full name", "a,b", "order:id", "tab\there"
Contains brackets or braces	"[index]", "{key}"
Leading hyphen	"-lead"
Numeric-only key	"123"
Empty key	""

Notes:

• Quotes and control characters in keys are escaped (e.g., "he said \"hi\"", "line\nbreak").

String Values

String values are quoted when any of the following is true:

Condition	Examples
Empty string	""
Contains active delimiter, colon, quote, backslash, or control chars	"a,b" (comma), "a\tb" (tab), "a|b" (pipe), "a:b", "say \"hi\"", "C:\\Users", "line1\\nline2"
Leading or trailing spaces	" padded ", " "
Looks like boolean/number/null	"true", "false", "null", "42", "-3.14", "1e-6", "05"
Starts with "- " (list-like)	"- item"
Looks like structural token	"[5]", "{key}", "[3]: x,y"

Important

Delimiter-aware quoting: Unquoted strings never contain : or the active delimiter. This makes TOON reliably parseable with simple heuristics: split key/value on first : , and split array values on the delimiter declared in the array header. When using tab or pipe delimiters, commas don't need quoting – only the active delimiter triggers quoting for both array values and object values.

Examples

note: "hello, world"
items[3]: foo,"true","- item"
hello 👋 world         // unquoted
" padded "             // quoted
value: null            // null value
name: ""               // empty string (quoted)
text: "line1\nline2"   // multi-line string (escaped)

Tabular Format Requirements

For arrays of objects to use the efficient tabular format, all of the following must be true:

Requirement	Detail
All elements are objects	No primitives in the array
Identical key sets	No missing or extra keys across rows
Primitive values only	No nested arrays or objects
Header delimiter	Comma is implicit in headers ([N]{f1,f2}); tab and pipe are explicit ([N ]{f1 f2}, `[N
Header key order	Taken from the first object
Header key quoting	Same rules as object keys; keys containing the active delimiter must be quoted
Row value quoting	Same rules as string values; values containing the active delimiter must be quoted

If any condition fails, TOON falls back to list format.

Type Conversions

Some non-JSON types are automatically normalized for LLM-safe output:

Input	Output
Number (finite)	Decimal form, no scientific notation; -0 → 0
Number (NaN, ±Infinity)	null
BigInt	Decimal digits (no quotes)
Date	ISO string in quotes (e.g., "2025-01-01T00:00:00.000Z")
undefined	null
function	null
symbol	null

Number normalization examples:

-0    → 0
1e6   → 1000000
1e-6  → 0.000001

API

encode(value: unknown, options?: EncodeOptions): string

Converts any JSON-serializable value to TOON format.

Parameters:

• value – Any JSON-serializable value (object, array, primitive, or nested structure). Non-JSON-serializable values (functions, symbols, undefined, non-finite numbers) are converted to null. Dates are converted to ISO strings, and BigInts are emitted as decimal integers (no quotes).
• options – Optional encoding options:
  • indent?: number – Number of spaces per indentation level (default: 2)
  • delimiter?: ',' | '\t' | '|' – Delimiter for array values and tabular rows (default: ',')
  • lengthMarker?: '#' | false – Optional marker to prefix array lengths (default: false)

Returns:

A TOON-formatted string with no trailing newline or spaces.

Example:

import { encode } from '@byjohann/toon'

const items = [
  { sku: 'A1', qty: 2, price: 9.99 },
  { sku: 'B2', qty: 1, price: 14.5 }
]

console.log(encode({ items }))

Output:

items[2]{sku,qty,price}:
  A1,2,9.99
  B2,1,14.5

Delimiter Options

The delimiter option allows you to choose between comma (default), tab, or pipe delimiters for array values and tabular rows. Alternative delimiters can provide additional token savings in specific contexts.

Tab Delimiter (\t)

Using tab delimiters instead of commas can reduce token count further, especially for tabular data:

import { encode } from '@byjohann/toon'

const data = {
  items: [
    { sku: 'A1', name: 'Widget', qty: 2, price: 9.99 },
    { sku: 'B2', name: 'Gadget', qty: 1, price: 14.5 }
  ]
}

console.log(encode(data, { delimiter: '\t' }))

Output:

items[2	]{sku	name	qty	price}:
  A1	Widget	2	9.99
  B2	Gadget	1	14.5

Benefits:

• Tabs are single characters and often tokenize more efficiently than commas.
• Tabs rarely appear in natural text, reducing the need for quote-escaping.
• The delimiter is explicitly encoded in the array header, making it self-descriptive.

Considerations:

• Some terminals and editors may collapse or expand tabs visually.
• String values containing tabs will still require quoting.

Pipe Delimiter (|)

Pipe delimiters offer a middle ground between commas and tabs:

console.log(encode(data, { delimiter: '|' }))

Output:

items[2|]{sku|name|qty|price}:
  A1|Widget|2|9.99
  B2|Gadget|1|14.5

Length Marker Option

The lengthMarker option adds an optional hash (#) prefix to array lengths to emphasize that the bracketed value represents a count, not an index:

import { encode } from '@byjohann/toon'

const data = {
  tags: ['reading', 'gaming', 'coding'],
  items: [
    { sku: 'A1', qty: 2, price: 9.99 },
    { sku: 'B2', qty: 1, price: 14.5 },
  ],
}

console.log(encode(data, { lengthMarker: '#' }))
// tags[#3]: reading,gaming,coding
// items[#2]{sku,qty,price}:
//   A1,2,9.99
//   B2,1,14.5

// Works with custom delimiters
console.log(encode(data, { lengthMarker: '#', delimiter: '|' }))
// tags[#3|]: reading|gaming|coding
// items[#2|]{sku|qty|price}:
//   A1|2|9.99
//   B2|1|14.5

Using TOON in LLM Prompts

When incorporating TOON into your LLM workflows:

• Wrap TOON data in a fenced code block in your prompt.
• Tell the model: "Do not add extra punctuation or spaces; follow the exact TOON format."
• When asking the model to generate TOON, specify the same rules (2-space indentation, no trailing spaces, quoting rules).

Notes and Limitations

• Token counts vary by tokenizer and model. Benchmarks use a GPT-style tokenizer (cl100k/o200k); actual savings will differ with other models (e.g., SentencePiece).
• TOON is designed for LLM contexts where human readability and token efficiency matter. It's not a drop-in replacement for JSON in APIs or storage.
• Tabular arrays require all objects to have exactly the same keys with primitive values only. Arrays with mixed types (primitives + objects/arrays), non-uniform objects, or nested structures will use a more verbose list format.
• Object key order is preserved from the input. In tabular arrays, header order follows the first object's keys.
• Arrays mixing primitives and objects/arrays always use list form:

  items[2]:
    - a: 1
    - [2]: 1,2
  

• Deterministic formatting: 2-space indentation, stable key order, no trailing spaces/newline.

Quick Reference

// Object
{ id: 1, name: 'Ada' }          → id: 1
                                  name: Ada

// Nested object
{ user: { id: 1 } }             → user:
                                    id: 1

// Primitive array (inline)
{ tags: ['foo', 'bar'] }        → tags[2]: foo,bar

// Tabular array (uniform objects)
{ items: [                      → items[2]{id,qty}:
  { id: 1, qty: 5 },                1,5
  { id: 2, qty: 3 }                 2,3
]}

// Mixed / non-uniform (list)
{ items: [1, { a: 1 }, 'x'] }   → items[3]:
                                    - 1
                                    - a: 1
                                    - x

// Array of arrays
{ pairs: [[1, 2], [3, 4]] }     → pairs[2]:
                                    - [2]: 1,2
                                    - [2]: 3,4

// Root array
['x', 'y']                      → [2]: x,y

// Empty containers
{}                              → (empty output)
{ items: [] }                   → items[0]:

// Special quoting
{ note: 'hello, world' }        → note: "hello, world"
{ items: ['true', true] }       → items[2]: "true",true

License

MIT License © 2025-PRESENT Johann Schopplich