docs: add dedicated docs website

2026-01-29 23:34:10 +08:00 · 2025-11-18 07:23:10 +01:00
parent 3e08f3b72b
commit 4b4f7c05f9
38 changed files with 4399 additions and 541 deletions
--- a/docs/reference/api.md
+++ b/docs/reference/api.md
@@ -0,0 +1,296 @@
+# API Reference
+
+TypeScript/JavaScript API documentation for the `@toon-format/toon` package. For format rules, see the [Format Overview](/guide/format-overview) or the [Specification](/reference/spec). For other languages, see [Implementations](/ecosystem/implementations).
+
+## Installation
+
+::: code-group
+
+```bash [npm]
+npm install @toon-format/toon
+```
+
+```bash [pnpm]
+pnpm add @toon-format/toon
+```
+
+```bash [yarn]
+yarn add @toon-format/toon
+```
+
+:::
+
+## `encode(value, options?)`
+
+Converts any JSON-serializable value to TOON format.
+
+```ts
+import { encode } from '@toon-format/toon'
+
+const toon = encode(data, {
+  indent: 2,
+  delimiter: ',',
+  keyFolding: 'off',
+  flattenDepth: Infinity
+})
+```
+
+### Parameters
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `value` | `unknown` | Any JSON-serializable value (object, array, primitive, or nested structure) |
+| `options` | `EncodeOptions?` | Optional encoding options (see below) |
+
+### Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `indent` | `number` | `2` | Number of spaces per indentation level |
+| `delimiter` | `','` \| `'\t'` \| `'\|'` | `','` | Delimiter for array values and tabular rows |
+| `keyFolding` | `'off'` \| `'safe'` | `'off'` | Enable key folding to collapse single-key wrapper chains into dotted paths |
+| `flattenDepth` | `number` | `Infinity` | Maximum number of segments to fold when `keyFolding` is enabled (values 0-1 have no practical effect) |
+
+### Return Value
+
+Returns a TOON-formatted string with no trailing newline or spaces.
+
+### Type Normalization
+
+Non-JSON-serializable values are normalized before encoding:
+
+| Input | Output |
+|-------|--------|
+| Finite number | Canonical decimal (no exponent, no leading/trailing zeros: `1e6` → `1000000`, `-0` → `0`) |
+| `NaN`, `Infinity`, `-Infinity` | `null` |
+| `BigInt` (within safe range) | Number |
+| `BigInt` (out of range) | Quoted decimal string (e.g., `"9007199254740993"`) |
+| `Date` | ISO string in quotes (e.g., `"2025-01-01T00:00:00.000Z"`) |
+| `undefined`, `function`, `symbol` | `null` |
+
+### Example
+
+```ts
+import { encode } from '@toon-format/toon'
+
+const items = [
+  { sku: 'A1', qty: 2, price: 9.99 },
+  { sku: 'B2', qty: 1, price: 14.5 }
+]
+
+console.log(encode({ items }))
+```
+
+**Output:**
+
+```yaml
+items[2]{sku,qty,price}:
+  A1,2,9.99
+  B2,1,14.5
+```
+
+### Delimiter Options
+
+::: code-group
+
+```ts [Comma (default)]
+encode(data, { delimiter: ',' })
+```
+
+```ts [Tab]
+encode(data, { delimiter: '\t' })
+```
+
+```ts [Pipe]
+encode(data, { delimiter: '|' })
+```
+
+:::
+
+::: details Why Use Tab Delimiters?
+Tab delimiters (`\t`) often tokenize more efficiently than commas:
+- Tabs are single characters
+- Tabs rarely appear in natural text, reducing quote-escaping
+- The delimiter is explicitly encoded in the array header
+
+Example:
+
+```yaml
+items[2	]{sku	name	qty	price}:
+  A1	Widget	2	9.99
+  B2	Gadget	1	14.5
+```
+
+For maximum token savings on large tabular data, combine tab delimiters with key folding:
+```ts
+encode(data, { delimiter: '\t', keyFolding: 'safe' })
+```
+:::
+
+## `decode(input, options?)`
+
+Converts a TOON-formatted string back to JavaScript values.
+
+```ts
+import { decode } from '@toon-format/toon'
+
+const data = decode(toon, {
+  indent: 2,
+  strict: true,
+  expandPaths: 'off'
+})
+```
+
+### Parameters
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `input` | `string` | A TOON-formatted string to parse |
+| `options` | `DecodeOptions?` | Optional decoding options (see below) |
+
+### Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `indent` | `number` | `2` | Expected number of spaces per indentation level |
+| `strict` | `boolean` | `true` | Enable strict validation (array counts, indentation, delimiter consistency) |
+| `expandPaths` | `'off'` \| `'safe'` | `'off'` | Enable path expansion to reconstruct dotted keys into nested objects (pairs with `keyFolding: 'safe'`) |
+
+### Return Value
+
+Returns a JavaScript value (object, array, or primitive) representing the parsed TOON data.
+
+### Strict Mode
+
+By default (`strict: true`), the decoder validates input strictly:
+
+- **Invalid escape sequences**: Throws on `\x`, unterminated strings
+- **Syntax errors**: Throws on missing colons, malformed headers
+- **Array length mismatches**: Throws when declared length doesn't match actual count
+- **Delimiter mismatches**: Throws when row delimiters don't match header
+- **Indentation errors**: Throws when leading spaces aren't exact multiples of `indentSize`
+
+Set `strict: false` to skip validation for lenient parsing.
+
+### Example
+
+```ts
+import { decode } from '@toon-format/toon'
+
+const toon = `
+items[2]{sku,qty,price}:
+  A1,2,9.99
+  B2,1,14.5
+`
+
+const data = decode(toon)
+console.log(data)
+```
+
+**Output:**
+
+```json
+{
+  "items": [
+    { "sku": "A1", "qty": 2, "price": 9.99 },
+    { "sku": "B2", "qty": 1, "price": 14.5 }
+  ]
+}
+```
+
+### Path Expansion
+
+When `expandPaths: 'safe'` is enabled, dotted keys are split into nested objects:
+
+```ts
+import { decode } from '@toon-format/toon'
+
+const toon = 'data.metadata.items[2]: a,b'
+
+const data = decode(toon, { expandPaths: 'safe' })
+console.log(data)
+// { data: { metadata: { items: ['a', 'b'] } } }
+```
+
+This pairs with `keyFolding: 'safe'` for lossless round-trips.
+
+::: details Expansion Conflict Resolution
+When multiple expanded keys construct overlapping paths, the decoder merges them recursively:
+- **Object + Object**: Deep merge recursively
+- **Object + Non-object** (array or primitive): Conflict
+  - With `strict: true` (default): Error
+  - With `strict: false`: Last-write-wins (LWW)
+
+Example conflict (strict mode):
+
+```ts
+const toon = 'a.b: 1\na: 2'
+decode(toon, { expandPaths: 'safe', strict: true })
+// Error: "Expansion conflict at path 'a' (object vs primitive)"
+```
+
+Example conflict (lenient mode):
+
+```ts
+const toon = 'a.b: 1\na: 2'
+decode(toon, { expandPaths: 'safe', strict: false })
+// { a: 2 } (last write wins)
+```
+:::
+
+## Round-Trip Compatibility
+
+TOON provides lossless round-trips after normalization:
+
+```ts
+import { decode, encode } from '@toon-format/toon'
+
+const original = {
+  users: [
+    { id: 1, name: 'Alice', role: 'admin' },
+    { id: 2, name: 'Bob', role: 'user' }
+  ]
+}
+
+const toon = encode(original)
+const restored = decode(toon)
+
+console.log(JSON.stringify(original) === JSON.stringify(restored))
+// true
+```
+
+### With Key Folding
+
+```ts
+import { decode, encode } from '@toon-format/toon'
+
+const original = { data: { metadata: { items: ['a', 'b'] } } }
+
+// Encode with folding
+const toon = encode(original, { keyFolding: 'safe' })
+// → "data.metadata.items[2]: a,b"
+
+// Decode with expansion
+const restored = decode(toon, { expandPaths: 'safe' })
+// → { data: { metadata: { items: ['a', 'b'] } } }
+
+console.log(JSON.stringify(original) === JSON.stringify(restored))
+// true
+```
+
+## Types
+
+```ts
+interface EncodeOptions {
+  indent?: number
+  delimiter?: ',' | '\t' | '|'
+  keyFolding?: 'off' | 'safe'
+  flattenDepth?: number
+}
+
+interface DecodeOptions {
+  indent?: number
+  strict?: boolean
+  expandPaths?: 'off' | 'safe'
+}
+```
--- a/docs/reference/spec.md
+++ b/docs/reference/spec.md
@@ -0,0 +1,137 @@
+# Specification
+
+The [TOON specification](https://github.com/toon-format/spec) is the authoritative reference for implementing encoders, decoders, and validators. It defines the concrete syntax, normative encoding/decoding behavior, and strict-mode validation rules.
+
+You don't need this page to *use* TOON. It's mainly for implementers and contributors. If you're looking to learn how to use TOON, start with the [Getting Started](/guide/getting-started) guide instead.
+
+> [!TIP]
+> TOON is production-ready, but also an idea in progress. Nothing's set in stone – help shape where it goes by contributing to the spec or sharing feedback.
+
+## Current Version
+
+**Spec v2.0** (2025-11-10) is the current stable version.
+
+## Guided Tour of the Spec
+
+### Core Concepts
+
+**[§1 Terminology and Conventions](https://github.com/toon-format/spec/blob/main/SPEC.md#1-terminology-and-conventions)**
+Defines key terms like "indentation level", "active delimiter", "strict mode", and RFC2119 keywords (MUST, SHOULD, MAY).
+
+**[§2 Data Model](https://github.com/toon-format/spec/blob/main/SPEC.md#2-data-model)**
+Specifies the JSON data model (objects, arrays, primitives), array/object ordering requirements, and canonical number formatting (no exponent notation, no leading/trailing zeros).
+
+**[§3 Encoding Normalization](https://github.com/toon-format/spec/blob/main/SPEC.md#3-encoding-normalization-reference-encoder)**
+Defines how non-JSON types (Date, BigInt, NaN, Infinity, undefined, etc.) are normalized before encoding. Required reading for encoder implementers.
+
+**[§4 Decoding Interpretation](https://github.com/toon-format/spec/blob/main/SPEC.md#4-decoding-interpretation-reference-decoder)**
+Specifies how decoders map text tokens to host values (quoted strings, unquoted primitives, numeric parsing with leading-zero handling). Decoders default to strict mode (`strict = true`) in the reference implementation; strict-mode errors are enumerated in §14.
+
+### Syntax Rules
+
+**[§5 Concrete Syntax and Root Form](https://github.com/toon-format/spec/blob/main/SPEC.md#5-concrete-syntax-and-root-form)**
+Defines TOON's line-oriented, indentation-based notation and how to determine whether the root is an object, array, or primitive.
+
+**[§6 Header Syntax](https://github.com/toon-format/spec/blob/main/SPEC.md#6-header-syntax-normative)**
+Normative ABNF grammar for array headers: `key[N<delim?>]{fields}:`. Specifies bracket segments, delimiter symbols, and field lists.
+
+**[§7 Strings and Keys](https://github.com/toon-format/spec/blob/main/SPEC.md#7-strings-and-keys)**
+Complete quoting rules (when strings MUST be quoted), escape sequences (only `\\`, `\"`, `\n`, `\r`, `\t` are valid), and key encoding requirements.
+
+**[§8 Objects](https://github.com/toon-format/spec/blob/main/SPEC.md#8-objects)**
+Object field encoding (key: value), nesting rules, key order preservation, and empty object handling.
+
+**[§9 Arrays](https://github.com/toon-format/spec/blob/main/SPEC.md#9-arrays)**
+Covers all array forms: primitive (inline), arrays of objects (tabular), mixed/non-uniform (list), and arrays of arrays. Includes tabular detection requirements.
+
+**[§10 Objects as List Items](https://github.com/toon-format/spec/blob/main/SPEC.md#10-objects-as-list-items)**
+Indentation rules for objects appearing in list items (first field on hyphen line, nested object rules).
+
+**[§11 Delimiters](https://github.com/toon-format/spec/blob/main/SPEC.md#11-delimiters)**
+Delimiter scoping (document vs active), delimiter-aware quoting, and parsing rules for comma/tab/pipe delimiters.
+
+**[§12 Indentation and Whitespace](https://github.com/toon-format/spec/blob/main/SPEC.md#12-indentation-and-whitespace)**
+Encoding requirements (consistent spaces, no tabs in indentation, no trailing spaces/newlines) and decoding rules (strict vs non-strict indentation handling).
+
+### Conformance and Validation
+
+**[§13 Conformance and Options](https://github.com/toon-format/spec/blob/main/SPEC.md#13-conformance-and-options)**
+Defines conformance classes (encoder, decoder, validator), required options, and conformance checklists.
+
+**[§13.4 Key Folding and Path Expansion](https://github.com/toon-format/spec/blob/main/SPEC.md#134-key-folding-and-path-expansion)**
+Optional encoder feature (key folding) and decoder feature (path expansion) for collapsing/expanding dotted paths. Specifies safety requirements and conflict resolution.
+
+**[§14 Strict Mode Errors and Diagnostics](https://github.com/toon-format/spec/blob/main/SPEC.md#14-strict-mode-errors-and-diagnostics-authoritative-checklist)**
+**Authoritative checklist** of all strict-mode errors: array count mismatches, syntax errors, indentation errors, structural errors, and path expansion conflicts.
+
+### Implementation Guidance
+
+**[§19 TOON Core Profile](https://github.com/toon-format/spec/blob/main/SPEC.md#19-toon-core-profile-normative-subset)**
+Normative subset of the most common, memory-friendly rules. Useful for minimal implementations.
+
+**[Appendix G: Host Type Normalization Examples](https://github.com/toon-format/spec/blob/main/SPEC.md#appendix-g-host-type-normalization-examples-informative)**
+Non-normative guidance for Go, JavaScript, Python, and Rust implementations on normalizing language-specific types.
+
+**[Appendix C: Test Suite and Compliance](https://github.com/toon-format/spec/blob/main/SPEC.md#appendix-c-test-suite-and-compliance-informative)**
+Reference test suite at [github.com/toon-format/spec/tree/main/tests](https://github.com/toon-format/spec/tree/main/tests) for validating implementations.
+
+## Spec Sections at a Glance
+
+| Section | Topic | When to Read |
+|---------|-------|--------------|
+| §1-4 | Data model, normalization, decoding | Implementing encoders/decoders |
+| §5-6 | Syntax, headers, root form | Implementing parsers |
+| §7 | Strings, keys, quoting, escaping | Implementing string handling |
+| §8-10 | Objects, arrays, list items | Implementing structure encoding |
+| §11-12 | Delimiters, indentation, whitespace | Implementing formatting and validation |
+| §13 | Conformance, options, key folding | Implementing options and features |
+| §14 | Strict-mode errors | Implementing validators |
+| §19 | Core profile | Minimal implementations |
+
+## Conformance Checklists
+
+The spec includes three conformance checklists:
+
+### [Encoder Checklist (§13.1)](https://github.com/toon-format/spec/blob/main/SPEC.md#131-encoder-conformance-checklist)
+
+Key requirements:
+- Produce UTF-8 with LF line endings
+- Use consistent indentation (default 2 spaces, no tabs)
+- Escape only `\\`, `\"`, `\n`, `\r`, `\t` in quoted strings
+- Quote strings with active delimiter, colon, or structural characters
+- Emit array lengths `[N]` matching actual count
+- Preserve object key order
+- Normalize numbers to non-exponential decimal form
+- Convert `-0` to `0`, `NaN`/±Infinity to `null`
+- No trailing spaces or trailing newline
+
+### [Decoder Checklist (§13.2)](https://github.com/toon-format/spec/blob/main/SPEC.md#132-decoder-conformance-checklist)
+
+Key requirements:
+- Parse array headers per §6 (length, delimiter, fields)
+- Split inline arrays and tabular rows using active delimiter only
+- Unescape quoted strings with only valid escapes
+- Type unquoted primitives: true/false/null → booleans/null, numeric → number, else → string
+- Enforce strict-mode rules when `strict=true`
+- Preserve array order and object key order
+
+### [Validator Checklist (§13.3)](https://github.com/toon-format/spec/blob/main/SPEC.md#133-validator-conformance-checklist)
+
+Validators should verify:
+- Structural conformance (headers, indentation, list markers)
+- Whitespace invariants (no trailing spaces/newlines)
+- Delimiter consistency between headers and rows
+- Array length counts match declared `[N]`
+- All strict-mode requirements
+
+## Versioning
+
+The spec uses semantic versioning (major.minor):
+- **Major version** (e.g., v2.0): Breaking changes, incompatible with previous versions
+- **Minor version** (e.g., v1.5 → v1.6): Clarifications, additional requirements, or backward-compatible additions
+
+See [Appendix D: Document Changelog](https://github.com/toon-format/spec/blob/main/SPEC.md#appendix-d-document-changelog-informative) for detailed version history.
+
+## Contributing to the Spec
+
+The spec is community-maintained at [github.com/toon-format/spec](https://github.com/toon-format/spec). We welcome contributions of all kinds: reporting ambiguities or errors, proposing clarifications and examples, adding test cases to the reference suite, or discussing edge cases and normative behavior. Your feedback helps shape the format.
--- a/docs/reference/syntax-cheatsheet.md
+++ b/docs/reference/syntax-cheatsheet.md
@@ -0,0 +1,333 @@
+# Syntax Cheatsheet
+
+Quick reference for mapping JSON to TOON format. For rigorous, normative syntax rules and edge cases, see the [specification](/reference/spec).
+
+## Objects
+
+::: code-group
+
+```json [JSON]
+{
+  "id": 1,
+  "name": "Ada"
+}
+```
+
+```yaml [TOON]
+id: 1
+name: Ada
+```
+
+:::
+
+## Nested Objects
+
+::: code-group
+
+```json [JSON]
+{
+  "user": {
+    "id": 1,
+    "name": "Ada"
+  }
+}
+```
+
+```yaml [TOON]
+user:
+  id: 1
+  name: Ada
+```
+
+:::
+
+## Primitive Arrays
+
+::: code-group
+
+```json [JSON]
+{
+  "tags": ["foo", "bar", "baz"]
+}
+```
+
+```yaml [TOON]
+tags[3]: foo,bar,baz
+```
+
+:::
+
+## Tabular Arrays
+
+::: code-group
+
+```json [JSON]
+{
+  "items": [
+    { "id": 1, "qty": 5 },
+    { "id": 2, "qty": 3 }
+  ]
+}
+```
+
+```yaml [TOON]
+items[2]{id,qty}:
+  1,5
+  2,3
+```
+
+:::
+
+## Mixed / Non-Uniform Arrays
+
+::: code-group
+
+```json [JSON]
+{
+  "items": [1, { "a": 1 }, "x"]
+}
+```
+
+```yaml [TOON]
+items[3]:
+  - 1
+  - a: 1
+  - x
+```
+
+:::
+
+## Arrays of Arrays
+
+::: code-group
+
+```json [JSON]
+{
+  "pairs": [[1, 2], [3, 4]]
+}
+```
+
+```yaml [TOON]
+pairs[2]:
+  - [2]: 1,2
+  - [2]: 3,4
+```
+
+:::
+
+## Root Arrays
+
+::: code-group
+
+```json [JSON]
+["x", "y", "z"]
+```
+
+```yaml [TOON]
+[3]: x,y,z
+```
+
+:::
+
+## Empty Containers
+
+::: code-group
+
+```json [Empty Object]
+{}
+```
+
+```yaml [Empty Object]
+(empty output)
+```
+
+:::
+
+::: code-group
+
+```json [Empty Array]
+{
+  "items": []
+}
+```
+
+```yaml [Empty Array]
+items[0]:
+```
+
+:::
+
+## Quoting Special Cases
+
+### Strings That Look Like Literals
+
+::: code-group
+
+```json [JSON]
+{
+  "version": "123",
+  "enabled": "true"
+}
+```
+
+```yaml [TOON]
+version: "123"
+enabled: "true"
+```
+
+:::
+
+These strings must be quoted because they look like numbers/booleans.
+
+### Strings with Active Delimiter
+
+::: code-group
+
+```json [JSON]
+{
+  "note": "hello, world"
+}
+```
+
+```yaml [TOON]
+note: "hello, world"
+```
+
+:::
+
+Strings containing the active delimiter (comma by default) must be quoted.
+
+### Strings with Leading/Trailing Spaces
+
+::: code-group
+
+```json [JSON]
+{
+  "message": " padded "
+}
+```
+
+```yaml [TOON]
+message: " padded "
+```
+
+:::
+
+### Empty String
+
+::: code-group
+
+```json [JSON]
+{
+  "name": ""
+}
+```
+
+```yaml [TOON]
+name: ""
+```
+
+:::
+
+## Quoting Rules Summary
+
+Strings **must** be quoted if they:
+
+- Are empty (`""`)
+- Have leading or trailing whitespace
+- Equal `true`, `false`, or `null` (case-sensitive)
+- Look like numbers (e.g., `"42"`, `"-3.14"`, `"1e-6"`, `"05"`)
+- Contain special characters: `:`, `"`, `\`, `[`, `]`, `{`, `}`, newline, tab, carriage return
+- Contain the active delimiter (comma by default, or tab/pipe if declared in header)
+- Equal `"-"` or start with `"-"` followed by any character
+
+Otherwise, strings can be unquoted. Unicode and emoji are safe:
+
+```yaml
+message: Hello 世界 👋
+note: This has inner spaces
+```
+
+## Escape Sequences
+
+Only five escape sequences are valid in quoted strings:
+
+| Character | Escape |
+|-----------|--------|
+| Backslash (`\`) | `\\` |
+| Double quote (`"`) | `\"` |
+| Newline | `\n` |
+| Carriage return | `\r` |
+| Tab | `\t` |
+
+All other escapes (e.g., `\x`, `\u`) are invalid.
+
+## Array Headers
+
+### Basic Header
+
+```
+key[N]:
+```
+
+- `N` = array length
+- Default delimiter: comma
+
+### Tabular Header
+
+```
+key[N]{field1,field2,field3}:
+```
+
+- `N` = array length
+- `{fields}` = column names
+- Default delimiter: comma
+
+### Alternative Delimiters
+
+::: code-group
+
+```yaml [Tab Delimiter]
+items[2	]{id	name}:
+  1	Alice
+  2	Bob
+```
+
+```yaml [Pipe Delimiter]
+items[2|]{id|name}:
+  1|Alice
+  2|Bob
+```
+
+:::
+
+The delimiter symbol appears inside the brackets and braces.
+
+## Key Folding (Optional)
+
+Standard nesting:
+
+```yaml
+data:
+  metadata:
+    items[2]: a,b
+```
+
+With key folding (`keyFolding: 'safe'`):
+
+```yaml
+data.metadata.items[2]: a,b
+```
+
+See [Format Overview – Key Folding](/guide/format-overview#key-folding-optional) for details.
+
+## Type Conversions
+
+| Input | Output |
+|-------|--------|
+| Finite number | Canonical decimal (no exponent, no trailing zeros) |
+| `NaN`, `Infinity`, `-Infinity` | `null` |
+| `BigInt` (safe range) | Number |
+| `BigInt` (out of range) | Quoted decimal string |
+| `Date` | ISO string (quoted) |
+| `undefined`, `function`, `symbol` | `null` |