github/toon
1
0
Fork 0
mirror of https://github.com/voson-wang/toon.git synced 2026-01-29 23:34:10 +08:00
Code Issues Packages Projects Releases Wiki Activity

docs: overhaul API reference page

Browse Source
This commit is contained in:
Johann Schopplich
2025-11-23 19:47:37 +01:00
parent 5e8822bea8
commit 3389179979
1 changed files with 191 additions and 246 deletions
Download Patch File Download Diff File Expand all files Collapse all files

437
docs/reference/api.md
View File

@@ -20,7 +20,9 @@ yarn add @toon-format/toon
::: :::
## `encode(input, options?)` ## Encoding Functions
### `encode(input, options?)`
Converts any JSON-serializable value to TOON format. Converts any JSON-serializable value to TOON format.
@@ -35,27 +37,18 @@ const toon = encode(data, {
}) })
``` ```
### Parameters #### Parameters
| Parameter | Type | Description | | Parameter | Type | Description |
|-----------|------|-------------| |-----------|------|-------------|
| `input` | `unknown` | Any JSON-serializable value (object, array, primitive, or nested structure) | | `input` | `unknown` | Any JSON-serializable value (object, array, primitive, or nested structure) |
| `options` | `EncodeOptions?` | Optional encoding options (see below) | | `options` | `EncodeOptions?` | Optional encoding options (see [Configuration Reference](#configuration-reference)) |
### Options #### Return Value
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `indent` | `number` | `2` | Number of spaces per indentation level |
| `delimiter` | `','` \| `'\t'` \| `'\|'` | `','` | Delimiter for array values and tabular rows |
| `keyFolding` | `'off'` \| `'safe'` | `'off'` | Enable key folding to collapse single-key wrapper chains into dotted paths |
| `flattenDepth` | `number` | `Infinity` | Maximum number of segments to fold when `keyFolding` is enabled (values 0-1 have no practical effect) |
### Return Value
Returns a TOON-formatted string with no trailing newline or spaces. Returns a TOON-formatted string with no trailing newline or spaces.
### Type Normalization #### Type Normalization
Non-JSON-serializable values are normalized before encoding: Non-JSON-serializable values are normalized before encoding:
@@ -68,7 +61,7 @@ Non-JSON-serializable values are normalized before encoding:
| `Date` | ISO string in quotes (e.g., `"2025-01-01T00:00:00.000Z"`) | | `Date` | ISO string in quotes (e.g., `"2025-01-01T00:00:00.000Z"`) |
| `undefined`, `function`, `symbol` | `null` | | `undefined`, `function`, `symbol` | `null` |
### Example #### Example
```ts ```ts
import { encode } from '@toon-format/toon' import { encode } from '@toon-format/toon'
@@ -89,45 +82,7 @@ items[2]{sku,qty,price}:
B2,1,14.5 B2,1,14.5
``` ```
### Delimiter Options ### `encodeLines(input, options?)`
::: code-group
```ts [Comma (default)]
encode(data, { delimiter: ',' })
```
```ts [Tab]
encode(data, { delimiter: '\t' })
```
```ts [Pipe]
encode(data, { delimiter: '|' })
```
:::
::: details Why Use Tab Delimiters?
Tab delimiters (`\t`) often tokenize more efficiently than commas:
- Tabs are single characters
- Tabs rarely appear in natural text, reducing quote-escaping
- The delimiter is explicitly encoded in the array header
Example:
```yaml
items[2 ]{sku name qty price}:
A1 Widget 2 9.99
B2 Gadget 1 14.5
```
For maximum token savings on large tabular data, combine tab delimiters with key folding:
```ts
encode(data, { delimiter: '\t', keyFolding: 'safe' })
```
:::
## `encodeLines(input, options?)`
**Preferred method for streaming TOON output.** Converts any JSON-serializable value to TOON format as a sequence of lines, without building the full string in memory. Suitable for streaming large outputs to files, HTTP responses, or process stdout. **Preferred method for streaming TOON output.** Converts any JSON-serializable value to TOON format as a sequence of lines, without building the full string in memory. Suitable for streaming large outputs to files, HTTP responses, or process stdout.
@@ -149,14 +104,14 @@ for (const line of lines) {
const lineArray = Array.from(encodeLines(data)) const lineArray = Array.from(encodeLines(data))
``` ```
### Parameters #### Parameters
| Parameter | Type | Description | | Parameter | Type | Description |
|-----------|------|-------------| |-----------|------|-------------|
| `input` | `unknown` | Any JSON-serializable value (object, array, primitive, or nested structure) | | `input` | `unknown` | Any JSON-serializable value (object, array, primitive, or nested structure) |
| `options` | `EncodeOptions?` | Optional encoding options (same as `encode()`) | | `options` | `EncodeOptions?` | Optional encoding options (see [Configuration Reference](#configuration-reference)) |
### Return Value #### Return Value
Returns an `Iterable<string>` that yields TOON lines one at a time. **Each yielded string is a single line without a trailing newline character** — you must add `\n` when writing to streams or stdout. Returns an `Iterable<string>` that yields TOON lines one at a time. **Each yielded string is a single line without a trailing newline character** — you must add `\n` when writing to streams or stdout.
@@ -167,7 +122,7 @@ Array.from(encodeLines(value, options)).join('\n')
``` ```
::: :::
### Example #### Example
```ts ```ts
import { createWriteStream } from 'node:fs' import { createWriteStream } from 'node:fs'
@@ -189,22 +144,9 @@ for (const line of encodeLines(data, { delimiter: '\t' })) {
stream.end() stream.end()
``` ```
## Choosing a Decode Function ## Decoding Functions
| Function | Input | Output | Async | Path Expansion | Use When | ### `decode(input, options?)`
|----------|-------|--------|-------|----------------|----------|
| `decode()` | String | Value | No | Yes | You have a complete TOON string |
| `decodeFromLines()` | Lines | Value | No | Yes | You have lines and want the full value |
| `decodeStreamSync()` | Lines | Events | No | No | You need event-by-event processing (sync) |
| `decodeStream()` | Lines | Events | Yes | No | You need event-by-event processing (async) |
::: info Key Differences
- **Value vs. Events**: Functions ending in `Stream` yield events without building the full value in memory.
- **Path expansion**: Only `decode()` and `decodeFromLines()` support `expandPaths: 'safe'`.
- **Async support**: Only `decodeStream()` accepts async iterables (useful for file/network streams).
:::
## `decode(input, options?)`
Converts a TOON-formatted string back to JavaScript values. Converts a TOON-formatted string back to JavaScript values.
@@ -218,34 +160,18 @@ const data = decode(toon, {
}) })
``` ```
### Parameters #### Parameters
| Parameter | Type | Description | | Parameter | Type | Description |
|-----------|------|-------------| |-----------|------|-------------|
| `input` | `string` | A TOON-formatted string to parse | | `input` | `string` | A TOON-formatted string to parse |
| `options` | `DecodeOptions?` | Optional decoding options | | `options` | `DecodeOptions?` | Optional decoding options (see [Configuration Reference](#configuration-reference)) |
### Options #### Return Value
See [Decode Options Reference](#decode-options-reference) below.
### Return Value
Returns a JavaScript value (object, array, or primitive) representing the parsed TOON data. Returns a JavaScript value (object, array, or primitive) representing the parsed TOON data.
### Strict Mode #### Example
By default (`strict: true`), the decoder validates input strictly:
- **Invalid escape sequences**: Throws on `\x`, unterminated strings
- **Syntax errors**: Throws on missing colons, malformed headers
- **Array length mismatches**: Throws when declared length doesn't match actual count
- **Delimiter mismatches**: Throws when row delimiters don't match header
- **Indentation errors**: Throws when leading spaces aren't exact multiples of `indentSize`
Set `strict: false` to skip validation for lenient parsing.
### Example
```ts ```ts
import { decode } from '@toon-format/toon' import { decode } from '@toon-format/toon'
@@ -271,68 +197,24 @@ console.log(data)
} }
``` ```
### Path Expansion ### `decodeFromLines(lines, options?)`
When `expandPaths: 'safe'` is enabled, dotted keys are split into nested objects:
```ts
import { decode } from '@toon-format/toon'
const toon = 'data.metadata.items[2]: a,b'
const data = decode(toon, { expandPaths: 'safe' })
console.log(data)
// { data: { metadata: { items: ['a', 'b'] } } }
```
This pairs with `keyFolding: 'safe'` for lossless round-trips.
::: details Expansion Conflict Resolution
When multiple expanded keys construct overlapping paths, the decoder merges them recursively:
- **Object + Object**: Deep merge recursively
- **Object + Non-object** (array or primitive): Conflict
- With `strict: true` (default): Error
- With `strict: false`: Last-write-wins (LWW)
Example conflict (strict mode):
```ts
const toon = 'a.b: 1\na: 2'
decode(toon, { expandPaths: 'safe', strict: true })
// Error: "Expansion conflict at path 'a' (object vs primitive)"
```
Example conflict (lenient mode):
```ts
const toon = 'a.b: 1\na: 2'
decode(toon, { expandPaths: 'safe', strict: false })
// { a: 2 } (last write wins)
```
:::
## `decodeFromLines(lines, options?)`
Decodes TOON format from pre-split lines into a JavaScript value. This is a streaming-friendly wrapper around the event-based decoder that builds the full value in memory. Decodes TOON format from pre-split lines into a JavaScript value. This is a streaming-friendly wrapper around the event-based decoder that builds the full value in memory.
Useful when you already have lines as an array or iterable (e.g., from file streams, readline interfaces, or network responses) and want the standard decode behavior with path expansion support. Useful when you already have lines as an array or iterable (e.g., from file streams, readline interfaces, or network responses) and want the standard decode behavior with path expansion support.
### Parameters #### Parameters
| Parameter | Type | Description | | Parameter | Type | Description |
|-----------|------|-------------| |-----------|------|-------------|
| `lines` | `Iterable<string>` | Iterable of TOON lines (without trailing newlines) | | `lines` | `Iterable<string>` | Iterable of TOON lines (without trailing newlines) |
| `options` | `DecodeOptions?` | Optional decoding configuration | | `options` | `DecodeOptions?` | Optional decoding configuration (see [Configuration Reference](#configuration-reference)) |
### Options #### Return Value
See [Decode Options Reference](#decode-options-reference) below.
### Return Value
Returns a `JsonValue` (the parsed JavaScript value: object, array, or primitive). Returns a `JsonValue` (the parsed JavaScript value: object, array, or primitive).
### Example #### Example
**Basic usage with arrays:** **Basic usage with arrays:**
@@ -368,54 +250,47 @@ const value = decodeFromLines(lines, { expandPaths: 'safe' })
// { user: { name: 'Alice', age: 30 } } // { user: { name: 'Alice', age: 30 } }
``` ```
## `decodeStreamSync(lines, options?)` ### Choosing the Right Decoder
| Function | Input | Output | Async | Path Expansion | Use When |
|----------|-------|--------|-------|----------------|----------|
| `decode()` | String | Value | No | Yes | You have a complete TOON string |
| `decodeFromLines()` | Lines | Value | No | Yes | You have lines and want the full value |
| `decodeStreamSync()` | Lines | Events | No | No | You need event-by-event processing (sync) |
| `decodeStream()` | Lines | Events | Yes | No | You need event-by-event processing (async) |
::: info Key Differences
- **Value vs. Events**: Functions ending in `Stream` yield events without building the full value in memory.
- **Path expansion**: Only `decode()` and `decodeFromLines()` support `expandPaths: 'safe'`.
- **Async support**: Only `decodeStream()` accepts async iterables (useful for file/network streams).
:::
## Streaming Decoders
### `decodeStreamSync(lines, options?)`
Synchronously decodes TOON lines into a stream of JSON events. This function yields structured events that represent the JSON data model without building the full value tree. Synchronously decodes TOON lines into a stream of JSON events. This function yields structured events that represent the JSON data model without building the full value tree.
Useful for streaming processing, custom transformations, or memory-efficient parsing of large datasets where you don't need the full value in memory. Useful for streaming processing, custom transformations, or memory-efficient parsing of large datasets where you don't need the full value in memory.
::: info Event Streaming ::: tip Event Streaming
This is a low-level API that returns individual parse events. For most use cases, [`decodeFromLines()`](#decodeFromLines-lines-options) or [`decode()`](#decode-input-options) are more convenient. This is a low-level API that returns individual parse events. For most use cases, [`decodeFromLines()`](#decodefromlines-lines-options) or [`decode()`](#decode-input-options) are more convenient.
Path expansion (`expandPaths: 'safe'`) is **not supported** in streaming mode since it requires the full value tree. Path expansion (`expandPaths: 'safe'`) is **not supported** in streaming mode since it requires the full value tree.
::: :::
### Parameters #### Parameters
| Parameter | Type | Description | | Parameter | Type | Description |
|-----------|------|-------------| |-----------|------|-------------|
| `lines` | `Iterable<string>` | Iterable of TOON lines (without trailing newlines) | | `lines` | `Iterable<string>` | Iterable of TOON lines (without trailing newlines) |
| `options` | `DecodeStreamOptions?` | Optional streaming decoding configuration (see below) | | `options` | `DecodeStreamOptions?` | Optional streaming decoding configuration (see [Configuration Reference](#configuration-reference)) |
### Options #### Return Value
See [Decode Options Reference](#decode-options-reference) below. Returns an `Iterable<JsonStreamEvent>` that yields structured events (see [TypeScript Types](#typescript-types) for event structure).
::: warning Path Expansion Not Supported #### Example
Path expansion requires building the full value tree, which is incompatible with event streaming. Use [`decodeFromLines()`](#decodeFromLines-lines-options) if you need path expansion.
:::
### Return Value
Returns an `Iterable<JsonStreamEvent>` that yields structured events.
### Event Types
Events represent the structure of the JSON data model:
```ts
type JsonStreamEvent
= | { type: 'startObject' }
| { type: 'endObject' }
| { type: 'startArray', length: number }
| { type: 'endArray' }
| { type: 'key', key: string, wasQuoted?: boolean }
| { type: 'primitive', value: JsonPrimitive }
type JsonPrimitive = string | number | boolean | null
```
### Example
**Basic event streaming:** **Basic event streaming:**
@@ -453,32 +328,24 @@ for (const event of decodeStreamSync(lines)) {
} }
``` ```
## `decodeStream(source, options?)` ### `decodeStream(source, options?)`
Asynchronously decodes TOON lines into a stream of JSON events. This is the async version of [`decodeStreamSync()`](#decodeStreamSync-lines-options), supporting both synchronous and asynchronous iterables. Asynchronously decodes TOON lines into a stream of JSON events. This is the async version of [`decodeStreamSync()`](#decodestreamsync-lines-options), supporting both synchronous and asynchronous iterables.
Useful for processing file streams, network responses, or other async sources where you want to handle data incrementally as it arrives. Useful for processing file streams, network responses, or other async sources where you want to handle data incrementally as it arrives.
### Parameters #### Parameters
| Parameter | Type | Description | | Parameter | Type | Description |
|-----------|------|-------------| |-----------|------|-------------|
| `source` | `AsyncIterable<string>` \| `Iterable<string>` | Async or sync iterable of TOON lines (without trailing newlines) | | `source` | `AsyncIterable<string>` \| `Iterable<string>` | Async or sync iterable of TOON lines (without trailing newlines) |
| `options` | `DecodeStreamOptions?` | Optional streaming decoding configuration (see below) | | `options` | `DecodeStreamOptions?` | Optional streaming decoding configuration (see [Configuration Reference](#configuration-reference)) |
### Options #### Return Value
See [Decode Options Reference](#decode-options-reference) below. Returns an `AsyncIterable<JsonStreamEvent>` that yields structured events asynchronously (see [TypeScript Types](#typescript-types) for event structure).
::: warning Path Expansion Not Supported #### Example
Path expansion requires building the full value tree, which is incompatible with event streaming. Use [`decodeFromLines()`](#decodeFromLines-lines-options) if you need path expansion.
:::
### Return Value
Returns an `AsyncIterable<JsonStreamEvent>` that yields structured events asynchronously.
### Example
**Streaming from file:** **Streaming from file:**
@@ -496,43 +363,95 @@ for await (const event of decodeStream(rl)) {
} }
``` ```
**Processing events with state tracking:** ## Configuration Reference
```ts ### `EncodeOptions`
import { decodeStream } from '@toon-format/toon'
const lines = getAsyncLineSource() // AsyncIterable<string> Configuration for [`encode()`](#encode-input-options) and [`encodeLines()`](#encodelines-input-options):
// Track state between events | Option | Type | Default | Description |
let nextIsId = false |--------|------|---------|-------------|
for await (const event of decodeStream(lines, { strict: true })) { | `indent` | `number` | `2` | Number of spaces per indentation level |
if (event.type === 'key' && event.key === 'id') { | `delimiter` | `','` \| `'\t'` \| `'\|'` | `','` | Delimiter for array values and tabular rows |
nextIsId = true | `keyFolding` | `'off'` \| `'safe'` | `'off'` | Enable key folding to collapse single-key wrapper chains into dotted paths |
} | `flattenDepth` | `number` | `Infinity` | Maximum number of segments to fold when `keyFolding` is enabled (values 0-1 have no practical effect) |
else if (nextIsId && event.type === 'primitive') {
console.log('Found ID:', event.value) **Delimiter options:**
nextIsId = false
} ::: code-group
}
```ts [Comma (default)]
encode(data, { delimiter: ',' })
``` ```
**Auto-detection of sync/async sources:** ```ts [Tab]
encode(data, { delimiter: '\t' })
```ts
// Works with sync iterables
const syncLines = ['name: Alice', 'age: 30']
for await (const event of decodeStream(syncLines)) {
console.log(event)
}
// Works with async iterables
const asyncLines = readLinesFromNetwork()
for await (const event of decodeStream(asyncLines)) {
console.log(event)
}
``` ```
## Round-Trip Compatibility ```ts [Pipe]
encode(data, { delimiter: '|' })
```
:::
See [Delimiter Strategies](#delimiter-strategies) for guidance on choosing delimiters.
### `DecodeOptions`
Configuration for [`decode()`](#decode-input-options) and [`decodeFromLines()`](#decodefromlines-lines-options):
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `indent` | `number` | `2` | Expected number of spaces per indentation level |
| `strict` | `boolean` | `true` | Enable strict validation (array counts, indentation, delimiter consistency) |
| `expandPaths` | `'off'` \| `'safe'` | `'off'` | Enable path expansion to reconstruct dotted keys into nested objects (pairs with `keyFolding: 'safe'`) |
By default (`strict: true`), the decoder validates input strictly:
- **Invalid escape sequences**: Throws on `\x`, unterminated strings
- **Syntax errors**: Throws on missing colons, malformed headers
- **Array length mismatches**: Throws when declared length doesn't match actual count
- **Delimiter mismatches**: Throws when row delimiters don't match header
- **Indentation errors**: Throws when leading spaces aren't exact multiples of `indent`
Set `strict: false` to skip validation for lenient parsing.
See [Key Folding & Path Expansion](#key-folding-path-expansion) for more details on path expansion behavior and conflict resolution.
### `DecodeStreamOptions`
Configuration for [`decodeStreamSync()`](#decodestreamsync-lines-options) and [`decodeStream()`](#decodestream-source-options):
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `indent` | `number` | `2` | Expected number of spaces per indentation level |
| `strict` | `boolean` | `true` | Enable strict validation (array counts, indentation, delimiter consistency) |
::: warning Path Expansion Not Supported
Path expansion requires building the full value tree, which is incompatible with event streaming. Use [`decodeFromLines()`](#decodefromlines-lines-options) if you need path expansion.
:::
## TypeScript Types
### `JsonStreamEvent`
Events emitted by [`decodeStreamSync()`](#decodestreamsync-lines-options) and [`decodeStream()`](#decodestream-source-options):
```ts
type JsonStreamEvent
= | { type: 'startObject' }
| { type: 'endObject' }
| { type: 'startArray', length: number }
| { type: 'endArray' }
| { type: 'key', key: string, wasQuoted?: boolean }
| { type: 'primitive', value: JsonPrimitive }
type JsonPrimitive = string | number | boolean | null
```
## Guides & Examples
### Round-Trip Compatibility
TOON provides lossless round-trips after normalization: TOON provides lossless round-trips after normalization:
@@ -553,7 +472,7 @@ console.log(JSON.stringify(original) === JSON.stringify(restored))
// true // true
``` ```
### With Key Folding **With Key Folding:**
```ts ```ts
import { decode, encode } from '@toon-format/toon' import { decode, encode } from '@toon-format/toon'
@@ -572,40 +491,66 @@ console.log(JSON.stringify(original) === JSON.stringify(restored))
// true // true
``` ```
## Types ### Key Folding & Path Expansion
**Key Folding** (`keyFolding: 'safe'`) collapses single-key wrapper chains during encoding:
```ts ```ts
interface EncodeOptions { import { encode } from '@toon-format/toon'
indent?: number
delimiter?: ',' | '\t' | '|'
keyFolding?: 'off' | 'safe'
flattenDepth?: number
}
interface DecodeOptions { const data = { data: { metadata: { items: ['a', 'b'] } } }
indent?: number
strict?: boolean // Without folding
expandPaths?: 'off' | 'safe' encode(data)
} // data:
// metadata:
// items[2]: a,b
// With folding
encode(data, { keyFolding: 'safe' })
// data.metadata.items[2]: a,b
``` ```
## Decode Options Reference **Path Expansion** (`expandPaths: 'safe'`) reverses this during decoding:
### `DecodeOptions` ```ts
import { decode } from '@toon-format/toon'
Used by `decode()` and `decodeFromLines()`: const toon = 'data.metadata.items[2]: a,b'
| Option | Type | Default | Description | const data = decode(toon, { expandPaths: 'safe' })
|--------|------|---------|-------------| console.log(data)
| `indent` | `number` | `2` | Expected number of spaces per indentation level | // { data: { metadata: { items: ['a', 'b'] } } }
| `strict` | `boolean` | `true` | Enable strict validation (array counts, indentation, delimiter consistency) | ```
| `expandPaths` | `'off'` \| `'safe'` | `'off'` | Enable path expansion to reconstruct dotted keys into nested objects (pairs with `keyFolding: 'safe'`) |
### `DecodeStreamOptions` **Expansion Conflict Resolution:**
Used by `decodeStreamSync()` and `decodeStream()`: When multiple expanded keys construct overlapping paths, the decoder merges them recursively:
- **Object + Object**: Deep merge recursively
- **Object + Non-object** (array or primitive): Conflict
- With `strict: true` (default): Error
- With `strict: false`: Last-write-wins (LWW)
| Option | Type | Default | Description | ### Delimiter Strategies
|--------|------|---------|-------------|
| `indent` | `number` | `2` | Expected number of spaces per indentation level | Tab delimiters (`\t`) often tokenize more efficiently than commas, as Tabs are single characters that rarely appear in natural text. This reduces the need for quote-escaping, leading to smaller token counts in large datasets.
| `strict` | `boolean` | `true` | Enable strict validation (array counts, indentation, delimiter consistency) |
Example:
```yaml
items[2 ]{sku name qty price}:
A1 Widget 2 9.99
B2 Gadget 1 14.5
```
For maximum token savings on large tabular data, combine tab delimiters with key folding:
```ts
encode(data, { delimiter: '\t', keyFolding: 'safe' })
```
**Choosing a Delimiter:**
- **Comma (`,`)**: Default, widely understood, good for simple tabular data.
- **Tab (`\t`)**: Best for LLM token efficiency, excellent for large datasets.
- **Pipe (`|`)**: Alternative when commas appear frequently in data.