github/toon
1
0
Fork 0
mirror of https://github.com/voson-wang/toon.git synced 2026-01-29 15:24:10 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(cli): add --stats flag to show token savings (#51)

Browse Source 
* feat(cli): add --stats flag to show token efficiency

- Add --stats boolean flag to display token count comparison
- Calculate approximate tokens using char length / 4 heuristic
- Show JSON vs TOON token counts with savings percentage
- Opt-in feature, default behavior unchanged

* feat: use tokenx for more accurate estimates

---------

Co-authored-by: Johann Schopplich <mail@johannschopplich.com>
This commit is contained in:
SangheeSon
2025-11-01 08:35:54 +09:00
committed by GitHub
parent af068f995d
commit 2b882870f7
4 changed files with 35 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
README.md
View File

@@ -499,11 +499,15 @@ toon input.json
| `--delimiter <char>` | Array delimiter: `,` (comma), `\t` (tab), `\|` (pipe) |
| `--indent <number>` | Indentation size (default: `2`) |
| `--length-marker` | Add `#` prefix to array lengths (e.g., `items[#3]`) |
| `--stats` | Show token count estimates and savings (encode only) |
| `--no-strict` | Disable strict validation when decoding |
### Examples
```bash
# Show token savings when encoding
toon data.json --stats -o output.toon
# Tab-separated output (often more token-efficient)
toon data.json --delimiter "\t" -o output.toon