mfltok · open source · MIT
Every time an AI assistant reads a file, you pay by the token. Most tools that count them are estimating. This one measures — and it is one static binary, with no Python and no runtime.
Verified against the reference implementation
166,350 checks · 0 mismatches
Hand-picked edge cases, 20,000 seeded random inputs per encoding, every whitespace
string up to four characters exhaustively, and an exhaustive Unicode letter-class
corpus. Byte-identical to tiktoken on all of them.
The usual shortcut is characters ÷ 4. We measured our own estimator against the real thing: it undercounted by 18.6 % — and, worse, unevenly. Dense files were off by 25 %, simple ones by 6 %. So the list of your biggest files came out in the wrong order. The ranking was wrong, not just the totals.
445 kB of source, identical output — 214,650 tokens both sides:
| Start → answer | Encode | Memory | |
|---|---|---|---|
| mfltok | 0.08 s | 57 ms | 56 MB |
| Python + tiktoken (Rust core) | 0.37 s | 87 ms | 68 MB |
Written in machin (MFL), which had no tokenizer library and no package registry — so the Unicode tables, the pretokenizer and the merge loop are all from scratch. Building the repo scanner pushed four new builtins into the language itself.
Once you can measure honestly, most common advice stops surviving the numbers:
| Suggested fix | Effect |
|---|---|
| Split a big file into smaller ones | −0.03 % |
| Reflow long lines | +0.18 % |
| Don't load the file at all | −100 % |
You cannot reformat your way to a smaller bill. Rearranging the same words costs the same. The only lever is deciding what the assistant never needs to see.
git clone https://github.com/javimosch/mfltok cd mfltok && ./setup.sh && ./build.sh
mfltok scan . --human # a report you read mfltok scan . # JSON, for an agent mfltok count FILE # one file
Honours .gitignore, skips binaries by content rather than by filename,
and counts every exclusion — a scanner that quietly drops files
while implying it measured everything is the thing this was built to avoid.
It never claims to have saved you anything. It is a measuring instrument: it does not sit between you and the assistant, and it cannot lower a bill by itself.
A well-known tool in this space advertised “96.2 million tokens saved” while a controlled experiment found its users' bills had gone up 7.6 %. Any tool that grades its own homework can produce a number like that — so this one publishes no savings figure at all.