So the thing is, bytes are not actually a more "natural" encoding than tokens. I mean, jesus christ Unicode, right? A lot of what the BPE is doing is just un-doing the wacky arbitrary standards we use to encode characters into bytes, like, say, merging the 3-4 random bytes used to specify the codepoint of some CJK character.
There are so many possible naturalistic encodings. Waveforms, one-hot encoded discrete articulatory features, motor programs for simulated vocal apparatus, IPA, bitmaps of printed characters (handwritten? averaged across a bunch of fonts?), SVG programs to draw characters, decomposing big CJK chars via Ideographic Description Sequence, treating all unicode codepoints equally (kinda halfway between bytes and tokens)...
I don't think a thorough survey exists.
vatsachak
They are though. Bytes are raw data and the architecture chooses the tokens
augment_me
Its cool, but it goes against the whole optimization that NLP has gone through with BPE and redundancy/repetition in large data in general.
Put this on a coding task or English and you will underperform against tokenizer that exploit multi-char patterns and can decode much faster.
Given where things are going with spec decode (I've seen 32 token spec-decode pop up in the frontier models), you will need a very domain-specific use for this to be useful imo like massive tabular data, but at that point you already have everything structured, and might as well use an LLM with polars/sql/arrow.
yorwba
The whole point of this architecture is that it can also make use of multi-char patterns, but unlike BPE, they aren't hard-coded. And you can still use speculative decoding. Whether it works better or worse is something that you have to actually measure.
nivter
> you will need a very domain-specific use for this to be useful imo
The main premise of this idea is how general and widely applicable it is. They also show examples of how it works on images.
A common tokenizing paradigm would be a huge step towards consolidating how different modalities are preprocessed - something I believe is quite useful for intelligence to scale well across modalities.
vatsachak
Oh man, that haskell quicksort tokenization has got me feeling great
There are so many possible naturalistic encodings. Waveforms, one-hot encoded discrete articulatory features, motor programs for simulated vocal apparatus, IPA, bitmaps of printed characters (handwritten? averaged across a bunch of fonts?), SVG programs to draw characters, decomposing big CJK chars via Ideographic Description Sequence, treating all unicode codepoints equally (kinda halfway between bytes and tokens)...
I don't think a thorough survey exists.
Put this on a coding task or English and you will underperform against tokenizer that exploit multi-char patterns and can decode much faster.
Given where things are going with spec decode (I've seen 32 token spec-decode pop up in the frontier models), you will need a very domain-specific use for this to be useful imo like massive tabular data, but at that point you already have everything structured, and might as well use an LLM with polars/sql/arrow.
The main premise of this idea is how general and widely applicable it is. They also show examples of how it works on images.
A common tokenizing paradigm would be a huge step towards consolidating how different modalities are preprocessed - something I believe is quite useful for intelligence to scale well across modalities.