SOURCE-LINKED INTELLIGENCE

Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining

arXiv · AI, language, vision and robotics · article · Sep 18, 2026 · UTC

Conventional tokenizers represent text as characters or statistically derived subwords, overlooking the internal phonological structure of syllables and often requiring large vocabularies. We introduce \textbf{Phonemic Tokenizer}, a linguistically motivated tokenizer for Vietnamese and Chinese that converts each syllable into IPA and factorizes it into three phonological components: onset, rime, and tone. The three components jointly occupy one contextual position, preserving syllable-level sequence length while enabling representation sharing across phonologically related syllables. Non-phono

Read original source ↗ Open in workspace

recordType

paper

region

Global