| SPDX-FileCopyrightText | 2024-2026 PyThaiNLP Project |
|---|---|
| SPDX-License-Identifier | Apache-2.0 |
Command-line interface for nlpO3 Thai word tokenization.
- Added tokenizer selection in CLI:
newmm(default)nf(FST dictionary backend)deepcut(neural tokenizer)
- Improved error handling and user-facing messages.
- Better support for long-input processing.
For implementation details, see ../docs/implementation.md.
cargo install nlpo3-cli- Rust Edition 2024
- rustc 1.88.0 or newer
nlpo3 --help
nlpo3 segment --help
echo "ฉันกินข้าว" | nlpo3 segment| Option | Type | Default | Description |
|---|---|---|---|
--safe |
flag | off | Avoid long run times on inputs with many ambiguous word boundaries |
--parallel [CHUNK_SIZE] |
optional integer | 65536 (when flag is set without value) |
Enable chunked parallel processing; if no value is provided, uses 65536 bytes |
Other useful segment flags:
--tokenizer(newmm,nf,deepcut)--dict-path(for dictionary-based tokenizers)--word-delimiter
Tokenize from standard input using the built-in dictionary:
echo "ฉันกินข้าว" | nlpo3 segmentSelect a tokenizer:
# FST backend (less memory)
echo "ฉันกินข้าว" | nlpo3 segment --tokenizer nf
# Short form
echo "ฉันกินข้าว" | nlpo3 segment -t nf
# Neural CNN tokenizer (no dictionary needed)
echo "ฉันกินข้าว" | nlpo3 segment --tokenizer deepcutUse a custom dictionary file:
echo "ฉันกินข้าว" | nlpo3 segment --dict-path /path/to/dict.txt
echo "ฉันกินข้าว" | nlpo3 segment -t nf --dict-path /path/to/dict.txtUse a custom word delimiter:
echo "ฉันกินข้าว" | nlpo3 segment --word-delimiter " "Enable safe mode (avoids long run times on ambiguous input):
echo "ฉันกินข้าว" | nlpo3 segment --safeEnable parallel processing for larger text:
echo "ฉันกินข้าว" | nlpo3 segment -p
echo "ฉันกินข้าว" | nlpo3 segment -p 16384nlpo3-cli is copyrighted by its authors and licensed under terms of the Apache Software License 2.0 (Apache-2.0). See file LICENSE for details.