Skip to content

Latest commit

 

History

History

nlpo3-cli

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

SPDX-FileCopyrightText 2024-2026 PyThaiNLP Project
SPDX-License-Identifier Apache-2.0

nlpo3-cli

crates.io Apache-2.0

Command-line interface for nlpO3 Thai word tokenization.

Overview

  • Added tokenizer selection in CLI:
    • newmm (default)
    • nf (FST dictionary backend)
    • deepcut (neural tokenizer)
  • Improved error handling and user-facing messages.
  • Better support for long-input processing.

For implementation details, see ../docs/implementation.md.

Install

cargo install nlpo3-cli

Requirements

  • Rust Edition 2024
  • rustc 1.88.0 or newer

Quick start

nlpo3 --help
nlpo3 segment --help

echo "ฉันกินข้าว" | nlpo3 segment

Segment options

Option Type Default Description
--safe flag off Avoid long run times on inputs with many ambiguous word boundaries
--parallel [CHUNK_SIZE] optional integer 65536 (when flag is set without value) Enable chunked parallel processing; if no value is provided, uses 65536 bytes

Other useful segment flags:

  • --tokenizer (newmm, nf, deepcut)
  • --dict-path (for dictionary-based tokenizers)
  • --word-delimiter

Examples

Tokenize from standard input using the built-in dictionary:

echo "ฉันกินข้าว" | nlpo3 segment

Select a tokenizer:

# FST backend (less memory)
echo "ฉันกินข้าว" | nlpo3 segment --tokenizer nf

# Short form
echo "ฉันกินข้าว" | nlpo3 segment -t nf

# Neural CNN tokenizer (no dictionary needed)
echo "ฉันกินข้าว" | nlpo3 segment --tokenizer deepcut

Use a custom dictionary file:

echo "ฉันกินข้าว" | nlpo3 segment --dict-path /path/to/dict.txt
echo "ฉันกินข้าว" | nlpo3 segment -t nf --dict-path /path/to/dict.txt

Use a custom word delimiter:

echo "ฉันกินข้าว" | nlpo3 segment --word-delimiter " "

Enable safe mode (avoids long run times on ambiguous input):

echo "ฉันกินข้าว" | nlpo3 segment --safe

Enable parallel processing for larger text:

echo "ฉันกินข้าว" | nlpo3 segment -p
echo "ฉันกินข้าว" | nlpo3 segment -p 16384

Support

License

nlpo3-cli is copyrighted by its authors and licensed under terms of the Apache Software License 2.0 (Apache-2.0). See file LICENSE for details.