Plucene::Analysis::Standard::StandardTokenizer3pm
Standard tokenizer
Usage
# isa Plucene::Analysis::CharTokenizerOverview
This is the standard tokenizer.
This should be a good tokenizer for most European-language documents.
Methods
token_re
The regular expression for tokenising.
normalize
Remove 's and .
Releases
11 releases since Debian 3.1 (Sarge); the text changed 5 times.