Plucene::Analysis::Standard::StandardTokenizer (3pm)

Standard tokenizer

  1. Carta.tech
  2. Man Pages
  3. MISSING SECTION
  4. Plucene::Analysis::Standard::StandardTokenizer: Standard tokenizer
  1. Carta.tech
  2. Packages
  3. libplucene-perl
  4. Plucene::Analysis::Standard::StandardTokenizer: Standard tokenizer

SYNOPSIS

        # isa Plucene::Analysis::CharTokenizer

DESCRIPTION

This is the standard tokenizer.

This should be a good tokenizer for most European-language documents.

METHODS

token_re

The regular expression for tokenising.

normalize

Remove 's and .