Plucene::Analysis::CharTokenizer3pm

Base class for character tokenisers

Usage

        # isa Plucene::Analysis::Tokenizer
        my $next = $chartokenizer->next;

This is an abstract base class for simple, character-oriented tokenizers.

This should be defined in subclasses.

This will normalise the character before it is added to the token.

        my $next = $chartokenizer->next;

This will return the next token in the string, or undef at the end of the string.

11 releases since Debian 3.1 (Sarge); the text changed 5 times.