hfst-tokenize1
=perform matching/lookup on text streams
Usage
hfst-tokenize [{--segment | --xerox | --cg | --giella-cg}]
[OPTIONS ...]
RULESET
Overview
perform matching/lookup on text streams
Common options:
-
-h, --help -
Print help message
-
-V, --version -
Print version info
-
-v, --verbose -
Print verbosely while processing
-
-q, --quiet -
Only print fatal erros and requested output
-
-s, --silent -
Alias of
--quiet
-
-n, --newline -
Newline as input separator (default is blank line)
-
-a, --print-all -
Print nonmatching text
-
-w, --print-weight -
Print weights (overrides earlier
-Woption)
-
-W, --no-weights -
Don't print weights (default; overrides earlier
-w, or-wimplied by-g, options)
-m,--tokenize-multicharTokenize multicharacter symbols-
(by default only one utf-8 character is tokenized at a time regardless of what is present in the alphabet)
-
-b, --beam=B -
Output only analyses whose weight is within B from best result
-
-tS, --time-cutoff=S -
Limit search after having used S seconds per input
-
-lN, --weight-classes=N -
Output no more than N best weight classes (where analyses with equal weight constitute a class
-
-u, --unique -
Remove duplicate analyses
-
-z, --segment -
Segmenting / tokenization mode (default)
-
-i, --space-separated -
Tokenization with one sentence per line, space-separated tokens
-
-x, --xerox -
Xerox output
-
-c, --cg -
Constraint Grammar output
-
-S, --superblanks -
Ignore contents of unescaped [] (cf. apertium-destxt); flush on NUL
-
-g, --giella-cg -
CG format used in Giella infrastructure (implies
-wand-l2, treats @PMATCH_INPUT_MARK@ as subreading separator, expects tags to be Multichar_symbols, flush on NUL)
-
-C --conllu -
CoNLL-U format
-
-f, --finnpos -
FinnPos output
-
-L, --visl -
VISL input and output (implies
-W, handles <s> as blocks and <STYLE> inline)
Use standard streams for input and output (for now).
Releases
5 releases since Debian 9 (Stretch); the text changed 2 times.
--beam since Debian 10 (Buster);
--conllu since Debian 10 (Buster);
--giella-cg since Debian 10 (Buster);
--no-weights since Debian 10 (Buster);
--space-separated since Debian 10 (Buster);
--superblanks since Debian 10 (Buster);
--unique since Debian 10 (Buster);
--visl since Debian 10 (Buster);
--weight-classes since Debian 10 (Buster);
-C since Debian 10 (Buster);
-L since Debian 10 (Buster);
-S since Debian 10 (Buster);
-W since Debian 10 (Buster);
-b since Debian 10 (Buster);
-c since Debian 10 (Buster);
-f since Debian 10 (Buster);
-g since Debian 10 (Buster);
-i since Debian 10 (Buster);
-lN since Debian 10 (Buster);
-tS since Debian 10 (Buster);
-u since Debian 10 (Buster);
-x since Debian 10 (Buster);
-z since Debian 10 (Buster).
--tokenize-multichar until Debian 9 (Stretch);
-t until Debian 9 (Stretch).