xtokid1

Print all tokens found in a source file

Usage

xtokid [OPTION] ... [FILE] ...

Overview

Print all tokens found in a source file.

-i, --include=LANGS: include languages in LANGS (default: "C C++ asm")

-x, --exclude=LANGS: exclude languages in LANGS

-l, --lang-option=L:OPT pass OPT as a default for language L (see below)

-m, --lang-map=MAPFILE: use MAPFILE to map file names onto source language

-d, --default-lang=LANG: make LANG the default source language

-p, --prune=NAMES: exclude the named files and/or directories

--files0-from=F: tokenize only the files specified by NUL-terminated names in file F

--help: display this help and exit --version output version information and exit

The following arguments apply to the language-specific scanners:

C language:

-k,--keep=CHARS: Allow CHARS in single-token strings, keep the result

-i,--ignore=CHARS: Allow CHARS in single-token strings, toss the result

-u,--strip-underscore: Strip a leading underscore from single-token strings

C++ language:

-k,--keep=CHARS: Allow CHARS in single-token strings, keep the result

-i,--ignore=CHARS: Allow CHARS in single-token strings, toss the result

-u,--strip-underscore: Strip a leading underscore from single-token strings

Java language:

-k,--keep=CHARS: Allow CHARS in single-token strings, keep the result

-i,--ignore=CHARS: Allow CHARS in single-token strings, toss the result

-u,--strip-underscore: Strip a leading underscore from single-token strings

Assembly language:

-c,--comment=CHARS: Any of CHARS starts a comment until end-of-line

-k,--keep=CHARS: Allow CHARS in tokens, and keep the result

-i,--ignore=CHARS: Allow CHARS in tokens, and toss the result

-u,--strip-underscore: Strip a leading underscore from tokens

-n,--no-cpp: Don't handle C pre-processor directives

Text language:

-i,--include=CHAR-CLASS: Treat characters of CHAR-CLASS as token constituents

-x,--exclude=CHAR-CLASS: Treat characters of CHAR-CLASS as token delimiters

Perl language:

-i,--include=CHAR-CLASS: Treat characters of CHAR-CLASS as token constituents

-x,--exclude=CHAR-CLASS: Treat characters of CHAR-CLASS as token delimiters

-d,--dtags: Include documentation tags

Lisp language:

Releases

10 releases since Debian 2.2 (Potato); the text changed 7 times.

--default-lang since Debian 7 (Wheezy); --dtags since Debian 5.0 (Lenny); --files0-from since Debian 7 (Wheezy); --lang-map since Debian 4.0 (Etch); --prune since Debian 4.0 (Etch); -d since Debian 5.0 (Lenny); -m since Debian 4.0 (Etch); -p since Debian 4.0 (Etch).

--version until Debian 6.0 (Squeeze).