xtract1

Ncbi entrez direct xml conversion and transformation tool

Usage

xtract [-help] [-strict] [-mixed] [-self] [-accent] [-ascii] [-compress] [-stops] [-input filename] [-transform filename] [-aliases filename] [-pattern expr] [-group expr] [-block expr] [-subset expr] [-path path] [-if expr [constraint]] [-unless expr [constraint]] [-and condition] [-or condition] [-else] [-position pos] [-equals str] [-contains str] [-mimics str] [-excludes str] [-includes str] [-is-within str] [-starts-with str] [-ends-with str] [-is-not str] [-is-before str] [-is-after str] [-consists-of str] [-matches str] [-resembles str] [-is-equal-to expr] [-differs-from expr] [-gt N] [-ge N] [-lt N] [-le N] [-eq N] [-ne N] [-ret str] [-tab str] [-sep str] [-pfx str] [-sfx str] [-rst] [-clr] [-pfc str] [-deq str] [-def str] [-lbl str] [-set tag] [-rec tag] [-wrp tag] [-enc tag] [-plg str] [-elg str] [-pkg tag] [-fwd str] [-awd str] [-tag tag] [-att key str] [-atr key element] [-cls] [-slf] [-end tag] [-bkt] [-element element] [-first element] [-last element] [-first element] [-last element] [-backward element] [-NAME] [--STATS] [-num element] [-len element] [-sum element] [-acc element] [-min element] [-max element] [-inc element] [-dec element] [-sub element] [-avg element] [-dev element] [-med element] [-mul element] [-div element] [-mod element] [-geo element] [-hrm element] [-rms element] [-sqt element] [-lge element] [-lg2 element] [-log element] [-bin element] [-oct element] [-hex element] [-bit element] [-pad element] [-encode element] [-decode element] [-upper element] [-lower element] [-chain element] [-title element] [-mirror element] [-alpha element] [-alnum element] [-basic element] [-plain element] [-simple element] [-author element] [-journal element] [-prose element] [-terms element] [-words element] [-pairs element] [-split element -with str] [-order element] [-reverse element] [-letters element] [-clauses element] [-pentamers element] [-year element] [-month element] [-date element] [-page element] [-auth element] [-initials element] [-trim element] [-wct element] [-doi element] [-accession element] [-numeric element] [-translate element] [-classify element] [-replace -reg target -exp replacement] [-fasta] [-revcomp] [-nucleic] [-ncbi2na] [-ncbi4na] [-cds2prot [-gcode N] [-frame N]] [-molwt] [-molwt-m] [-molwt-f] [-pept] [-0-based element] [-1-based element] [-ucsc-based element] [-insd arg ...] [-insdx] [-histogram] [-indexer element] [-head str] [-tail str] [-hd str] [-tl str] [-select condition] [-in filename] [-sort [-fwd] element] [-sort-rev element] [-format fmt [-unicode style]] [-verify] [-test] [-outline] [-synopsis] [-contour [delimiter]] [-examples] [-unix] [-version]

Overview

xtract converts an XML document into a table of data values according to user-specified rules.

Options

Processing Flags

-strict: Remove HTML and MathML tags.

-mixed: Allow mixed content XML.

-self: Allow detection of empty self-closing tags.

-accent: Delete Unicode accents and diacritical marks.

-ascii: Convert Unicode to numeric HTML character entities.

-compress: Compress runs of spaces.

-stops: Retain stop words in selected phrases.

Data Source

-input filename: Read XML from file instead of standard input.

-transform filename: File of substitutions for -translate.

-aliases filename: Mappings file for -classify operation.

Exploration Argument Hierarchy

-pattern expr

-group expr

-block expr

-subset expr: Name of record within set. Use of different argument names allows command-line control of nested looping.

Path Navigation

-path path: Explore by list of adjacent object names.

Exploration Constructs

`Object`	`DateRevised`
`Parent/Child`	`Book/AuthorList`
`Path`	`MedlineCitation/Article/Journal/JournalIssue/PubDate`
`Heterogeneous`	`"PubmedArticleSet/*"`
`Exhaustive`	`"History/**"`
`Nested`	`"*/Taxon"`

Conditional Execution

`-if expr [constraint]`	Element (or @attribute) must exist and satisfy any specified constraint.
`-unless expr [constraint]`	Skip if element matches.

-and condition: Preceding and following tests must both pass.

-or condition: Any passing test suffices.

-else: Execute if conditional test failed.

-position pos: first/last/outer/inner/even/odd/all.

String Constraints

-equals str: String must match exactly.

-contains str: Substring must be present.

-mimics str: Containment test after converting punctuation to space.

-excludes str: Substring must be absent.

-includes str: Substring must match at word boundaries.

-is-within str: String must be present.

-starts-with str: Substring must be at beginning.

-ends-with str: Substring must be at end.

-is-not str: String must not match.

-is-before str: First string < second string.

-is-after str: First string > second string.

-consists-of str: String must only contain specified characters.

-matches str: Matches without commas or semicolons.

-resembles str: Requires all words, but in any order.

Object Constraints

-is-equal-to expr: Object values must match.

-differs-from expr: Object values must differ.

Numeric Constraints

-gt N: Greater than.

-ge N: Greater than or equal to.

-lt N: Less than to.

-le N: Less than or equal to.

-eq N: Equal to.

-ne N: Not equal to.

Format Customization

-ret str: Override line break between patterns.

-tab str: Replace tab character between fields.

-sep str: Separator between group members.

-pfx str: Prefix to print before group.

-sfx str: Suffix to print after group.

-rst: Reset -sep through -elg.

-clr: Clear queued tab separator.

-pfc str: Preface combines -clr and -pfx.

-deq str: Delete and replace queued tab separator.

-def str: Default placeholder for missing fields.

-lbl str: Insert arbitrary text.

XML Generation

-set tag: XML tag for entire set.

-rec tag: XML tag for each record.

-wrp tag: Wrap elements in XML object.

-enc tag: Encase instance in XML object.

-plg str: Prologue to print before instance.

-elg str: Epilogue to print after instance.

-pkg tag: Package subset in XML object.

-fwd str: Foreword to print before subset.

-awd str: Afterword to print after subset.

Tag and Attribute Construction

-tag tag: Start with <tag.

`-att key str`	Attribute key and literal string.
`-att key element`	Attribute key and element name.

-cls: Close with >.

-slf: Self-close with />.

-end tag: End contents with </tag>.

FASTA Parsable Fields

-bkt: Wrap elements in bracketed fields.

Element Selection

-element element: Print all items that match tag name.

-first element: Only print value of first item.

-last element: Only print value of last item.

-even element: Only print value of even items.

-odd element: Only print value of odd items.

-backward element: Print values in reverse order.

-NAME: Record value in named variable.

--STATS: Accumulate values into variable.

-element Constructs

`Tag`	`Caption`
`Group`	`Initials,LastName`
`Parent/Child`	`MedlineCitation/PMID`
`Recursive`	`"**/Gene-commentary_accession"`
`Unrestricted`	`PubDate/*`
`Attribute`	`DescriptorName@MajorTopicYN`
`Range`	`MedlineDate[1:4]`
`Substring`	"Title[phospholipase \| rattlesnake]"
`Alternative`	"[can contain ^ vertical bar]"
`Object Count`	`"#Author"`
`Item Length`	`"%Title"`
`Element Depth`	"^PMID"
`Variable`	"&NAME"

Special -element Operations

`Parent Index`	"+"
`Object Name`	"?"
`Object Value`	"~"
`XML Subtree`	*""**
`Children`	"$"
`Attributes`	"@"
`ASN.1 Record`	"."
`JSON Record`	"%"

Numeric (Integer) Processing

-num element: Count.

-len element: Length.

-sum element: Sum.

-acc element: Accumulator.

-min element: Minimum.

-max element: Maximum.

-inc element: Increment.

-dec element: Decrement.

-sub element: Difference.

-avg element: Arithmetic mean.

-dev element: Deviation.

-med element: Median.

-mul element: Product.

-div element: Quotient.

-mod element: Remainder.

-geo element: Geometric mean.

-hrm element: Harmonic mean.

-rms element: Root mean square.

-sqt element: Square root.

-lge element: Natural logarithm.

-lg2 element: Logarithm base two.

-log element: Logarithm base ten.

-bin element: Binary.

-oct element: Octal.

-hex element: Hexadecimal.

-bit element: Number of bits set.

Leading Zero Padding

-pad element: Zero-pad to eight digits.

Character Processing

-encode element: XML-encode <, >, &, ", and ' characters.

-decode element: Base64-decode object embedded in XML.

-upper element: Convert text to uppercase.

-lower element: Convert text to lowercase.

-chain element: Change spaces to underscores.

-title element: Capitalize initial letters of words.

-mirror element: Reverse order of letters.

-alnum element: Non-alphabetic characters to space.

-alnum element: Non-alphanumeric characters to space.

String Processing

-basic element: Convert superscripts and subscripts.

-plain element: Remove embedded mixed-content markup tags.

-simple element: Normalize accented letters; spell Greek letters.

-author element: Multi-step author cleanup.

-jour element: Journal capitalization and punctuation punctuation.

-prose element: Text conversion to ASCII.

Text Processing

-terms element: Partition text at spaces.

-words element: Split at punctuation marks.

-pairs element: Adjacent informative words.

-split element: Split using -with for delimiter.

-order element: Rearrange words in sorted order.

-reverse element: Reverse words in string.

-letters element: Separate individual letters.

-clauses element: Break at phrase separators.

-pentamers element: Sliding window of pentamers.

Citation Functions

-year element: Extract first 4-digit year from string.

-month element: Match first month name and return a corresponding integer.

-date element: YYYY/MM/DD from -unit "PubDate" -date "*"

-page element: Get digits (and letters) of first page number.

-auth element: Change GenBank authors to Medline form.

-initials element: Parse initials from forename or given name.

-trim element: Remove extra spaces and leading zeros.

-wct element: Count number of -words in a string.

-doi element: Add https://doi.org/ prefix, URL encode.

-accession element: Allow indexing of full accession.version.

-numeric element: Only accept items that are entirely digits.

Value Transformation

-translate element: Substitute values with -transform table.

-classify element: Substring word or phrase matches to -aliases table.

Regular Expression

-replace

Substitute text using regular expressions.

-reg target: Target expression.

-exp pattern: Replacement pattern.

Sequence Processing

-fasta: Split sequence into blocks of 70 uppercase letters.

Nucleotide Processing

-revcomp: Reverse complement nucleotide sequence.

-nucleic: Subrange determines forward or revcomp.

-ncbi2na: Expand ncbi2na to IUPAC. (May need to truncate result to actual sequence length.)

-ncbi4na: Expand ncbi4na to IUPAC. (May need to truncate result to actual sequence length.)

-cds2prot [-gcode N] [-frame N]: Translate coding region using -gcode and (1-based) -frame (both 1 by default).

Protein Processing

-molwt: Calculate molecular weight of peptide.

-molwt-m: Molecular weight retaining initial methionine.

-molwt-f: Keep initial M residue as formyl-methionine.

-pept: Split amino acid runs at *, -, x, or X.

Sequence Coordinates

-0-based element: Zero-based.

-1-based element: One-based.

-ucsc-based element: Half-open.

Command Generator

-insd arg ...

Generate INSDSeq extraction commands. Print them if invoked standalone; run them if invoked as part of a pipeline. Requires one or more arguments, which may appear in the following order:

Descriptor(s): INSDSeq_sequence/INSDSeq_definition/INSDSeq_division/... [...]

Completeness: complete/partial

Feature(s): CDS/mRNA/...[,...]

Qualifier(s): INSDFeature_key/"#INSDInterval"/gene/product/feat_location/sub_sequence/... [...]

-insdx: Process -insd output table into XML.

Frequency Table

-histogram: Collects data for sort-uniq-count(1) on entire set of records.

Entrez Indexing

-indexer element: Positional index using -wrp for field name.

Output Organization

-head str: Print before everything else.

-tail str: Print after everything else.

-hd str: Print before each record.

-tl str: Print after each record.

Record Selection

-select condition: Select record subset by conditions.

-in filename: File of identifiers to use for selection.

Record Rearrangement

-sort[-fwd] element: Element to use as sort key.

-sort-rev element: Sort records in reverse order.

Reformatting

-format fmt

copy: Fast block copy (still applies processing flags).

compact: Compress runs of spaces.

flush: Suppress line indentation.

indent: Indent according to nesting depth.

expand: Place each attribute on a separate line.

Validation

-verify: Report XML data integrity problems.

-test: Check field for visible combining accents and invisible Unicode.

Summary

-outline: Display outline of XML structure.

-synopsis: Display individual XML paths.

-contour [delimiter]: Display XML paths to leaf nodes (delimited by / by default).

Full Exploration Command Precedence

-pattern

-path

-division

-group

-branch

-block

-section

-subset

-unit

Documentation

-help: Print usage information and some example argument combinations.

-examples: Complete usage examples, involving additional Entrez Direct tools.

-unix: Illustrate common Unix command arguments.

-version: Print version number.

Notes

String constraints use case-insensitive comparisons.

Numeric constraints and selection arguments use integer values.

-num and -len selections are synonyms for Object Count (#) and Item Length (%).

-words, -pairs, and -indices convert to lower case.

Releases

5 releases since Debian 9 (Stretch); the text changed 4 times.

--STATS since Debian 11 (Bullseye); -acc since Debian 12 (Bookworm); -accent since Debian 10 (Buster); -accession since Debian 13 (Trixie); -aliases since Debian 12 (Bookworm); -alnum since Debian 12 (Bookworm); -ascii since Debian 10 (Buster); -auth since Debian 12 (Bookworm); -author since Debian 12 (Bookworm); -awd since Debian 11 (Bullseye); -backward since Debian 12 (Bookworm); -basic since Debian 12 (Bookworm); -bin since Debian 10 (Buster); -bit since Debian 10 (Buster); -bkt since Debian 13 (Trixie); -branch since Debian 13 (Trixie); -chain since Debian 11 (Bullseye); -classify since Debian 12 (Bookworm); -clauses since Debian 10 (Buster); -cls since Debian 13 (Trixie); -consists-of since Debian 13 (Trixie); -contour since Debian 11 (Bullseye); -date since Debian 12 (Bookworm); -decode since Debian 13 (Trixie); -deq since Debian 10 (Buster); -differs-from since Debian 11 (Bullseye); -div since Debian 11 (Bullseye); -division since Debian 13 (Trixie); -doi since Debian 11 (Bullseye); -elg since Debian 10 (Buster); -enc since Debian 11 (Bullseye); -end since Debian 13 (Trixie); -even since Debian 13 (Trixie); -excludes since Debian 13 (Trixie); -exp since Debian 11 (Bullseye); -fasta since Debian 11 (Bullseye); -fwd since Debian 11 (Bullseye); -geo since Debian 13 (Trixie); -hex since Debian 12 (Bookworm); -histogram since Debian 11 (Bullseye); -hrm since Debian 13 (Trixie); -in since Debian 11 (Bullseye); -includes since Debian 12 (Bookworm); -indexer since Debian 13 (Trixie); -initials since Debian 12 (Bookworm); -insdx since Debian 13 (Trixie); -is-after since Debian 11 (Bullseye); -is-before since Debian 11 (Bullseye); -is-equal-to since Debian 11 (Bullseye); -is-within since Debian 10 (Buster); -jour since Debian 12 (Bookworm); -lg2 since Debian 13 (Trixie); -lge since Debian 13 (Trixie); -log since Debian 13 (Trixie); -matches since Debian 11 (Bullseye); -med since Debian 10 (Buster); -mimics since Debian 13 (Trixie); -mirror since Debian 12 (Bookworm); -mixed since Debian 10 (Buster); -mod since Debian 11 (Bullseye); -molwt since Debian 11 (Bullseye); -molwt-f since Debian 13 (Trixie); -molwt-m since Debian 13 (Trixie); -month since Debian 12 (Bookworm); -mul since Debian 11 (Bullseye); -ncbi2na since Debian 11 (Bullseye); -ncbi4na since Debian 11 (Bullseye); -nucleic since Debian 10 (Buster); -numeric since Debian 13 (Trixie); -oct since Debian 12 (Bookworm); -odd since Debian 13 (Trixie); -order since Debian 11 (Bullseye); -pad since Debian 12 (Bookworm); -page since Debian 12 (Bookworm); -path since Debian 11 (Bullseye); -pentamers since Debian 13 (Trixie); -pept since Debian 13 (Trixie); -pkg since Debian 11 (Bullseye); -plain since Debian 11 (Bullseye); -plg since Debian 10 (Buster); -prose since Debian 12 (Bookworm); -rec since Debian 11 (Bullseye); -reg since Debian 11 (Bullseye); -replace since Debian 11 (Bullseye); -resembles since Debian 11 (Bullseye); -revcomp since Debian 10 (Buster); -reverse since Debian 10 (Buster); -rms since Debian 13 (Trixie); -section since Debian 13 (Trixie); -select since Debian 10 (Buster); -self since Debian 12 (Bookworm); -set since Debian 11 (Bullseye); -simple since Debian 12 (Bookworm); -slf since Debian 13 (Trixie); -sort-rev since Debian 12 (Bookworm); -split since Debian 13 (Trixie); -sqt since Debian 13 (Trixie); -stops since Debian 10 (Buster); -strict since Debian 10 (Buster); -tag since Debian 13 (Trixie); -test since Debian 13 (Trixie); -transform since Debian 10 (Buster); -translate since Debian 10 (Buster); -trim since Debian 12 (Bookworm); -unit since Debian 13 (Trixie); -unix since Debian 11 (Bullseye); -verify since Debian 11 (Bullseye); -wct since Debian 12 (Bookworm); -wrp since Debian 10 (Buster); -year since Debian 10 (Buster).

-abstract until Debian 12 (Bookworm); -article until Debian 12 (Bookworm); -cleanup until Debian 9 (Stretch); -e2index until Debian 12 (Bookworm); -exclude until Debian 10 (Buster); -indices until Debian 12 (Bookworm); -mathml until Debian 10 (Buster); -paragraph until Debian 12 (Bookworm); -phrase until Debian 9 (Stretch); -require until Debian 10 (Buster); -script until Debian 10 (Buster); -sort until Debian 11 (Bullseye); -stemmed until Debian 12 (Bookworm); -unicode until Debian 10 (Buster).