API reference¶

Yield one chunk per non-overlapping match of pattern.

Here the pattern matches a whole record (rather than a delimiter), so each match is a chunk and the text between matches is dropped. No lexer is involved — fast, but not token-aware; see split_on_pattern and the "Chunking: lexer vs regex" notes in docs/performance.md for the speed/correctness trade-off.

Parameters:

• text (str) –

  The source to split.

• pattern (str | Pattern[str]) –

  A regular expression matching one record (a str or compiled pattern).

• flags (int | RegexFlag, default: 0 ) –

  re flags, used only when pattern is a str.

• sourcename (str, default: '' ) –

  Optional source name (e.g. a filename) recorded on each Chunk, surfaced during a walk as sourcename.

• trim (bool, default: True ) –

  When True (the default) each match is stripped of surrounding whitespace and a whitespace-only match is dropped. Pass False to keep each match verbatim, dropping only truly empty (zero-length) matches.

Yields:

• Chunk –

  One Chunk per match, carrying its source position; by default trimmed of surrounding whitespace with whitespace-only matches skipped (see trim).

Yield each occurrence of a grammar rule as a chunk.

Unlike the token and regex chunkers, this parses the whole input (with the grammar's ParserInterpreter) to find where the rule occurs — the structural counterpart to the heuristic splitters. The parse runs entirely in C++ and only the resulting (start, stop) spans cross into Python, so it stays cheap relative to the per-chunk walk_parallel re-parse it feeds; prefer it when that callback work dominates, or when no token/regex delimiter cleanly marks a record. See docs/performance.md.

Parameters:

• text (str) –

  The source to split.

• rule (RuleTypes) –

  The rule to chunk by — a rule name or index, or several of them (e.g. "function", or {"function", "class"}).

• start_rule (str | int | None, default: None ) –

  The rule the whole input parses as — a name, an index, or None for the grammar's start rule (index 0).

• outermost (bool, default: True ) –

  When True (default), only top-level occurrences are emitted; a matched rule nested inside another match is skipped. False emits every occurrence (which would overlap).

• sourcename (str, default: '' ) –

  Optional source name (e.g. a filename) recorded on each Chunk, surfaced during a walk as sourcename.

Yields:

• Chunk –

  One Chunk per matched rule occurrence, in source order, trimmed of surrounding whitespace and carrying its position. An empty occurrence (a rule that consumed no token) is skipped.

Release this grammar's cached parser/lexer state.

Repeated walks reuse deserialized parser/lexer state cached per generated class, so clear_cache is never needed for correctness — the caches are keyed by the generated lexer/parser classes, and regenerating a parser means re-importing its module, which yields new classes (and so fresh cache entries) anyway. Call it to release the memory of a grammar you are done with. The per-thread copies built by walk_parallel are thread-local and released with their worker threads, not from here.

Return the innermost open rule's name, or None outside any tracked rule.

Inside a terminal callback this is the rule containing the token; inside an enter/exit callback it is that rule.

Return the current nesting depth: the number of open rule scopes.

The rule_stack holds exactly the rules whose enter/exit events cross into Python — the rules you subscribe to (so this is the depth of the constructs you track). Override enterEveryRule / exitEveryRule to track every rule (true full parse-tree depth).

No-op hook called on entry to every rule; override to use.

Overriding this (or exitEveryRule) subscribes to all rule events, so depth / rule_stack then track the full parse tree. rule_index is the rule's index; rule_name or current_rule gives its name.

No-op hook called on exit from every rule; override to use.

See enterEveryRule — including its warning: useful for debugging and tracing, but overriding disables the native rule filtering, so every rule event crosses into Python.

Tokenize text with the grammar's lexer (no parsing).

This efficiently generates a stream of tokens on all channels.

Use it to inspect the token stream directly: to recover off-channel tokens such as comments that the parser drops and so never reach visitTerminal, to gather token statistics or run a cheap pre-scan, or to build a custom token-aware splitter. It is also the pass the built-in split_on_token / split_between_tokens chunkers run on.

lex works in memory and has no streaming variant: for a large file, chunk it with the stream_* methods and lex each chunk's text (see the "streaming records" recipe in the docs).

Parameters:

• text (str) –

  The source to tokenize.

• keep (Iterable[int] | None, default: None ) –

  Optional set of token types to return; the lexer drops every other token in C++ so only these cross into Python. None returns all tokens.

Yields:

• Every token the lexer produces, in source order, on all channels –

  the default channel plus any hidden or custom channel a rule routes to with -> channel(...) (each token carries its own channel). Excluded are tokens the lexer discards via -> skip (and the partials merged by -> more, which never become standalone tokens), the EOF sentinel, and — when keep is given — any token type not listed.

  On illegal input the lexer applies ANTLR's default recovery — discard the offending character and continue — rather than raising or stopping, so malformed input still yields a token stream for the rest of the text. Those lexer error diagnostics are not surfaced by lex; parse with walk if you need syntax_errors.

Return the (line, column) of the current event's start, or None.

Returns:

• LineCol | None –

  The 1-based line and 0-based column of the current event's start, or None when the event has no source span (e.g. an empty rule). For a walk_parallel chunk the position is reported against the whole source via the chunk's start. The SourceMap is built once per walk on first use.

Return the grammar rule name for a rule index (e.g. from enterEveryRule).

Raises:

• IndexError –

  If index is not a valid rule index (negative indices are not treated as from-the-end positions).

Return the names of the currently open rules, outermost first.

See depth for which rules are tracked. Test containment with "obj" in self.rule_stack() to ask "am I anywhere inside an obj?".

Return the name of the source being parsed (empty if none was set).

This is the sourcename of the Chunk being parsed (the chunkers can set it; the streaming ones default it to their file path), or whatever was passed to walk. Use it with line_col to report a position as sourcename:line:column.

Return the (start, stop) character offsets of the current event.

Offsets are into the whole source: for a walk_parallel chunk they include the chunk's start offset. (-1, -1) when the event has no span.

Yield a chunk for each region bounded by an opener/closer pair.

Token-aware, like split_on_token: it lexes the whole input, so a bracket inside a string or comment is ignored, at the cost of being slower than a plain regex over the text (see the "Chunking: lexer vs regex" notes in docs/performance.md).

Parameters:

• text (str) –

  The source to split.

• pairs (Pair | list[Pair]) –

  One (open, close) pair, or a list of them. Each side is a token type or several equivalent types (e.g. (MyLexer.LBRACE, MyLexer.RBRACE), or ({MyLexer.BEGIN, MyLexer.DO}, {MyLexer.END})). With multiple pairs (e.g. [(LPAREN, RPAREN), (LBRACK, RBRACK)]) each opener is matched only by a closer of its own pair, so distinct bracket kinds nest correctly. Open and close types should be disjoint.

• nested (bool, default: False ) –

  When True, treat regions as balanced — match openers to closers by depth (respecting pair identity) and emit the outermost regions. When False (default), each opener pairs with the next closer of its pair and scanning resumes after it.

• channel (int | None, default: DEFAULT_CHANNEL ) –

  Only tokens on this channel are considered (default: the default channel). Pass None to consider all channels.

• sourcename (str, default: '' ) –

  Optional source name (e.g. a filename) recorded on each Chunk, surfaced during a walk as sourcename.

• trim (bool, default: True ) –

  When True (the default) the region is stripped of surrounding whitespace; pass False to keep it verbatim. The span runs from the opener to the closer, so there is rarely any to strip.

Yields:

• Chunk –

  One Chunk per region — the text from the opener's start to the closer's stop, inclusive — in source order. An unmatched opener yields nothing; a closer with no matching open is ignored.

Split text into chunks at each match of a delimiter regex.

The regex analogue of split_on_token. No lexer is involved, so it is much faster — roughly an order of magnitude at finding delimiters and a few times end to end (the per-chunk Python work is shared) — but not token-aware: a match inside a string or comment still delimits. Prefer it when the delimiter can't appear in disguise; otherwise use the token-based splitter. See the "Chunking: lexer vs regex" notes in docs/performance.md.

Parameters:

• text (str) –

  The source to split.

• pattern (str | Pattern[str]) –

  The delimiter regular expression (a str or compiled pattern).

• where (str, default: 'before' ) –

  "before" starts each chunk at a match; "after" ends each chunk at a match (see split_on_token).

• flags (int | RegexFlag, default: 0 ) –

  re flags, used only when pattern is a str.

• sourcename (str, default: '' ) –

  Optional source name (e.g. a filename) recorded on each Chunk, surfaced during a walk as sourcename.

• trim (bool, default: True ) –

  When True (the default) each chunk is stripped of surrounding whitespace and whitespace-only regions are dropped. Pass False to keep every region verbatim — preserving a leading or trailing whitespace terminator such as a record's mandatory newline — dropping only truly empty (zero-length) regions.

Yields:

• Chunk –

  One Chunk per region between matches, carrying its source position; by default trimmed of surrounding whitespace with whitespace-only regions skipped (see trim).

Split text into chunks at each delimiter token.

Token-aware: because it runs the grammar's lexer, a delimiter that appears inside a string or comment token never causes a split. The price is lexing the whole input — roughly an order of magnitude slower than the regex split_on_pattern at finding the delimiters (~4x end to end), though still negligible next to the parse it feeds. See the "Chunking: lexer vs regex" notes in docs/performance.md.

Parameters:

• text (str) –

  The source to split.

• token_types (TokenTypes) –

  The delimiter token type, or several types that all act as delimiters (e.g. MyLexer.RECORD, or {MyLexer.RECORD, MyLexer.NOTE}).

• where (str, default: 'before' ) –

  "before" starts a new chunk at each delimiter (each chunk begins with one), so content before the first delimiter is its own leading chunk; "after" ends a chunk at each delimiter (each chunk ends with one), so content after the last delimiter is a trailing chunk.

• channel (int | None, default: DEFAULT_CHANNEL ) –

  Only tokens on this channel are split on (default: the default channel). Pass None to consider all channels.

• sourcename (str, default: '' ) –

  Optional source name (e.g. a filename) recorded on each Chunk, surfaced during a walk as sourcename.

• trim (bool, default: True ) –

  When True (the default) each chunk is stripped of surrounding whitespace and whitespace-only regions are dropped. Pass False to keep every region verbatim — preserving a leading or trailing whitespace terminator such as a record's mandatory newline — dropping only truly empty (zero-length) regions.

Yields:

• Chunk –

  One Chunk per region between delimiters, carrying its source position; by default trimmed of surrounding whitespace with whitespace-only regions skipped (see trim).

Stream chunks from a file that is a sequence of a grammar rule.

The streaming counterpart of chunk_by_rule, for input that is a top-level sequence of records — each record an occurrence of rule (or one of several rules). It parses one record at a time over a bounded-memory pipeline (the native layer opens the file and runs lexer → parser over a sliding window), yielding each as a positioned Chunk without holding the whole token stream or parse tree. The chunks feed walk_parallel like any other.

Unlike chunk_by_rule — which parses the whole input and finds the rule anywhere in the tree — this is the bounded-memory "file of records" form. Records must be directly adjacent: only lexer-skipped tokens (whitespace, comments) may sit between them. With several candidate rules, the next token chooses which to parse (via each rule's start-token set), so the candidates should have disjoint leading tokens (e.g. class vs def); on overlap the first listed wins. An on-channel separator between records (e.g. a comma) is not supported — use chunk_by_rule or stream_on_token there.

Parameters:

• path (str | PathLike[str]) –

  Filesystem path to the source (opened by the native layer as UTF-8).

• rule (RuleTypes) –

  The record rule — a rule name or index, or several of them (a set of top-level record types, e.g. {"classdef", "funcdef"}).

• sourcename (str, default: '' ) –

  Source name recorded on each Chunk (and surfaced as sourcename during a walk). Defaults to str(path).

• encoding (str, default: 'utf-8' ) –

  The source encoding. Only UTF-8 is supported today (Python codec aliases are accepted); the keyword is reserved for future encodings.

• batch (int, default: 256 ) –

  How many records to pull from C++ per call — a throughput knob.

• _block_bytes (int, default: 0 ) –

  Internal test hook: the file read-block size in bytes (0 = the default). Not part of the public API.

Yields:

• Chunk –

  One Chunk per record, in source order, carrying its position. The whole input must be a sequence of records: the stream stops cleanly only at end of input (trailing lexer-skipped whitespace/comments are fine).

Raises:

• ValueError –

  If encoding is not UTF-8, or a rule name is unknown.

• RuntimeError –

  If an on-channel token begins no candidate rule (e.g. an unsupported header or record separator) or a chosen record fails to make progress — the message names the offending token and the candidate rules. Records parsed before the offending token are yielded first, so the error surfaces only after them. (Previously the stream stopped silently here, which could hide a malformed input as an empty or truncated result.)

Stream chunks from a text source at each delimiter regex match.

The streaming counterpart of split_on_pattern: it reads source incrementally and yields positioned Chunks without holding the whole input, so paired with walk_parallel the pipeline stays bounded. Because the regex is Python's, encoding is handled on the Python side (unlike the lexer-based stream_on_token, which reads UTF-8 in C++) — any encoding a text file supports works.

A delimiter match is only committed once a character past it has been read (or the source ends), so a match is never split across a read boundary, as long as the delimiter fits within the read window. A region with no delimiter is buffered in full (like the other streamers).

Parameters:

• source (TextSource) –

  A filesystem path (opened with encoding), an already-open text file object, or any iterable of str pieces (e.g. lines, or a generator). For an in-memory string, use split_on_pattern instead.

• pattern (str | Pattern[str]) –

  The delimiter regular expression (a str or compiled pattern).

• where (str, default: 'before' ) –

  "before" starts each chunk at a match; "after" ends each chunk at a match (see split_on_pattern).

• flags (int | RegexFlag, default: 0 ) –

  re flags, used only when pattern is a str.

• window_chars (int | None, default: 65536 ) –

  Read/search increment in characters (default 64K). Controls how often the regex re-runs and how much read-ahead is buffered, not the output. Used when reading a path or file object.

• window_lines (int | None, default: None ) –

  If set, read this many lines per step instead (still capped by window_chars if both are given) — convenient for line-oriented delimiters. Ignored for a plain str iterable, which is consumed as-is.

• encoding (str, default: 'utf-8' ) –

  Text encoding, used only when source is a path. Any codec Python supports.

• sourcename (str, default: '' ) –

  Source name recorded on each Chunk (and surfaced as sourcename during a walk). Defaults to the path when source is a path, else None — pass it for a stream or iterable that has no path.

• trim (bool, default: True ) –

  When True (the default) each chunk is stripped of surrounding whitespace and whitespace-only regions are dropped. Pass False to keep every region verbatim — preserving a leading or trailing whitespace terminator such as a record's mandatory newline — dropping only truly empty (zero-length) regions.

Yields:

• Chunk –

  One Chunk per region between matches, carrying its source position; by default trimmed of surrounding whitespace with whitespace-only regions skipped (see trim). Equivalent to split_on_pattern over the source's decoded text.

Raises:

• ValueError –

  If where is not "before"/"after".

Stream chunks from a file at each delimiter token, without holding it all.

The streaming counterpart of split_on_token: instead of taking the whole source as a str, it opens path in C++ and lexes it incrementally over a sliding window, slicing out and freeing each chunk as it goes — so peak memory is roughly one chunk rather than the whole file. The yielded Chunks drop straight into walk_parallel, which pulls them lazily, keeping the whole pipeline bounded.

Parameters:

• path (str | PathLike[str]) –

  Filesystem path to the source (opened by the native layer as UTF-8).

• token_types (TokenTypes) –

  The delimiter token type, or several types that all act as delimiters (see split_on_token).

• where (str, default: 'before' ) –

  "before" starts a new chunk at each delimiter; "after" ends a chunk at each delimiter (see split_on_token).

• encoding (str, default: 'utf-8' ) –

  The source encoding. Only UTF-8 is supported today (Python codec aliases such as "utf8" are accepted); the keyword is reserved so other encodings can be added later. For a non-UTF-8 source now, decode it in Python (Path(p).read_text(encoding=...)) and use the in-memory split_on_token.

• channel (int | None, default: DEFAULT_CHANNEL ) –

  Only tokens on this channel are split on (default: the default channel). Pass None to consider all channels.

• sourcename (str, default: '' ) –

  Source name recorded on each Chunk (and surfaced as sourcename during a walk). Defaults to str(path).

• trim (bool, default: True ) –

  When True (the default) each chunk is stripped of surrounding whitespace and whitespace-only regions are dropped. Pass False to keep every region verbatim — preserving a leading or trailing whitespace terminator such as a record's mandatory newline — dropping only truly empty (zero-length) regions.

• batch (int, default: 256 ) –

  How many chunk records to pull from C++ per call — a throughput knob, not observable in the output.

• _block_bytes (int, default: 0 ) –

  Internal test hook: the file read-block size in bytes (0 = the default). Not part of the public API.

Yields:

• Chunk –

  One Chunk per region between delimiters, carrying its source position; by default trimmed of surrounding whitespace with whitespace-only regions skipped (see trim). Equivalent to split_on_token over the file's text.

Raises:

• ValueError –

  If where is not "before"/"after", or encoding is not UTF-8.

Return the source text of the current event (rule or terminal).

Inside visitTerminal this is the token's text (the same string passed to it); inside an enter/exit rule callback it is the rule's whole extent — e.g. the full {...} of an object. Empty when the event has no span (an empty rule, or an inserted/missing error token).

Return the name for a token type — symbolic (e.g. STRING), else literal.

Falls back to the literal name (e.g. "'{'") for anonymous tokens, and to the type's number as a string for an unknown type. Useful for logging / debugging and generic visitTerminal handlers.

No-op error callback; override in a subclass to handle error nodes.

No-op terminal callback; override in a subclass to handle tokens.

Parse text with this listener's baked-in lexer/parser and return self.

Runs the parse in C++ and dispatches the event stream to this listener's callbacks. The lexer/parser are baked into the generated subclass, so no class arguments are needed.

Parameters:

• start_rule (int | str | None, default: None ) –

  The rule to parse as — a rule name, a rule index, or None for the grammar's default start rule.

• filtered (bool, default: True ) –

  When True (default), only overridden rules/tokens are emitted by C++; False forces the full event stream.

Returns:

• Self –

  self, so calls chain (e.g. result = Collector().walk(text).result).

Raises:

• TypeError –

  If called on a class that is not a generated <Grammar>EventListener subclass.

Parse independent chunks across a thread pool, one listener each.

The native parse releases the GIL, so the parses overlap across cores. Each worker thread uses its own copy of the parser state (built once per thread), so they never contend on a shared ATN. The per-event Python dispatch still holds the GIL, so parallel speedup scales with how parse-heavy the work is relative to per-callback Python work — see the "Parallel parsing" section of docs/performance.md.

Parameters:

• chunks (Iterable[str | Chunk]) –

  The pieces of source to parse, each a self-contained piece (e.g. one record or top-level definition) that parses as start_rule. A bare str is treated as contiguous with the previous chunk and its source position is computed; a Chunk pins an explicit offset / line / column so callbacks report span / line_col against the whole source. Mix freely: a Chunk re-anchors the running position for the contiguous str chunks that follow it.

• start_rule (int | str | None, default: None ) –

  The rule each chunk parses as — a rule name, a rule index, or None for the grammar's start rule.

• max_workers (int | None, default: None ) –

  The maximum number of parses in flight at once (also the ordering/buffer window). Defaults to os.cpu_count(). With 1 it runs inline, without a pool.

• filtered (bool, default: True ) –

  When True (default), only overridden rules/tokens are emitted by C++; False forces the full event stream.

• factory (Callable[[], Self] | None, default: None ) –

  A zero-argument callable returning a fresh listener, for subclasses whose constructor needs arguments. Defaults to cls.

Returns:

• Iterator[Self] –

  A lazy iterator of listeners, one per chunk, in input order. Chunks are pulled and parsed on demand with at most max_workers parses in flight, so neither the whole input nor all results are held at once — consume it incrementally (or list(...) it if you want them all). Each listener carries its accumulated state plus its syntax_errors.

Raises:

• TypeError –

  If called on a class that is not a generated <Grammar>EventListener subclass (no baked-in lexer/parser).

Return a Chunk for text positioned immediately after this one.

Computes the starting location based on the contents and starting position of the current chunk and copies the sourcename.

Return the (line, column) for a character offset.

Parameters:

• offset (int) –

  A non-negative character (codepoint) offset into the source.

Returns:

• LineCol –

  The 1-based line and 0-based column of offset.

Raises:

• ValueError –

  If offset is negative.

Return the character offset given line and column.

The inverse of line_col: offset(*line_col(o)) == o for any valid offset o.

Parameters:

• line (int) –

  A 1-based line number.

• column (int, default: 0 ) –

  A 0-based column within the line. Added to the line's start offset without bounds-checking against the line length, so a column past the end of the line yields an offset into a later line.

Returns:

• The character (codepoint) offset of `line` –

  column.

Raises:

• ValueError –

  If line is outside 1..number-of-lines.

Adds other line/column offsets to this LineCol.

If offset line is 1, this simply adds the column, otherwise it adds the line and sets the column from offset.