API reference¶

Yield one chunk per non-overlapping match of pattern.

Here the pattern matches a whole record (rather than a delimiter), so each match is a chunk and the text between matches is dropped. No lexer is involved, so it is fast but not token-aware. See split_on_pattern and the "Chunking: lexer vs regex" notes in docs/performance.md for the trade-off between speed and correctness.

Parameters:

• text (str) –

  The source to split.

• pattern (str | Pattern[str]) –

  A regular expression matching one record (a str or compiled pattern).

• flags (int | RegexFlag, default: 0 ) –

  re flags, used only when pattern is a str.

• sourcename (str, default: '' ) –

  Optional source name (e.g. a filename) recorded on each Chunk, surfaced during a walk as sourcename.

• trim (bool, default: True ) –

  When True (the default) each match is stripped of surrounding whitespace and a whitespace-only match is dropped. Pass False to keep each match verbatim. Only empty (zero-length) matches are then dropped.

Yields:

• Chunk –

  One Chunk per match, carrying its source position; by default trimmed of surrounding whitespace with whitespace-only matches skipped (see trim).

Yield each occurrence of a grammar rule as a chunk.

Unlike the token and regex chunkers, this parses the whole input with the grammar's ParserInterpreter to find where the rule occurs, which makes it the structural counterpart to those heuristic splitters. The parse runs entirely in C++ and only the (start, stop) spans cross into Python, so it stays cheap compared with the per-chunk walk_parallel parse it feeds. Use it when the per-chunk callback work dominates, or when no token or regex delimiter cleanly marks a record. See docs/performance.md.

Parameters:

• text (str) –

  The source to split.

• rule (RuleTypes) –

  The rule to chunk by: a rule name or index, or several of them (e.g. "function", or {"function", "class"}).

• start_rule (str | int | None, default: None ) –

  The rule to parse the whole input as: a name, an index, or None for the grammar's start rule (index 0).

• outermost (bool, default: True ) –

  When True (default), only top-level occurrences are emitted; a matched rule nested inside another match is skipped. False emits every occurrence, so nested occurrences overlap.

• sourcename (str, default: '' ) –

  Optional source name (e.g. a filename) recorded on each Chunk, surfaced during a walk as sourcename.

Yields:

• Chunk –

  One Chunk per matched rule occurrence, in source order, trimmed of surrounding whitespace and carrying its position. An empty occurrence (a rule that consumed no token) is skipped.

Release this grammar's cached parser and lexer state.

Repeated walks reuse deserialized parser and lexer state cached per generated class. You never need clear_cache for correctness. The caches are keyed by the generated lexer and parser classes, and regenerating a parser means re-importing its module, which produces new classes and therefore new cache entries. Call it to release the memory of a grammar you are done with. The per-thread copies built by walk_parallel are thread-local and released with their worker threads, not from here.

Return the innermost open rule's name, or None outside any tracked rule.

Inside a terminal callback this is the rule containing the token; inside an enter<Rule> or exit<Rule> callback it is that rule.

Return the current nesting depth: the number of open rule scopes.

The rule_stack holds only the rules you subscribe to, because only their events cross into Python. The depth therefore counts the constructs you track. Override enterEveryRule or exitEveryRule to track every rule (the full parse-tree depth).

No-op hook called on entry to every rule; override it to use it.

Overriding this (or exitEveryRule) subscribes to all rule events, so depth and rule_stack track the full parse tree. rule_index is the rule's index; rule_name or current_rule gives its name.

No-op hook called on exit from every rule; override it to use it.

See enterEveryRule, including its warning. This hook is useful for debugging and tracing, but overriding it disables the native rule filtering, so every rule event crosses into Python.

Tokenize text with the grammar's lexer (no parsing).

This efficiently generates a stream of tokens on all channels.

Use it to inspect the token stream directly: to recover off-channel tokens such as comments that the parser drops and so never reach visitTerminal, to gather token statistics or run a cheap pre-scan, or to build a custom token-aware splitter. It is also the pass the built-in split_on_token and split_between_tokens chunkers run on.

lex works in memory and has no streaming variant: for a large file, chunk it with the stream_* methods and lex each chunk's text (see the "streaming records" recipe in the docs).

Parameters:

• text (str) –

  The source to tokenize.

• keep (Iterable[int] | None, default: None ) –

  Optional set of token types to return. The lexer drops every other token in C++, so only these cross into Python. None returns all tokens.

Yields:

• Every token the lexer produces, in source order, on all channels –

  the default channel plus any hidden or custom channel a rule routes to with -> channel(...) (each token carries its own channel). The stream excludes tokens the lexer discards with -> skip, the partial tokens that -> more merges into the next token, the EOF sentinel, and, if keep is given, any token type not in it.

  On illegal input the lexer applies ANTLR's default recovery (it discards the offending character and continues) instead of raising or stopping, so malformed input still yields a token stream for the rest of the text. Those lexer error diagnostics are not surfaced by lex; parse with walk if you need syntax_errors.

Return the (line, column) of the current event's start, or None.

Returns:

• LineCol | None –

  The 1-based line and 0-based column of the current event's start, or None when the event has no source span (e.g. an empty rule). For a walk_parallel chunk the position is reported against the whole source via the chunk's start. The SourceMap is built once per walk on first use.

Return the grammar rule name for a rule index (e.g. from enterEveryRule).

Raises:

• IndexError –

  If index is not a valid rule index (negative indices are not treated as from-the-end positions).

Return the names of the currently open rules, outermost first.

See depth for which rules are tracked. Test containment with "obj" in self.rule_stack() to ask "am I anywhere inside an obj?".

Return the name of the source being parsed (empty if none was set).

This is the sourcename of the Chunk being parsed (set by the chunkers; the streaming chunkers default it to the file path). Use it with line_col to report a position as sourcename:line:column.

Return the (start, stop) character offsets of the current event.

Offsets are into the whole source: for a walk_parallel chunk they include the chunk's start offset. Returns (-1, -1) when the event has no span.

Yield a chunk for each region bounded by an opener and closer pair.

Token-aware, like split_on_token: it lexes the whole input, so a bracket inside a string or comment is ignored, at the cost of being slower than a plain regex over the text (see the "Chunking: lexer vs regex" notes in docs/performance.md).

Parameters:

• text (str) –

  The source to split.

• pairs (Pair | list[Pair]) –

  One (open, close) pair, or a list of them. Each side is a token type or several equivalent types (e.g. (MyLexer.LBRACE, MyLexer.RBRACE), or ({MyLexer.BEGIN, MyLexer.DO}, {MyLexer.END})). With multiple pairs (e.g. [(LPAREN, RPAREN), (LBRACK, RBRACK)]) each opener is matched only by a closer from its own pair, so distinct bracket kinds nest correctly. Open and close types should be disjoint.

• nested (bool, default: False ) –

  When True, treat regions as balanced: match openers to closers by depth (respecting pair identity) and emit the outermost regions. When False (default), each opener pairs with the next closer of its pair and scanning resumes after it.

• channel (int | None, default: DEFAULT_CHANNEL ) –

  Only tokens on this channel are considered (default: the default channel). Pass None to consider all channels.

• sourcename (str, default: '' ) –

  Optional source name (e.g. a filename) recorded on each Chunk, surfaced during a walk as sourcename.

• trim (bool, default: True ) –

  When True (the default) the region is stripped of surrounding whitespace; pass False to keep it verbatim. The span runs from the opener to the closer, so there is rarely any to strip.

Yields:

• Chunk –

  One Chunk per region (the text from the opener's start to the closer's stop, inclusive), in source order. An unmatched opener yields nothing; a closer with no matching opener is ignored.

Split text into chunks at each match of a delimiter regex.

The regex analogue of split_on_token. No lexer is involved, so it is much faster: about 7x faster at finding delimiters and about 4x faster end to end (the per-chunk Python work is the same). However, it is not token-aware, so a match inside a string or comment still counts as a delimiter. Use it when the delimiter text cannot appear inside other tokens; otherwise use the token-based splitter. See the "Chunking: lexer vs regex" notes in docs/performance.md.

Parameters:

• text (str) –

  The source to split.

• pattern (str | Pattern[str]) –

  The delimiter regular expression (a str or compiled pattern).

• where (str, default: 'before' ) –

  "before" starts each chunk at a match; "after" ends each chunk at a match (see split_on_token).

• flags (int | RegexFlag, default: 0 ) –

  re flags, used only when pattern is a str.

• sourcename (str, default: '' ) –

  Optional source name (e.g. a filename) recorded on each Chunk, surfaced during a walk as sourcename.

• trim (bool, default: True ) –

  When True (the default) each chunk is stripped of surrounding whitespace and whitespace-only regions are dropped. Pass False to keep every region verbatim, including a leading or trailing whitespace terminator such as a record's mandatory newline. Only empty (zero-length) regions are then dropped.

Yields:

• Chunk –

  One Chunk per region between matches, carrying its source position; by default trimmed of surrounding whitespace with whitespace-only regions skipped (see trim).

Split text into chunks at each delimiter token.

Token-aware: because it runs the grammar's lexer, a delimiter that appears inside a string or comment token never causes a split. The cost is lexing the whole input. Finding the delimiters is about 7x slower than with the regex-based split_on_pattern (about 4x end to end), but still negligible compared with the parse it feeds. See the "Chunking: lexer vs regex" notes in docs/performance.md.

Parameters:

• text (str) –

  The source to split.

• token_types (TokenTypes) –

  The delimiter token type, or several types that all act as delimiters (e.g. MyLexer.RECORD, or {MyLexer.RECORD, MyLexer.NOTE}).

• where (str, default: 'before' ) –

  "before" starts a new chunk at each delimiter (each chunk begins with one), so content before the first delimiter is its own leading chunk; "after" ends a chunk at each delimiter (each chunk ends with one), so content after the last delimiter is a trailing chunk.

• channel (int | None, default: DEFAULT_CHANNEL ) –

  Only tokens on this channel are split on (default: the default channel). Pass None to consider all channels.

• sourcename (str, default: '' ) –

  Optional source name (e.g. a filename) recorded on each Chunk, surfaced during a walk as sourcename.

• trim (bool, default: True ) –

  When True (the default) each chunk is stripped of surrounding whitespace and whitespace-only regions are dropped. Pass False to keep every region verbatim, including a leading or trailing whitespace terminator such as a record's mandatory newline. Only empty (zero-length) regions are then dropped.

Yields:

• Chunk –

  One Chunk per region between delimiters, carrying its source position; by default trimmed of surrounding whitespace with whitespace-only regions skipped (see trim).

Stream chunks from a file that is a sequence of a grammar rule.

The streaming counterpart of chunk_by_rule, for input that is a top-level sequence of records, each an occurrence of rule (or of one of several rules). It parses one record at a time over a bounded-memory pipeline (the native layer opens the file and runs lexer → parser over a sliding window), yielding each as a positioned Chunk without holding the whole token stream or parse tree. The chunks feed walk_parallel like any other.

Unlike chunk_by_rule, which parses the whole input and finds the rule anywhere in the tree, this form is for a bounded-memory "file of records". Records must be directly adjacent: only lexer-skipped tokens (whitespace, comments) may appear between them. With several candidate rules, the next token decides which one to parse (using each rule's start-token set), so the candidates should begin with different tokens (e.g. class vs def). If they overlap, the first listed rule wins. An on-channel separator between records (e.g. a comma) is not supported; use chunk_by_rule or stream_on_token instead.

Parameters:

• path (str | PathLike[str]) –

  Filesystem path to the source (opened by the native layer as UTF-8).

• rule (RuleTypes) –

  The record rule: a rule name or index, or several of them (a set of top-level record types, e.g. {"classdef", "funcdef"}).

• sourcename (str, default: '' ) –

  Source name recorded on each Chunk (and surfaced as sourcename during a walk). Defaults to str(path).

• encoding (str, default: 'utf-8' ) –

  The source encoding. Only UTF-8 is supported today (Python codec aliases are accepted); the keyword is reserved for future encodings.

• batch (int, default: 256 ) –

  How many records to fetch from C++ per call. This affects throughput only.

• _block_bytes (int, default: 0 ) –

  Internal test hook: the file read-block size in bytes (0 = the default). Not part of the public API.

Yields:

• Chunk –

  One Chunk per record, in source order, carrying its position. The whole input must be a sequence of records: the stream stops cleanly only at end of input (trailing whitespace and comments that the lexer skips are fine).

Raises:

• ValueError –

  If encoding is not UTF-8, or a rule name is unknown.

• RuntimeError –

  If an on-channel token begins no candidate rule (e.g. an unsupported header or record separator) or a chosen record fails to make progress. The message names the offending token and the candidate rules. Records parsed before the offending token are yielded first, so the error is raised only after them.

Split a text source into chunks at each regex match, reading incrementally.

The streaming counterpart of split_on_pattern: it reads source incrementally and yields positioned Chunks without holding the whole input, so paired with walk_parallel the pipeline stays bounded. Because matching uses Python's re, decoding also happens in Python, so any encoding that Python text files support works. (The lexer-based stream_on_token reads UTF-8 in C++.)

A delimiter match is only committed once a character past it has been read (or the source ends), so a match is never split across a read boundary, provided the delimiter fits within the read window. A region with no delimiter is buffered in full (like the other streamers).

Parameters:

• source (TextSource) –

  A filesystem path (opened with encoding), an already-open text file object, or any iterable of str pieces (e.g. lines, or a generator). For an in-memory string, use split_on_pattern instead.

• pattern (str | Pattern[str]) –

  The delimiter regular expression (a str or compiled pattern).

• where (str, default: 'before' ) –

  "before" starts each chunk at a match; "after" ends each chunk at a match (see split_on_pattern).

• flags (int | RegexFlag, default: 0 ) –

  re flags, used only when pattern is a str.

• window_chars (int | None, default: 65536 ) –

  How many characters to read and search per step (default 64K). Controls how often the regex re-runs and how much read-ahead is buffered, not the output. Used when reading a path or file object.

• window_lines (int | None, default: None ) –

  If set, read this many lines per step instead (still capped by window_chars if both are given). Useful for line-oriented delimiters. Ignored for a plain str iterable, which is consumed as-is.

• encoding (str, default: 'utf-8' ) –

  Text encoding, used only when source is a path. Any codec Python supports.

• sourcename (str, default: '' ) –

  Source name recorded on each Chunk (and surfaced as sourcename during a walk). Defaults to the path when source is a path and to an empty string otherwise, so pass it for a stream or iterable.

• trim (bool, default: True ) –

  When True (the default) each chunk is stripped of surrounding whitespace and whitespace-only regions are dropped. Pass False to keep every region verbatim, including a leading or trailing whitespace terminator such as a record's mandatory newline. Only empty (zero-length) regions are then dropped.

Yields:

• Chunk –

  One Chunk per region between matches, carrying its source position; by default trimmed of surrounding whitespace with whitespace-only regions skipped (see trim). Equivalent to split_on_pattern over the source's decoded text.

Raises:

• ValueError –

  If where is not "before" or "after".

Split a file into chunks at each delimiter token without loading it whole.

The streaming counterpart of split_on_token: instead of taking the whole source as a str, it opens path in C++ and lexes it incrementally over a sliding window, slicing out and freeing each chunk as it goes, so peak memory is roughly one chunk rather than the whole file. The yielded Chunks can be passed directly to walk_parallel, which pulls them lazily, keeping the whole pipeline bounded.

Parameters:

• path (str | PathLike[str]) –

  Filesystem path to the source (opened by the native layer as UTF-8).

• token_types (TokenTypes) –

  The delimiter token type, or several types that all act as delimiters (see split_on_token).

• where (str, default: 'before' ) –

  "before" starts a new chunk at each delimiter; "after" ends a chunk at each delimiter (see split_on_token).

• encoding (str, default: 'utf-8' ) –

  The source encoding. Only UTF-8 is supported today (Python codec aliases such as "utf8" are accepted); the keyword is reserved so other encodings can be added later. For a non-UTF-8 source now, decode it in Python (Path(p).read_text(encoding=...)) and use the in-memory split_on_token.

• channel (int | None, default: DEFAULT_CHANNEL ) –

  Only tokens on this channel are split on (default: the default channel). Pass None to consider all channels.

• sourcename (str, default: '' ) –

  Source name recorded on each Chunk (and surfaced as sourcename during a walk). Defaults to str(path).

• trim (bool, default: True ) –

  When True (the default) each chunk is stripped of surrounding whitespace and whitespace-only regions are dropped. Pass False to keep every region verbatim, including a leading or trailing whitespace terminator such as a record's mandatory newline. Only empty (zero-length) regions are then dropped.

• batch (int, default: 256 ) –

  How many chunk records to fetch from C++ per call. This affects throughput only, not the output.

• _block_bytes (int, default: 0 ) –

  Internal test hook: the file read-block size in bytes (0 = the default). Not part of the public API.

Yields:

• Chunk –

  One Chunk per region between delimiters, carrying its source position; by default trimmed of surrounding whitespace with whitespace-only regions skipped (see trim). Equivalent to split_on_token over the file's text.

Raises:

• ValueError –

  If where is not "before" or "after", or encoding is not UTF-8.

Return the source text of the current event (rule or terminal).

Inside visitTerminal this is the token's text (the same string passed to it); inside an enter<Rule> or exit<Rule> callback it is the rule's full text, such as the whole {...} of an object. Empty when the event has no span (an empty rule, or an error token that recovery inserted or reported missing).

Return the name of a token type, such as STRING.

Uses the symbolic name if there is one and falls back to the literal name (e.g. "'{'") for anonymous tokens. For an unknown type it returns the type number as a string. Useful for logging, debugging, and generic visitTerminal handlers.

No-op error callback; override in a subclass to handle error nodes.

No-op terminal callback; override in a subclass to handle tokens.

Parse text with this grammar's lexer and parser and return self.

Runs the parse in C++ and dispatches the event stream to this listener's callbacks. The generated subclass already references the lexer and parser classes, so you do not pass them.

Parameters:

• text (str) –

  The source text to parse.

• start_rule (int | str | None, default: None ) –

  The rule to start parsing at: a rule name, a rule index, or None for the grammar's default start rule.

• filtered (bool, default: True ) –

  When True (default), the native parser emits events only for the rules and tokens whose callbacks you override. False emits the full event stream.

Returns:

• Self –

  self, so calls chain (e.g. result = Collector().walk(text).result).

Raises:

• TypeError –

  If called on a class that is not a generated <Grammar>EventListener subclass.

Parse independent chunks across a thread pool, one listener each.

The native parse releases the GIL, so the parses overlap across cores. Each worker thread builds its own copy of the parser state once, so threads never contend on a shared ATN. The per-event Python dispatch still holds the GIL, so the speedup depends on how much of the time goes to parsing rather than to Python callbacks. See the "Parallel parsing" section of docs/performance.md.

Parameters:

• chunks (Iterable[str | Chunk]) –

  The pieces of source to parse, each a self-contained piece (e.g. one record or top-level definition) that parses as start_rule. A bare str is treated as contiguous with the previous chunk and its source position is computed; a Chunk pins an explicit offset, line, and column so callbacks report span and line_col against the whole source. You can mix the two: a Chunk resets the running position for the contiguous str chunks that follow it.

• start_rule (int | str | None, default: None ) –

  The rule each chunk parses as: a rule name, a rule index, or None for the grammar's start rule.

• max_workers (int | None, default: None ) –

  The maximum number of parses running at once. This is also the size of the window used to buffer results and keep them in input order. Defaults to os.cpu_count(). With 1 it runs inline, without a pool.

• filtered (bool, default: True ) –

  When True (default), the native parser emits events only for the rules and tokens whose callbacks you override. False emits the full event stream.

• factory (Callable[[], Self] | None, default: None ) –

  A zero-argument callable returning a fresh listener, for subclasses whose constructor needs arguments. Defaults to cls.

Returns:

• Iterator[Self] –

  A lazy iterator of listeners, one per chunk, in input order. Chunks are pulled and parsed on demand with at most max_workers parses in flight, so neither the whole input nor all the results are held at once. Consume the iterator incrementally, or call list(...) on it to collect every result. Each listener carries its accumulated state plus its syntax_errors.

Raises:

• TypeError –

  If called on a class that is not a generated <Grammar>EventListener subclass.

Return a Chunk for text positioned immediately after this one.

Computes the starting location based on the contents and starting position of the current chunk, and copies the sourcename.

Return the (line, column) for a character offset.

Parameters:

• offset (int) –

  A non-negative character (codepoint) offset into the source.

Returns:

• LineCol –

  The 1-based line and 0-based column of offset.

Raises:

• ValueError –

  If offset is negative.

Return the character offset of a line and column.

The inverse of line_col: offset(*line_col(o)) == o for any valid offset o.

Parameters:

• line (int) –

  A 1-based line number.

• column (int, default: 0 ) –

  A 0-based column within the line. Added to the line's start offset without bounds-checking against the line length, so a column past the end of the line yields an offset into a later line.

Returns:

• The character (codepoint) offset of `line` –

  column.

Raises:

• ValueError –

  If line is outside 1..number-of-lines.

Return this position advanced by a relative offset.

If offset.line is 1 (or less), only offset.column is added. Otherwise the line advances by offset.line - 1 and the column is taken from offset.