返回资讯中心
外部精选
软件工程
#开发者实时流

Show HN: ISPA – Parser generator with grammar-level debugger and error recovery

ISPA is a static parser and lexer generator written from scratch in modern C++ (C++23). You describe a grammar once; ISPA compiles it ahead of time into a lexer and parser that build a fully typed AST. Generated C++ tar…

Hacker News 最新Sinfolke2026/10/26 · 20 天前30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

ISPA is a static parser and lexer generator written from scratch in modern C++ (C++23). You describe a grammar once; ISPA compiles it ahead of time into a lexer and parser that build a fully typed AST. Generated C++ targets C++17; Python output is also available.

The wiki covers setup, grammar syntax, generated APIs, algorithms, recovery, external parsers, packages, and debugging.

usage.mp4

It is built around one idea: easy to use, without giving up performance, extensibility, or features. In benchmarks it is about 12x faster than ANTLR; see Performance.

Static generation. Everything is computed at generation time. The grammar is compiled to DFA tables and plain code. Nothing is generated, interpreted, or compiled at runtime.

Small target runtime. Generated C++ uses headers and core.cpp from parser/stdlibs; generated Python imports ispa_std. Package exports can put these on your build or module path.

Multi-language output. Code generation goes through LangAPI, a language-agnostic IR. C++17 and Python emitters are implemented; Python currently uses LL generation.

Extensible grammar descriptions. Rules describe language structure and connect to generated parser code.

Scales from trivial to larger projects. Incremental generation and grammar imports help organize and rebuild shared grammars. Feature-gated and parametrized module work remains on the roadmap.

Works out of the box. Generate, include, call. The generated API returns a typed AST with no post-parsing step or callbacks required.

🚀 Current State

Area Status

Compiler (frontend + backend) Implemented; builds on Windows and Linux

Bootstrap Done. ISPA's own grammar is parsed by an ISPA-generated parser, which fully replaced the hand-written one

Lexer (TDFA, AST-building tokens) Done

Parser (LL) Production target; actively growing, with further work and testing ongoing

Parser (LR-family modes) Less capable and experimental for now; see LR backend status

Incremental generation Implemented (regenerates changed parts only)

C++17 target Implemented; output compiles and runs

Python target Implemented for LL generation; generated modules use ispa_std from the target runtime

Dynamic regex (back-references, counts from earlier captures) Syntax parsed, not yet lowered to output

Grammar syntax error reporting Automatic LL recovery and source diagnostics; syntax errors stop generation

The backends lower grammar structure and AST captures to C++ or Python. Support for advanced actions and recovery differs by algorithm; see LR backend status.

Generate and build a parser with the current generator, then run its existing executable under ispadbg:

./ispadbg --purpose matching --purpose ast -b grammar.isc:12
--record parser-trace.json -- ./my-parser input.txt ./ispadbg --replay parser-trace.json

For a graphical view, run ./ispadbg-gui (or the installed ispadbg-gui). Choose a parser program, enter an input string, then click Run. The input is passed as one command-line argument by default, as in examples/main.cpp; choose Standard input for parsers that read stdin. Use Step, Next, Finish, and Continue while paused. Select any trace row to inspect its variables, including ctx and AST values, in the expandable tree. Open trace and Save trace use the same JSON format as ispadbg --replay and --record. The GUI uses Python's Tkinter and needs a desktop display.

Debug instrumentation is enabled by default when generating parsers. Pass --no-debug-instrumentation to ispa to omit generated source checkpoints, scope events, and value serialization. Pass --debug-instrumentation to explicitly enable them.

Purpose IDs are matching, failure, and ast; failure is the default. The debugger prints the selected IDs at startup. The interactive commands are s (step), n (next), f (finish), c (continue), b file:line (breakpoint), p [name] (inspect a value), ctx, back, forward, and live. back and forward inspect recorded snapshots while the parser is paused; execution resumes from the live pause. Use --no-interactive to record without pausing.

C++ generated code emits rule and production scopes, match outcomes, typed captures, source checkpoints for converted Python statements, and typed AST snapshots. The Python target uses the same v2 event protocol through parser/stdlibs/target/ispa_debug.py; include that directory in PYTHONPATH alongside ispa_std.py when running a generated Python parser. Traces are JSON and can be replayed after the parser exits. Live parser sessions currently require POSIX file descriptors; the Windows ispadbg executable supports offline --replay and help.

ISPA is self-hosting for its own grammar. The .isc grammar that describes ISPA's input language was compiled by ISPA into the parser that ISPA now uses to read grammars, replacing the original hand-written parser. A Python-subset parser and a small module-scanner parser are generated the same way.

The generated bootstrap parser is committed under parser/ispa/bootstrapped_0 and is built with ISPA before the generator processes other grammars.

An external import calls a separately generated parser and captures its original node type:

import extern python ispa_python as python; expr: @ python.expr {@};

Use --dep NAME PREFIX to select an existing parser. PREFIX has no extension; ISPA reads PREFIX.ispa-export for exported symbols and uses PREFIX.h as its public header for C++ output or PREFIX.py for Python output. For example:

ispa -l cpp -a LL --dir parser/ispa/grammar parser/ispa/entry.isc
--dep python parser/python/bootstrapped/PyRawParser -o build/Parser

The generated C++ includes <PyRawParser.h> and uses PyRawParser::Types directly. Add the dependency directory to the compiler's include path and link its implementation as a static or shared library. The host does not generate copies of the imported parser's productions or types. Export entry declarations live in the dependency header; their definitions live in its .cpp file.

With --no-dep, ISPA discovers external grammar modules in the supplied --dir directories and generates each dependency beside the host, using the external dependency name as its output prefix. C++ dependencies remain separate .h/.cpp files and must be compiled and linked by the build system. Nested dependencies are generated before their consumers; dependency cycles are diagnosed. Explicit --dep bindings may supply existing libraries while --no-dep builds the rest.

External parser calls currently require LL generation and lazy tokenization. An eager lexer rejects grammars containing external calls with an AMBIGUOUS diagnostic. Lazy calls re-lex the imported span with the target lexer so token payloads match the target parser, then resume the host lexer at the returned source position.

For the Python target, generate with -l python -a LL. Put the generated modules and parser/stdlibs/target on PYTHONPATH so ispa_std is available:

PYTHONPATH=build:parser/stdlibs/target python3 -c 'import Parser; print(Parser.Parser().parse("input"))'

Python external imports use the generated dependency module (for example, import Foreign) and its exported parser bindings. Keep dependencies on PYTHONPATH beside the host parser. --dep and --no-dep work as above.

Lex + parse of the same input, compared against flex + bison (LALR(1)) and ANTLR (C++ target). Lower is better; the multiplier is relative to flex + bison without an AST.

Flex + bison figures are steady state over 10 iterations.

Matches tokens with a compact TDFA (tagged DFA) table, which keeps captures and actions inside a single automaton

Builds AST-like tokens natively during lexing. Capture only what you need

Matches tokens inside other tokens without losing structure, preserving each capture's data type

Matches non-terminals inside terminals during lexing, separately from parsing (not yet implemented)

Parser Capabilities

LL is the production parser target and is actively growing; alternative LR-family modes currently support fewer grammar features and remain experimental

Builds a fully typed, easy-to-traverse AST during parsing. No post-parsing pass or callbacks required

Advanced rules and fail blocks

See the separate Advanced Rules and Fail Blocks guide.

Install ISPA from a release, install its standard library package, and enable the package exports for future shells:

ispa-pkg install std ispa-pkg --export-permanent

Open a new shell. The installed shell helper exposes the include paths and CMake targets; run ispa-pkg --export in that shell whenever you want to refresh them. Then create grammar.isc:

module main -> demo;

NUMBER: @ [0-9]+ {@}; main: @ NUMBER {@};

Generate the C++ parser:

ispa -l cpp -a LL grammar.isc -o generated/Parser

Use the generated parser in your application and link the installed targets through CMake. In CMakeLists.txt:

cmake_minimum_required(VERSION 3.20) project(demo LANGUAGES CXX) include("$ENV{ISPA_TARGETS_CMAKE}") add_executable(demo main.cpp generated/Parser.cpp) target_link_libraries(demo PRIVATE ISPA::Targets)

This target supplies the installed ISPA headers and runtime libraries. See the wiki for a complete installed-user walkthrough, including Python output.

Lexer/LR actions and dynamic regex in generated code

Grammar syntax diagnostics with LL recovery (further improvements ongoing)

PLL algorithm: finalize the custom Predictive LL algorithm to resolve left recursion in LL parsers while keeping structural parity with LR parsers

Multi-target code generation

Python target emitter for LL

Full LR parser stabilization: LR(1), LALR, and related modes

Modules: feature-gated modules, disable feature X, parametrized modules

Grammar standard library: JSON, XML, math expressions, C-like statements

Features described in concepts/

Install ISPA

ISPA has prebuilt GitHub Release archives for Linux x86_64, Windows x64, and macOS Apple Silicon (ARM64). macOS Intel is supported by the installer only if its release archive exists. All commands install for the current user, with no administrator privileges needed.

Release installers provide ispa-pkg and ispadbg alongside ispa. For a source checkout, install both Python CLIs with python3 -m pip install . (Python 3.9+ and PyYAML). Tagged releases also publish standalone Nuitka builds named ispa-pkg-<platform> and ispadbg-<platform> (.tar.gz, or .zip on Windows) for the four supported Linux, Windows, and macOS targets. The ISPA platform archives include both executables, which the installers and updaters activate. The CLI lives in cli/pkg. Libraries are installed under ~/.local/share/ispa/libraries by default on Linux and macOS, or under the ISPA installation directory when launched from an installed release; use ISPA_PKG_ROOT to choose another location for both commands (ispa-pkg --root changes only the package command). ispa searches the matching directory for grammar and import py libraries automatically. It also loads its bundled algorithm and converter plugins beside the executable. The installed package has ispalib/grammar, ispalib/python, ispatarget/<language>, and ispaalgorithm directories. When compiling generated target code, add its ispatarget/<language> directory to your compiler or interpreter search path. Installed external parser manifests (NAME.ispa-export beside NAME.h or NAME.py) also provide --dep bindings automatically. A package with one manifest can be referenced by its package name; an explicit --dep wins.

ispa-pkg install std ispa-pkg install owner/repository --ref v1.2.0 ispa-pkg install https://example.org/mylib.tar.gz --name mylib ispa-pkg update ispa-pkg list ispa-pkg catalog update ispa-pkg catalog list

Run ispa-pkg --export-permanent once, then open a new bash, zsh, or PowerShell session. This installs a small shell function and loads the exports on startup. In that shell, ispa-pkg --export refreshes the exports in the current session without an eval command. The standalone executable prints shell code when the function is not loaded, beca

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →