Code reference detail
reqlan-token-builder.ts
packages/language/src/reqlan-token-builder.ts
| Idea | Status | Tags | Summary | Katz |
|---|---|---|---|---|
| analytical_rust_port | in-progress | Port ["../core-architecture.rq".analytical_package] to Rust: lexer, parser ( index path ), extractor, SQLite index, search, and export. `@reqlan/analytical` remains the published JS facade ( `createAnalysisRuntime` / `AnalysisApi` / Zustand host state ). Native work lives in a `crates/` Cargo workspace and ships as per-architecture binaries per [distribution.rust_binary_distribution]. The editor language server is [lsp_rust.language_server_rust]. This analytical port does not replace that process. Component plans: [rust_crate_layout], [lexer_rust], [parser_rust], [parser_align], [indexer.indexer_rust], [search_rust], [export_rust], [native_bridge], [cutover]. | 0.243 | |
| comment_span_align | done | On every frozen [golden_corpus] `.rq` file, `//` and `/* … */` comment character ranges must be identical across: - the Langium lexer ( hidden `SL_COMMENT` / `ML_COMMENT` ) - `findCommentSpansInText` C-style spans ( host `#` / triple-quote spans are not `.rq` comments ) - TextMate `comment.line.reqlan` and `comment.block.reqlan` scopes Same-line fences, glob `/**/`, URL `://`, and fenced ` ` ` bodies must not open comments. Mid-line triple backticks are prose. They must not open a fence or hide later ideas. Example: `@reqlan/*` in inline code must not open a block comment. Related: [parser_align], ["../language/syntax.rq".code_snippets], ["../language/syntax.rq".comments], ["../language/syntax-edge-cases.rq".fencing_comments], ["../language/syntax.rq".inline_code]. | 0.141 | |
| lexer_rust | done | Hand-written Rust lexer that ports ["../../packages/language/src/reqlan-token-builder.ts"]: structural vs prose `{` / `}`, fence opacity, quote-aware comments, `@` only at line start, glob `/**/` not a block comment. Linear in document size with sparse brace-depth lookup ( [syntax_edges.context_sensitive_lexer_scaling] ). Owned by `reqlan-parse`; consumed by [parser_rust] and by [lsp_rust.language_server_rust]. The Langium token builder remains only for the transitional `node` engine fallback and dual-parser tests. | 0.105 | |
| import_keywords | The keywords import, from, and as are reserved for import statements. Import keywords may also appear as ordinary words in idea body text when not at statement start. | 0.047 | ||
| wildcard_references | done | Qualified path + idea-pattern references match multiple ideas across files. Surface forms: bracket `[…]` and wikilink `[[…]]` ( same `WildcardReference` target ). Path is a quoted string that may include shell-style globs ( `*`, `**`, `?` ) and `@/` import-root aliases ( [configuration_import_root_alias] ). Relative globs resolve against the defining document directory; `.` / `..` segments are collapsed before matching. Idea pattern is an unquoted `WILDCARD_NAME` that must contain at least one `*` or `?` Exact quoted-path plus idea stays ["./syntax.rq".reference_qualified]. Matching uses anchored globs ( not full JS regex ): path `*` is one segment, `**` crosses segments; idea `*` / `?` match within the idea name. Zero matches produce a warning diagnostic. One match also produces a warning — a wildcard is for many ideas; use a concrete ["./syntax.rq".reference_qualified] when only one idea matches ( ["../core_analysis/check.rq".check_wildcard_one] ). Path segment `/**/` must not be lexed as an empty block comment — see ["./syntax.rq".comments]. Lexer note: `ID` is a prefix of many wildcard names ( `import_` vs `import_*` ); the token builder sets `ID.LONGER_ALT = WILDCARD_NAME` so `import_*` is one token. Body prose `*` tokens are accepted via `WILDCARD_NAME` in PlainText so ordinary `*` characters still parse. Without that longer-alt, a live wildcard inside a block body can fail Idea recovery and leave later top-level ideas unindexed ( soft mtime sync will not rebuild until the `.rq` file changes ). Navigation does not emit N LocationLinks. A document-link click opens `reqlan.openWildcardReference` with `{ pathPattern, ideaPattern, fromUri }` ( see [wildcard_references_webview] ). Go-to-definition does not open that panel. Indexing fans out per ["../indexer/indexer.rq".wildcard_reference_edges] ( owned under ["../indexer/indexer.rq".index] ): native extract writes one `wildcard_reference` edge per catalog match; empty catalog / zero matches → one unresolved edge of that kind labeled with the pattern text. ["../core_analysis/check.rq".check] handles 0 - match and 1 - match fan-out through ["../core_analysis/check.rq".check_wildcard_zero] and ["../core_analysis/check.rq".check_wildcard_one] ( ["../core_analysis/check.rq".check_wildcard_sparse] ). Ideas Summary graph toggles those edges via ["../extension/workspace-summary/ideas_summary/graphical_graph.rq".wildcard_refs_toggle]. The server can answer `reqlan/wildcardReferenceAt`. Go-to-definition does not call it. Related: ["./syntax.rq".reference_embedding], ["./syntax.rq".reference_brackets], ["./syntax.rq".reference_wikilink], ["../indexer/indexer.rq".wildcard_reference_edges], ["../core_analysis/check.rq".check_wildcard_sparse], ["../core_analysis/check.rq".check_wildcard_zero], ["../core_analysis/check.rq".check_wildcard_one]. Illustrative forms ( fenced so this file does not embed live wildcards ): | 0.486 | |
| context_sensitive_lexer_scaling | done | Context-sensitive token classification for structural / prose braces and top-level import keywords must remain linear in document size. The lexer records brace-depth changes sparsely ( not per-character tables ) and looks them up in log time while tokenising, so workspace language-server initialization and editor re-lexes stay bounded. Code fences are skipped during the brace scan so example braces cannot inflate work or desync depth. Bounded work still assumes the lexer / parser *returns*; unbounded hangs are cut by ["./parser_lexer.rq".parse_budget_timeout]. | 0.097 | |
| fencing_comments | done | A complete same-line fence keeps `//` and `/*` as prose, not comments. Closed double quotes: " / / this line " contains no comment. Closed backticks: `//this line` contains no comment. Closed backticks with a glob: `@reqlan/*` does not open a block comment. An unclosed backtick is not a fence, so this line ` An unclosed quote is not a fence, so as does " The Langium lexer, the Rust lexer, comment scanners, and TextMate must use this rule. Related: ["./syntax.rq".comments], ["./syntax.rq".inline_code], ["./syntax.rq".naked_strings_in_body], ["../core_analysis/rust_port.rq".comment_span_align]. | 0.078 | |
| nested_curly_braces | if there is a prose block containing curly braces, { such as this one } they should be treated as part of the prose. Matching braces that end a line — including adjacent pairs like { { SITE_URL } } — stay in the body; they must not close the enclosing idea. A lone `}` on its own line still closes the structural block so an unbalanced prose `{` cannot swallow the idea closer. | 0.108 | ||
| code_snippets | Idea bodies and block attribute values may embed fenced code blocks delimited by triple backticks. An opening fence is three backticks as the first non-whitespace characters on a line, optionally followed by a language tag such as python or rq. A closing fence is three backticks as the first non-whitespace characters on a line. Three backticks in the middle of a line are prose. They do not open a fence. Everything between the fences is literal opaque text — it is not parsed as reqlan syntax. Keywords, imports, attributes, braces, and quotes inside a fence are inert for the parser and may be shown as examples without parse errors. Fenced blocks may span multiple lines and preserve their inner content verbatim. Brace-depth tracking must skip fence interiors so example braces cannot desync surrounding structure ( [no_name_idea_safe_warning] ). Related: ["../core_analysis/rust_port.rq".comment_span_align]. | 0.113 | ||
| comments | done | Line comments use `//` through end of line ( hidden `SL_COMMENT` ). Block comments use `/* … */` and may span multiple lines ( hidden `ML_COMMENT` ). Meta comments on the same line after content are allowed ( `keep this // meta` ). Comments must not open inside a string / reference-string context: - Naked body quotes ( ["naked_strings_in_body"] ): a quoted slash-slash that is not a comment, quoted urls, `"../mod/**/*.rq"`. - Reference / import `STRING` paths ( `["…"]`, `from "…" import …` ) — the lexer consumes the whole quoted path as one `STRING`, so neither `//` nor `/*` inside the quotes starts a comment. Example: a file reference whose path text contains slash-slash, `["../extension/**/*.rq".*webview*]`. - URL / path-segment guard: `//` immediately after `:` or `/` is not a line comment. Same-line fences: ["./syntax-edge-cases.rq".fencing_comments] — complete `"…"`, `'…'`, and ` ` ` … ` ` ` spans keep ` Unclosed quote or backtick openers are not fences, so `//` after them is a comment. Comment-range identity across lexer, scanner, and TextMate: ["../core_analysis/rust_port.rq".comment_span_align]. Multiline / glob clash: the empty block form `/**/` is **not** a comment — it is the recursive glob segment used in paths such as `../mod/**/*.rq`. A real block comment needs a non-empty body ( `/* note */`, `/*\n note\n*/` ). The token builder rejects length-4 `/**/` matches; comment-span scanners skip the same sequence so embedded `rq:[…]` discovery stays aligned. Related: [comment_reference_ignore], [string_literals], ["./imports.rq".wildcard_references], ["../core_analysis/rust_port.rq".comment_span_align]. | 0.145 | |
| empty_attribute | A line that contains only @ is insignificant whitespace. Spaces or tabs may precede that @. Spaces or tabs may follow that @. The lexer consumes that @ as hidden whitespace. The parser does not build an attribute for that line. That line does not cause a parser error. That line does not cause a lexer error. That line does not cause a tokenizer error. That line does not cause an extension error. A flag keeps the form @ name. A negated flag keeps the form @ name!. A valued attribute keeps its value after the name. A syntax error on a different line is still reported. | 0.040 | ||
| file_suffix | Reqlan requirement documents use the. rq file suffix. | 0.052 | ||
| no_name_idea_safe_warning | done | A top-level nameless curly-brace block — `{... }` with no idea name — must not take down the parser or the language server workspace. Parse the block as a recoverable anonymous top-level element, emit a descriptive warning that a name is required before `{`, and continue so later ideas, imports, and links still work. Nameless blocks inside parenthesized lists remain valid list items per [lists]; only top-level nameless blocks warn. Related recovery posture: ["./imports.rq".import_error_recovery]. Block shape: [block_idea]. Fenced examples must stay opaque per [code_snippets] so braces inside snippets cannot desync structural depth. Non-termination ( lex / parse never finishes ) is a different failure mode: ["./parser_lexer.rq".parse_budget_timeout]. | 0.072 | |
| reference_url | done | A bracket reference may target an unquoted absolute URL ( `scheme://…` ). Example: `[https://reqlan.com/]`. The URL is one token; `://` is not a line comment. Unbracketed URLs in body prose stay text. Quoted remote paths stay [reference_file] ( `["https://example.com/file.ts"]` ). Index and check: ["../reference_types.rq".url_reference]. | 0.137 | |
| grammar_rule | done | Grammar rules describe the syntax features. Requirements: ["./language/syntax.rq"]; lexer bridge: ["./language/parser_lexer.rq".lexer_bridge_to_syntax]. Langium: [reqlan_grammar]; comment grammar: [comment_grammar]; index-path AST: [rust_ast]. | 0.040 | |
| idea | done | A reqlan core unit is an idea. Syntax: ["./language/syntax.rq".simple_idea], ["./language/syntax.rq".block_idea]. Grammar: [reqlan_grammar] `Idea` / `OneLinerIdea`; index records: [analytical_types] `IdeaRecord`. Extract: [idea_extractor], [rust_extract]; AST: [rust_ast]. Product surface: [ideas_summary_tabs] Ideas tab per ["./extension/workspace-summary/ideas_summary/webview.rq".ontology_aligned_tabs]. | 0.110 | |
| idea_reference | done | Idea-to-idea ["./ontology.rq".reference]: bracket `[name]` / `[ideaset.name]` targets a local, imported, or same-file idea. Syntax: ["./language/syntax.rq".reference_brackets], ["./language/syntax.rq".same_file_reference], ["./language/syntax.rq".reference_qualified], ["./language/syntax.rq".reference_embedding], ["./language/syntax.rq".references_to_subidea]. Index edge kind `references` on [analytical_types] / [rust_index_types]. Extract: [idea_extractor], [rust_extract]. A `[ideaset.member]` form records the edge on the member leaf, not on the ideaset container. An attribute uses the same brackets. See ["./language/syntax.rq".attribute_namespacing]. Wikilink surface for the same edge kind: [link_reference]. Fan-out across files: [wildcard_reference]. | 0.125 | |
| reference_edgecase | done | A quoted string is a [file_reference] only when it is inside brackets. Example of a file reference: ["./reference_types.rq"] Example of body text: " this is not a reference. rq " The file suffix does not change this rule. The parser must not make a file reference from quoted body text. The language server must not make a document link or a missing-file error from quoted body text. Related: [file_reference], ["./language/syntax.rq".naked_strings_in_body], ["./language/syntax.rq".reference_file]. | 0.112 | |
| url_reference | done | A bracket reference whose target is an absolute URL with a scheme and `://`. Example: [https://reqlan.com/]. Syntax: ["./language/syntax.rq".reference_url]. Index edge kind `url_reference` on [analytical_types] / [rust_index_types]. Extract: [idea_extractor], [rust_extract]. [check] does not treat a URL reference as a missing idea or a missing file. Unbracketed URLs in body prose stay text, not this kind. Related: [file_reference], [link_reference]. | 0.075 |
Reference labels
"../../packages/language/src/reqlan-token-builder.ts", "../packages/language/src/reqlan-token-builder.ts", ../../packages/language/src/reqlan-token-builder.ts, ../packages/language/src/reqlan-token-builder.ts