spec - lexer.rs

Referenced by7

Labels7

Idea	Status	Tags	Summary	Katz
lexer_bridge_to_syntax			The context-sensitive token builder is what makes [block_idea] and prose braces coexist: structural `{` / `}` are only those that open or close ideas, while [nested_curly_braces] stay body text. [context_sensitive_lexer_scaling] is the performance contract for that bridge; [code_snippets] is the opacity contract so fence interiors never feed the brace timeline. Index-path Rust port: ["../core_analysis/rust_port.rq".lexer_rust] / ["../core_analysis/rust_port.rq".parser_rust].	0.077
context_sensitive_lexer_scaling	done		Context-sensitive token classification for structural / prose braces and top-level import keywords must remain linear in document size. The lexer records brace-depth changes sparsely ( not per-character tables ) and looks them up in log time while tokenising, so workspace language-server initialization and editor re-lexes stay bounded. Code fences are skipped during the brace scan so example braces cannot inflate work or desync depth. Bounded work still assumes the lexer / parser returns; unbounded hangs are cut by ["./parser_lexer.rq".parse_budget_timeout].	0.097
fencing_comments	done		A complete same-line fence keeps `//` and `/` as prose, not comments. Closed double quotes: " / / this line " contains no comment. Closed backticks: `//this line` contains no comment. Closed backticks with a glob: `@reqlan/` does not open a block comment. An unclosed backtick is not a fence, so this line ` An unclosed quote is not a fence, so as does " The Langium lexer, the Rust lexer, comment scanners, and TextMate must use this rule. Related: ["./syntax.rq".comments], ["./syntax.rq".inline_code], ["./syntax.rq".naked_strings_in_body], ["../core_analysis/rust_port.rq".comment_span_align].	0.078
nested_curly_braces			if there is a prose block containing curly braces, { such as this one } they should be treated as part of the prose. Matching braces that end a line — including adjacent pairs like { { SITE_URL } } — stay in the body; they must not close the enclosing idea. A lone `}` on its own line still closes the structural block so an unbalanced prose `{` cannot swallow the idea closer.	0.108
attribute_namespacing	done		A bracket link `[idea.attribute]` targets one attribute of that idea. The first name is the idea name. The second name is the attribute name. The grammar types that name as an attribute or an idea declaration. `[attribute_namespacing.example]` targets the `example` attribute of this idea. The form `[ideaset.member]` still targets an ideaset member. See [references_to_subidea]. The editor opens the attribute when the first name is an idea. The editor reports an error when that idea has no attribute with the second name. The link target is only that attribute on that idea. HTML export uses the same link form. The export link opens that attribute on the idea page. See [attribute_namespacing.example]. Related: [reference_brackets], [attribute_marker], ["../core_analysis/html_export.rq".html_export_internal_links].	0.129
empty_attribute			A line that contains only @ is insignificant whitespace. Spaces or tabs may precede that @. Spaces or tabs may follow that @. The lexer consumes that @ as hidden whitespace. The parser does not build an attribute for that line. That line does not cause a parser error. That line does not cause a lexer error. That line does not cause a tokenizer error. That line does not cause an extension error. A flag keeps the form @ name. A negated flag keeps the form @ name!. A valued attribute keeps its value after the name. A syntax error on a different line is still reported.	0.040
idea-leading-whitespace			A block body can have spaces or tabs before the text. The lexer counts a space and a tab as whitespace. A tab is one whitespace character. The parser does not replace a tab with spaces. The base indent is the whitespace at the start of the first body line. The first body line is the first line of body text in the block. The first body line can start after column 0. The base indent is still the whitespace on that line. A body line with the same whitespace as the base indent has no leading whitespace in the parsed text. A body line keeps the whitespace that remains after the base indent. The parser removes the base indent only when the line starts with that same whitespace. An attribute on its own line stays an attribute. Example: the first body line has four spaces. A later line with four spaces has no leading whitespace. A later line with eight spaces has four spaces in the parsed text. This line has the same four spaces as the first body line. This line has eight spaces.	0.040

Reference labels

attribute_namespacing, context_sensitive_lexer_scaling, empty_attribute, fencing_comments, idea-leading-whitespace, lexer_bridge_to_syntax, nested_curly_braces