text_prefix_skip

Katz 0.092

Before any index content read, inspect at most the first 8 KiB.
Skip the file when that prefix has a NUL, a UTF-16 BOM ( `FF FE` or `FE FF` ), or invalid UTF-8.
A trailing incomplete UTF-8 code unit is not a skip: the rest of the file may finish the character.
Skip when the file is larger than 1 MiB.
A rejected prefix is not followed by a read of the remainder.
`.rq` files and allowlisted code files use the same check.
This is in addition to binary_ignore extension globs.

implementation

text.rs, sync.rs

status

done

tests

text.rs:nul_rejects, text.rs:utf16_bom_rejects, text.rs:invalid_utf8_rejects, text.rs:trailing_incomplete_utf8_is_not_a_reject, scan.rs:prefix_skip_does_not_index_non_text

Interactive page: text_prefix_skip