youmuliu/stgsharp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

remove yesterday's generatoer, use new generator

Browse Source
This commit is contained in:
Youmu Liu
2026-07-29 18:10:45 +08:00
parent 3db2bfd9af
commit 97576404cc
4 changed files with 328 additions and 132 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/StgSharp.RegularAnalysis/Text/TextRegexSourceGen.GenClass.MultiIR.cs
+195 -103
View File
@@ -14,109 +14,6 @@ namespace StgSharp.RegularAnalysis.Text
{
internal partial class TextRegexSourceGen
{
/*
* =======================================================================================
* L1, several regions plus a tail
* =======================================================================================
*
* Highest consumption in the dispatch chain, so these are offered the window first. Each
* shape here was already recognised once during prefix emission, so a generator should
* mirror the matching RegexAnalyzer.PrefixEmit recogniser rather than re-derive the shape.
*
* L1.2 ALT chain with a shared suffix
* window L1.1 followed by linear instructions in the parent region
* sample (ab|[cd]|.)x
* nature flattening, may return -1
*
* L1.3 group tail
* window region, POP FAIL with Prev_Fail, PACK SUCCESS with Prev_Success
* mirrors TryEmitTryGroupTail
* shape region block, capture written on the success branch
* sample (?<name>abc)
* nature flattening, may return -1
*
* L1.4 group tail wrapping an ALT chain
* sample (?<a>abcd|a(bc)c)
* nature flattening, may return -1
*
* L1.5 COUNT_COMPLEX plus body
* window RegexCountComplexIR, then CommandCount entries
* mirrors TryEmitComplexIR
* shape loop wrapping the generated body
* sample (ab){2,3}
* nature REQUIRED, no fallback exists
*
* L1.6 COUNT_COMPLEX whose body contains an ALT chain, plus a linear suffix
* sample (123(4|5)67){3,4}114514
* nature REQUIRED, no fallback exists
*
* L1.7 FIND_COMPLEX plus body
* window RegexFindComplexIR, then CommandCount entries
* shape search loop wrapping the generated body
* sample .*(ab|cd)
* nature REQUIRED, no fallback exists
*
* L1.5 to L1.7 are required because a complex body is not a region of its own. L2 cannot
* pick it up, so there is no degradation path if these are missing.
*
* CommandCount must be read from the prefixed IR. Prefix emission rebuilds these counts,
* so the value carried by the pre-prefix IR no longer applies.
*
* Open questions to settle while implementing:
* - how the cursor is restored between ALT cases
* - which successful outcome a PACK uses as its capture span
* - whether a region body ever deserves its own generated method instead of inlining
*
* =======================================================================================
* L2, one region
* =======================================================================================
*
* Consumes CommandCount + 1, the whole block a TRY lowers to. Together with L4 and L5 this
* is the correctness core: once these three exist, every expression can be generated, just
* with deeper nesting than L1 and L3 would produce.
*
* L2.1 RegexTryIR plus its body sample (abc)
* L2.2 the same, body containing a nested region sample (a(bc))
* L2.3 a single instruction region folded by prefix emission sample (a)
*
* Target shape, names owned by SourceGenContext:
*
* __rN_begin = __cur;
* __rN_ok = false;
* do
* {
* // body, a POP breaks out here
* __rN_ok = __ok;
* } while (false);
* if (!__rN_ok) { __cur = __rN_begin; }
* __ok = __rN_ok;
*
* Points the implementation has to settle:
* - the failure path restores the cursor to the region entry
* - break only leaves the nearest enclosing do block, so a nested region must not rely
* on it to exit an outer one
* - the whole match body is itself region zero, which is what makes a top level POP
* legal
*
* =======================================================================================
* L3, fixed multi instruction windows
* =======================================================================================
*
* Consumes two to eight entries. Pure code quality work, never required for correctness.
* Every one of these must return -1 cleanly when the window does not match, so the entry
* falls through to L4.
*
* L3.1 FIND_COMPLEX FURTHEST COUNT:-1, PACK, MATCH_SEQ 3 (?<name>.*)234
* L3.2 the same with a single element anchor 3 (?<name>.*)a
* L3.3 FIND_SEQ, MATCH_SEQ, merged into one search 2 .*abc
* L3.4 MATCH_SEQ, COUNT_SEQ, the expanded a+ shape 2 a{3,5}, a+
* L3.5 a run of MATCH_SEQ and MATCH_SET, merged into one condition 2 to 8 a[abc]d
* L3.6 trailing COUNT_SET over a dot, fast success 1 to 2 abc.*
*
* Two hazards to check for each of these:
* - merging must not break the line indices an Observe_Line prefix refers to
* - a merged window still has to leave __ok describing the whole window
*/
private static int TryGenerateAltChain(
int outer_level,
@@ -165,6 +62,201 @@ namespace StgSharp.RegularAnalysis.Text
}
return i;
}
/*
* =======================================================================================
* L1, several regions plus a tail
* =======================================================================================
*
* Highest consumption in the dispatch chain, so these are offered the window first. Each
* shape here was already recognised once during prefix emission, so a generator should
* mirror the matching RegexAnalyzer.PrefixEmit recogniser rather than re-derive the shape.
*
* L1.2 ALT chain with a shared suffix
* window L1.1 followed by linear instructions in the parent region
* sample (ab|[cd]|.)x
* nature flattening, may return -1
*
* L1.3 group tail
* window region, POP FAIL with Prev_Fail, PACK SUCCESS with Prev_Success
* mirrors TryEmitTryGroupTail
* shape region block, capture written on the success branch
* sample (?<name>abc)
* nature flattening, may return -1
*
* L1.4 group tail wrapping an ALT chain
* sample (?<a>abcd|a(bc)c)
* nature flattening, may return -1
*
* L1.5 COUNT_COMPLEX plus body
* window RegexCountComplexIR, then CommandCount entries
* mirrors TryEmitComplexIR
* shape loop wrapping the generated body
* sample (ab){2,3}
* nature REQUIRED, no fallback exists
*
* L1.6 COUNT_COMPLEX whose body contains an ALT chain, plus a linear suffix
* sample (123(4|5)67){3,4}114514
* nature REQUIRED, no fallback exists
*
* L1.7 FIND_COMPLEX with CommandCount >= 0
* window RegexFindComplexIR, then CommandCount entries
* shape candidate retry loop wrapping the body
* sample .*(ab|cd)
* nature REQUIRED, no fallback exists
*
* L1.8 FIND_COMPLEX with CommandCount == -1
* window RegexFindComplexIR, then the rest of the region
* shape candidate retry loop wrapping the rest of the region
* sample (?<name>.*)234
* nature REQUIRED, no fallback exists
*
* L1.9 FIND_SEQ or FIND_SET
* window the instruction, then the rest of the region
* shape candidate retry loop wrapping the rest of the region
* sample .*abc
* nature REQUIRED, no fallback exists
*
* L1.5 to L1.9 are required because none of their bodies is a region of its own. L2
* cannot pick them up, so there is no degradation path if these are missing.
*
* CommandCount must be read from the prefixed IR. Prefix emission rebuilds these counts,
* so the value carried by the pre-prefix IR no longer applies.
*
* ---------------------------------------------------------------------------------------
* Backtracking semantics of the FIND family
* ---------------------------------------------------------------------------------------
*
* A FIND does not yield one position, it yields a priority ordered sequence of candidates.
* FURTHEST starts at the last one and steps backwards on failure, NEAREST starts at the
* first and steps forwards. The body is the verifier for a candidate: when it fails, the
* next candidate has to be tried, and only an exhausted candidate list is a real failure.
* A single LastIndexOf is wrong, because .*abcX has to fall back to the second to last abc
* when the last one is not followed by an X.
*
* CommandCount == -1 means the body is the rest of the region. TryReplaceDotStar emits
* FindComplex(-1) for a bare .*, then TyrMergeFindRestTwice and TyrMergeFindRestTry try to
* collapse it into a definite length. Whatever survives stays at -1. FIND_SEQ and FIND_SET
* carry no CommandCount at all and behave like -1. Only a FIND that happens to be the last
* instruction of its region degrades into a single search, consuming one entry.
*
* Target shape:
*
* int __bound = text.Length;
* __ok = false;
* while (true)
* {
* int __at = text[__cur..__bound].LastIndexOf("abc");
* if (__at < 0) { __ok = false; break; } // candidates exhausted, only failure
* int __save = __cur;
* __cur = __cur + __at + 3;
*
* // body, or the rest of the region, expands here
*
* if (__ok) { break; } // verified
* __cur = __save;
* __bound = __save + __at; // step to the previous candidate
* }
*
* POP conflicts with the retry loop. A POP lowers to break, and the retry while is nearer
* than the region do block, so the break would leave the wrong scope. The fix without goto
* is a region completion flag:
*
* do // region N
* {
* bool __rN_done = false;
* while (!__rN_done) // find retry
* {
* // candidate search
* do // isolation layer, POP breaks out here
* {
* // body, POP sets __rN_ok, sets __rN_done, breaks
* } while (false);
* if (__rN_done || __ok) { break; }
* // step to the next candidate
* }
* } while (false);
*
* Two further hazards:
* - nested FIND produces nested retry loops and the cost multiplies, which is the
* catastrophic backtracking shape, worth a guard or at least a marker
* - __ok is overwritten across retry iterations, so an Observe_Line prefix pointing into
* a retry body reads the value of the last iteration, a case the current prefix model
* does not cover
*
* Open questions to settle while implementing:
* - how the cursor is restored between ALT cases
* - which successful outcome a PACK uses as its capture span
* - whether a region body ever deserves its own generated method instead of inlining
*
* =======================================================================================
* L2, one region
* =======================================================================================
*
* Consumes CommandCount + 1, the whole block a TRY lowers to. Together with L4 and L5 this
* is the correctness core: once these three exist, every expression can be generated, just
* with deeper nesting than L1 and L3 would produce.
*
* L2.1 RegexTryIR plus its body sample (abc)
* L2.2 the same, body containing a nested region sample (a(bc))
* L2.3 a single instruction region folded by prefix emission sample (a)
*
* Target shape, names owned by SourceGenContext:
*
* __rN_begin = __cur;
* __rN_ok = false;
* do
* {
* // body, a POP breaks out here
* __rN_ok = __ok;
* } while (false);
* if (!__rN_ok) { __cur = __rN_begin; }
* __ok = __rN_ok;
*
* Points the implementation has to settle:
* - the failure path restores the cursor to the region entry
* - break only leaves the nearest enclosing loop, and a FIND retry while counts as one,
* so a region containing a FIND needs the __rN_done flag described above instead of a
* plain break
* - a nested region must not rely on break to exit an outer one
* - the whole match body is itself region zero, which is what makes a top level POP
* legal
*
* =======================================================================================
* L3, fixed multi instruction windows
* =======================================================================================
*
* Consumes two to eight entries. Pure code quality work, never required for correctness.
* Every one of these must return -1 cleanly when the window does not match, so the entry
* falls through to L4.
*
* L3.1 FIND_COMPLEX FURTHEST COUNT:-1, PACK, MATCH_SEQ 3 (?<name>.*)234
* L3.2 the same with a single element anchor 3 (?<name>.*)a
* L3.3 FIND_SEQ, MATCH_SEQ, merged into one search 2 .*abc
* L3.4 MATCH_SEQ, COUNT_SEQ, the expanded a+ shape 2 a{3,5}, a+
* L3.5 a run of MATCH_SEQ and MATCH_SET, merged into one condition 2 to 8 a[abc]d
* L3.6 trailing COUNT_SET over a dot, fast success 1 to 2 abc.*
*
* Two hazards to check for each of these:
* - merging must not break the line indices an Observe_Line prefix refers to
* - a merged window still has to leave __ok describing the whole window
*/
private static int TryGenerateTryBlock(
int outer_level,
ReadOnlySpan<RegexIR> ir_span,
SourceGenContext sc,
in SequenceEmitter<string> code,
in List<SequenceEmitter<string>> local_func
)
{
if (ir_span[0]is not RegexTryIR _try) {
return -1;
}
ReadOnlySpan<RegexIR> block = ir_span[1..(1 + _try.CommandCount)];
// TODO generate here
return _try.CommandCount;
}
}
}
src/StgSharp.RegularAnalysis/Text/TextRegexSourceGen.GenClass.SingleIR.cs
+11 -6
View File
@@ -44,11 +44,16 @@ namespace StgSharp.RegularAnalysis.Text
* L4.5 MATCH_SET, any element .
* L4.6 COUNT_SEQ a*, a?
* L4.7 COUNT_SET [abc]*, \w+, .{11,13}, [.]*
* L4.8 FIND_SEQ degradation path of .*abc
* L4.9 FIND_SET -
* L4.10 POP ALT tail
* L4.11 PACK (abc), ordinal group
*
* L4.8 and L4.9 were removed. FIND_SEQ and FIND_SET have no single instruction form,
* because they have to drive a candidate retry loop over the rest of the region. See L1.9
* in TextRegexSourceGen.GenClass.MultiIR.cs. Only a FIND that happens to be the last
* instruction of its region degrades into a single search, and that is a branch of L1.9
* rather than a generator of its own. The numbers are left vacant so existing references
* do not shift.
*
* Semantics worth pinning down before writing these:
*
* - [.] and . are different sets. The former is a literal dot, the latter is any
@@ -56,10 +61,10 @@ namespace StgSharp.RegularAnalysis.Text
* - \w has to keep full .NET semantics. TextRegex.IsCharWord is inherited by the
* generated class and already carries them, so the generated code can call it
* directly instead of approximating with an ASCII test.
* - This backend does not backtrack. A lazy repetition therefore consumes exactly its
* lower bound and leaves the rest to the following anchor. Lazy forms needing more
* are expected to have been rewritten into a find by the IR optimizer, so confirm
* that before adding backtracking here.
* - Repetition in this layer does not backtrack. A lazy repetition therefore consumes
* exactly its lower bound and leaves the rest to the following anchor. Backtracking
* does exist in this backend, but only inside the FIND retry loop at L1.7 to L1.9,
* never here.
* - An epsilon sequence and an empty repetition unit both reach these generators.
* Decide whether they are vacuous successes or an optimizer bug worth reporting.
* - PACK decides nothing, so it should leave __ok untouched. Whether it runs at all is
src/StgSharp.RegularAnalysis/Text/TextRegexSourceGen.GenClass.cs
+18 -4
View File
@@ -48,9 +48,10 @@ namespace StgSharp.RegularAnalysis.Text
* 2. The dispatch chain offers a window to L1, then L2, then L3, then L4. Higher
* consumption wins, so a shape spanning a whole region plus a tail is never split by
* a smaller generator.
* 3. L2, L4 and L5 together are sufficient for correctness. L1 and L3 only flatten what
* those would already emit, so an unimplemented one must return -1 cleanly rather
* than emit a partial result.
* 3. L2, L4, L5 and the required entries of L1 together are sufficient for correctness.
* The flattening entries of L1 and all of L3 only improve the shape of what those
* would already emit, so an unimplemented one must return -1 cleanly rather than
* emit a partial result.
* 4. A prefix consumes no IR. L1 to L4 describe what an instruction does, L5 decides
* when it runs and who reads the outcome.
* 5. Scope follows Prefix.WorkingRegion, not expression nesting.
@@ -60,6 +61,10 @@ namespace StgSharp.RegularAnalysis.Text
* A region lowers to do { ... } while(false), and POP lowers to break. The goto plus
* label scheme used by the .NET generated regex backend is deliberately not adopted.
*
* A FIND lowers to a candidate retry loop nested inside the region block. That loop is
* nearer than the region do block, so a POP inside it needs the __rN_done flag described
* in TextRegexSourceGen.GenClass.MultiIR.cs rather than a plain break.
*
* Generated variable names are owned by SourceGenContext. No generator invents one.
*
* The per shape checklist this backend is rebuilt against lives in
@@ -90,7 +95,6 @@ namespace StgSharp.RegularAnalysis.Text
// TODO L0.1 Emit the Match signature from ROS_char and SourceGenContext.InputSpan.
// TODO L0.2 Emit region anchor and region result slots, count from RegexInfo.RegionCount.
// TODO L0.3 Emit line result slots, only for lines an Observe_Line prefix refers to.
// TODO L0.4 Emit the whole pattern guard from RegexInfo.MinPredictLength.
// TODO L0.5 Emit the cursor, the step result and any scratch declaration the context collected.
//
@@ -121,6 +125,7 @@ namespace StgSharp.RegularAnalysis.Text
{
// GenerateRegion(region, sc, match_e, sub_match_list);
_ = TryGenerateAltChain(0, region, sc, match_e, sub_match_list);
_ = TryGenerateTryBlock(0, region, sc, match_e, sub_match_list);
if (end >= ir_sp.Length)
{
break;
@@ -133,6 +138,15 @@ namespace StgSharp.RegularAnalysis.Text
_ = se.AppendLine($@"public override MatchResult Match({ROS_char} text)")//
.AppendLine(@"{");
int min_predict_length = info.MinPredictLength;
if (min_predict_length > 0) {
_ = se.AppendLine($@"if (text.Length < {min_predict_length})")
.AppendLine(@"{")
.AppendLine(@"return default;")
.AppendLine(@"}")
.AppendLine();
}
int region_count = info.RegionCount;
for (int i = 0; i < region_count; i++)
{