stop all work on text regex, ngra text regex is now frozen
This commit is contained in:
@@ -34,12 +34,24 @@ Numerics and linear algebra built on top of the core library:
|
||||
|
||||
Nitload General Regular Analysis (NGRA) infrastructure for recognizing and processing regular languages:
|
||||
|
||||
- **Regex/Automata Core**: Regular expression processing and automata utilities
|
||||
- **NFA Direction**: Planned generic ε-NFA sequence execution, followed by weighted automata and NGRA-Tensor research
|
||||
- **Parsing Primitives**: Reusable tokenizer and grammar building blocks
|
||||
- **Analysis Pipeline**: Structured processing for language-like inputs
|
||||
|
||||
NGRA is a foundation for higher-level language tooling. It does not replace `Nitload.Script`; the planned Script implementation depends on NGRA and has not started yet. The source currently under `Nitload.Script` is retained from the earlier EXPRESS implementation and is not part of the active core solution.
|
||||
|
||||
The former Text Regex parser, matcher, and source generator are [archived](src/Nitload.RegularAnalysis/Legacy/README.md)
|
||||
and excluded from compilation. Generic AST and token infrastructure remain active.
|
||||
|
||||
Although Text Regex development is frozen, it produced valuable reusable results: the five-link
|
||||
AST model, tree normalization and merging techniques, token/analysis infrastructure, and practical
|
||||
experience with COUNT continuations, group boundaries, and rollback. These results remain available
|
||||
to inform the new NFA architecture.
|
||||
|
||||
Future string-regex functionality will use .NET `GeneratedRegex`. NGRA will no longer pursue
|
||||
custom `Span<char>` or SIMD optimizations for string matching; its research focus is generalized
|
||||
sequence analysis and, later, weighted relation networks.
|
||||
|
||||
### Nitload.Graphics
|
||||
|
||||
Managed graphics infrastructure built on Common and Mathematics:
|
||||
|
||||
@@ -0,0 +1,916 @@
|
||||
# NGRA 转型计划
|
||||
|
||||
> 面向本地 Codex / 项目维护者的快速架构说明
|
||||
> 项目:Nitload General Regular Analyzer(NGRA)
|
||||
|
||||
## 1. 转型结论
|
||||
|
||||
NGRA 不再以“高性能字符串正则表达式引擎”为主要目标。
|
||||
|
||||
原先围绕 `Span<char>`、专用字符串匹配、复杂回溯、Source Generator 等方向继续投入的收益已经明显下降。后续核心工作转向:
|
||||
|
||||
1. 以 **ε-NFA / NFA** 为基础执行模型;
|
||||
2. 将输入从 `char` 推广为任意可比较、可判定类别的序列元素;
|
||||
3. 面向 Token Stream、语法前端、半上下文模式分析等场景;
|
||||
4. 支持自动机归并、分层自动机、状态/断言/前序引用;
|
||||
5. 在此基础上继续研究 **Weighted Automaton / NGRA-Tensor**。
|
||||
|
||||
因此,Text Regex 从“主架构”降级为“兼容前端 / 参考实现 / 测试子系统”。
|
||||
|
||||
---
|
||||
|
||||
## 2. 当前代码处理原则
|
||||
|
||||
### 2.1 保留现有 Text Regex 半成品
|
||||
|
||||
当前 Text Regex Source Generator、COUNT/CONCAT/ALT lowering、生成式 matcher 等代码不删除。
|
||||
|
||||
原因:
|
||||
|
||||
- 五角星 AST 模型仍然是有效成果;
|
||||
- AST 搜索、规范化、旋转、合并等代码仍可复用;
|
||||
- 当前实现记录了 COUNT continuation、group boundary、rollback 等真实工程问题;
|
||||
- 可作为未来自动机 IR 与旧实现行为的对照;
|
||||
- 删除会丢失大量探索成果。
|
||||
|
||||
建议:
|
||||
|
||||
- 将旧生成器移动到 `Legacy` / `Experimental` / `Archive` 目录;
|
||||
- 通过 `.csproj` 的 `Compile Remove` 或独立非构建项目,使其暂时不参与主编译;
|
||||
- 不再优先修补完整 `.NET Regex` 兼容性。
|
||||
|
||||
### 2.2 五角星 AST 继续保留
|
||||
|
||||
现有节点关系:
|
||||
|
||||
```text
|
||||
Parent
|
||||
↑
|
||||
Previous ← Node → Next
|
||||
↙ ↘
|
||||
Left Right
|
||||
```
|
||||
|
||||
继续作为通用语法树 / 结构树基础设施。
|
||||
|
||||
需要坚持:
|
||||
|
||||
- AST 用于表达语法和结构;
|
||||
- 自动机 / CFG / Relation Graph 用于表达执行与传播;
|
||||
- 不再要求 AST 本身承担全部 backtracking / continuation 语义。
|
||||
|
||||
---
|
||||
|
||||
## 3. 新主线:ε-NFA
|
||||
|
||||
第一阶段优先实现一个简单、正确、无性能野心的 ε-NFA 模拟器。
|
||||
|
||||
### 3.1 基础状态模型
|
||||
|
||||
```text
|
||||
Current Active States
|
||||
↓
|
||||
ε-closure
|
||||
↓
|
||||
consume(input)
|
||||
↓
|
||||
Next Active States
|
||||
↓
|
||||
ε-closure
|
||||
```
|
||||
|
||||
核心状态:
|
||||
|
||||
```text
|
||||
State
|
||||
Transition
|
||||
StartState
|
||||
AcceptStates
|
||||
ActiveStateSet
|
||||
```
|
||||
|
||||
Transition 分为:
|
||||
|
||||
```text
|
||||
ε transition
|
||||
matched transition
|
||||
```
|
||||
|
||||
输入类型不能写死为 `char`。
|
||||
|
||||
建议:
|
||||
|
||||
```csharp
|
||||
EpsilonNfa<TInput>
|
||||
```
|
||||
|
||||
或:
|
||||
|
||||
```csharp
|
||||
EpsilonNfa<TScalar>
|
||||
```
|
||||
|
||||
匹配条件可以抽象为:
|
||||
|
||||
```csharp
|
||||
bool Match(in TInput input)
|
||||
```
|
||||
|
||||
或者后续升级为更明确的 Predicate / Category Matcher。
|
||||
|
||||
### 3.2 第一阶段支持的 Pattern
|
||||
|
||||
第一版只要求:
|
||||
|
||||
```text
|
||||
literal / predicate
|
||||
concat
|
||||
alternation
|
||||
*
|
||||
+
|
||||
?
|
||||
{m,n}
|
||||
ordinary grouping
|
||||
```
|
||||
|
||||
暂不追求:
|
||||
|
||||
```text
|
||||
capture rollback
|
||||
复杂 greedy / lazy 行为
|
||||
lookaround
|
||||
backreference
|
||||
完整 .NET Regex compatibility
|
||||
```
|
||||
|
||||
其中 `{m,n}` 可以优先在构造期展开为 NFA 结构,而不是运行时维护复杂 COUNT 控制流。
|
||||
|
||||
---
|
||||
|
||||
## 4. 为什么以 NFA 为主,而不是 DFA
|
||||
|
||||
DFA 仍然有价值,但不再是必须经过的阶段。
|
||||
|
||||
核心路线:
|
||||
|
||||
```text
|
||||
Pattern
|
||||
↓
|
||||
ε-NFA
|
||||
↓
|
||||
NFA execution
|
||||
↓
|
||||
Weighted NFA
|
||||
↓
|
||||
NGRA-Tensor
|
||||
```
|
||||
|
||||
DFA 只作为可选 specialization:
|
||||
|
||||
```text
|
||||
Boolean
|
||||
Static
|
||||
No dynamic relation
|
||||
No weighted state
|
||||
```
|
||||
|
||||
满足这些条件的局部子图,可以由编译器确定化为 DFA fast path。
|
||||
|
||||
总体原则:
|
||||
|
||||
> NFA 是主语义模型,DFA 是可选优化后端。
|
||||
|
||||
原因:
|
||||
|
||||
普通 NFA 的活动状态:
|
||||
|
||||
```text
|
||||
a_i ∈ {0,1}
|
||||
```
|
||||
|
||||
加权机可自然推广为:
|
||||
|
||||
```text
|
||||
a_i ∈ R
|
||||
```
|
||||
|
||||
Tensor 版本可以继续推广为:
|
||||
|
||||
```text
|
||||
a_i ∈ R^d
|
||||
```
|
||||
|
||||
多状态同时活跃不再是需要“消除的非确定性”,而会成为模型本身的状态表示。
|
||||
|
||||
---
|
||||
|
||||
## 5. NGRA 的目标:广义序列分析
|
||||
|
||||
NGRA 不再局限于字符串。
|
||||
|
||||
典型输入:
|
||||
|
||||
```text
|
||||
char
|
||||
token
|
||||
syntax node
|
||||
semantic node
|
||||
sensor frame
|
||||
event
|
||||
instruction-like object
|
||||
```
|
||||
|
||||
Pattern 的作用也从:
|
||||
|
||||
```text
|
||||
“这个字符串是不是符合 regex”
|
||||
```
|
||||
|
||||
扩展为:
|
||||
|
||||
```text
|
||||
“这个序列在当前状态、历史、类别和上下文条件下,
|
||||
是否满足某个结构化模式”
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 半上下文 Token Analyzer
|
||||
|
||||
NGRA 的近期目标之一,是成为一种介于普通 Regex 与完整 Parser 之间的 Token Stream Analyzer。
|
||||
|
||||
可用能力包括:
|
||||
|
||||
```text
|
||||
Token 类别匹配
|
||||
Token 属性匹配
|
||||
前序状态引用
|
||||
局部断言
|
||||
Pattern 状态
|
||||
符号绑定
|
||||
分层子自动机
|
||||
局部上下文
|
||||
```
|
||||
|
||||
因此它适合作为:
|
||||
|
||||
```text
|
||||
Lexer 后端
|
||||
Token Regex
|
||||
语法前端
|
||||
编译器预分析器
|
||||
DSL Parser Frontend
|
||||
结构化流分析器
|
||||
```
|
||||
|
||||
“半上下文”的含义不是严格的形式语言分类,而是:
|
||||
|
||||
> Pattern 不只依赖当前 token,也可以依赖有限的已知状态、绑定、前序结果和结构信息,但不要求一开始就构造完整通用 Parser。
|
||||
|
||||
---
|
||||
|
||||
## 7. 自动机归并
|
||||
|
||||
大量 Pattern 应优先在 NFA 层归并,而不是各自独立执行。
|
||||
|
||||
例如:
|
||||
|
||||
```text
|
||||
ε → Pattern A
|
||||
/
|
||||
GlobalStart ─ε → Pattern B
|
||||
\
|
||||
ε → Pattern C
|
||||
```
|
||||
|
||||
Accepting State 可以附带:
|
||||
|
||||
```text
|
||||
PatternId
|
||||
MatchKind
|
||||
Payload
|
||||
```
|
||||
|
||||
后续目标:
|
||||
|
||||
- 合并公共路径;
|
||||
- 消除冗余 ε 边;
|
||||
- 状态压缩;
|
||||
- bitset active state;
|
||||
- pattern priority;
|
||||
- 多层自动机组合。
|
||||
|
||||
---
|
||||
|
||||
## 8. 分层自动机
|
||||
|
||||
后续重点不是继续增加字符串 Regex 特性,而是建立分层自动机。
|
||||
|
||||
示意:
|
||||
|
||||
```text
|
||||
Scalar / Token Layer
|
||||
↓
|
||||
Local Pattern Layer
|
||||
↓
|
||||
Phrase / Region Layer
|
||||
↓
|
||||
Sentence / Structure Layer
|
||||
↓
|
||||
Semantic / Relation Layer
|
||||
```
|
||||
|
||||
低层自动机可以生成高层节点,高层节点继续作为新序列输入。
|
||||
|
||||
即:
|
||||
|
||||
```text
|
||||
Input Stream
|
||||
↓
|
||||
Automaton L0
|
||||
↓
|
||||
Derived Nodes
|
||||
↓
|
||||
Automaton L1
|
||||
↓
|
||||
Higher-level Nodes
|
||||
```
|
||||
|
||||
这应成为 NGRA 进入编译器前端、自然语言结构分析和复杂流分析的主路径。
|
||||
|
||||
---
|
||||
|
||||
## 9. Assertion / Context 的重新定义
|
||||
|
||||
不再围绕字符串 Regex 的 lookahead / lookbehind 设计核心架构。
|
||||
|
||||
通用定义应更接近:
|
||||
|
||||
```text
|
||||
Assert(Context) -> bool
|
||||
```
|
||||
|
||||
或者:
|
||||
|
||||
```text
|
||||
RelationGate(State, Context, History) -> bool
|
||||
```
|
||||
|
||||
因此:
|
||||
|
||||
- 字符串前瞻/后顾只是特殊情况;
|
||||
- 前序引用只是状态引用的一种;
|
||||
- group/capture 不是核心抽象;
|
||||
- Context / Binding / Region / History 才是可推广概念。
|
||||
|
||||
---
|
||||
|
||||
# NGRA-Tensor 构想
|
||||
|
||||
## 10. 基本方向
|
||||
|
||||
NGRA-Tensor 不是简单地“给 NFA 边加 float”。
|
||||
|
||||
它希望把:
|
||||
|
||||
```text
|
||||
Boolean multi-state automaton
|
||||
```
|
||||
|
||||
推广为:
|
||||
|
||||
```text
|
||||
Weighted / Tensor Relation Network
|
||||
```
|
||||
|
||||
基础演化:
|
||||
|
||||
```text
|
||||
NFA
|
||||
a_i ∈ {0,1}
|
||||
|
||||
↓ weighted
|
||||
|
||||
Weighted Automaton
|
||||
a_i ∈ R
|
||||
|
||||
↓ tensor/state
|
||||
|
||||
NGRA-Tensor
|
||||
a_i ∈ R^d
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. Relation 成为一等对象
|
||||
|
||||
不再采用固定 `Port4` 作为核心语义。
|
||||
|
||||
推荐模型:
|
||||
|
||||
```text
|
||||
Source Node
|
||||
↓
|
||||
[ Relation ]
|
||||
↓
|
||||
Destination Node
|
||||
```
|
||||
|
||||
Relation 包含:
|
||||
|
||||
```text
|
||||
Source
|
||||
Destination
|
||||
Type
|
||||
Parameters
|
||||
Weight / Weight Function
|
||||
Optional State
|
||||
Eligibility / Gate
|
||||
```
|
||||
|
||||
语义上:
|
||||
|
||||
```text
|
||||
Relation:
|
||||
决定一条联系是否生效,
|
||||
以及它如何把 Source 信息贡献给 Destination。
|
||||
```
|
||||
|
||||
Node 负责保存状态。
|
||||
|
||||
Blend 负责汇合多条 Relation 的贡献。
|
||||
|
||||
---
|
||||
|
||||
## 12. Pull,而不是主动广播
|
||||
|
||||
GPU / 执行层优先采用 destination pull:
|
||||
|
||||
```text
|
||||
Destination Node
|
||||
↓
|
||||
请求自己依赖的 Relations
|
||||
↓
|
||||
读取 Source states
|
||||
↓
|
||||
Relation Eval
|
||||
↓
|
||||
Blend
|
||||
```
|
||||
|
||||
而不是:
|
||||
|
||||
```text
|
||||
Source Node
|
||||
↓
|
||||
主动向大量 Destination 广播
|
||||
```
|
||||
|
||||
优点:
|
||||
|
||||
- 减少 scatter;
|
||||
- 减少原子写冲突;
|
||||
- 减少 readiness 管理;
|
||||
- 更适合 GPU destination-centric execution。
|
||||
|
||||
---
|
||||
|
||||
## 13. Relation 的三段语义
|
||||
|
||||
推荐拆成:
|
||||
|
||||
```text
|
||||
Eligibility
|
||||
↓
|
||||
Compatibility
|
||||
↓
|
||||
Transfer
|
||||
```
|
||||
|
||||
即:
|
||||
|
||||
1. **Eligibility**:这条 Relation 当前能不能参与;
|
||||
2. **Compatibility**:参与后有多重要;
|
||||
3. **Transfer**:真正传递什么信息。
|
||||
|
||||
形式上:
|
||||
|
||||
```text
|
||||
gate_ij = G(state, context, relation)
|
||||
score_ij = F(query, key, relation)
|
||||
value_ij = T(source_state, relation)
|
||||
```
|
||||
|
||||
最终:
|
||||
|
||||
```text
|
||||
output_i = Blend(gate_ij * weight_ij * value_ij)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14. 与 Attention / QKV 的关系
|
||||
|
||||
QKV 可以作为 Relation 内部算子的第一种 baseline,但不应决定 NGRA-Tensor 的外层拓扑。
|
||||
|
||||
标准 Attention:
|
||||
|
||||
```text
|
||||
Q = “我需要什么”
|
||||
K = “我适不适合被引用”
|
||||
V = “真正传递的内容”
|
||||
```
|
||||
|
||||
数学上:
|
||||
|
||||
```text
|
||||
score_ij = q_i · k_j
|
||||
weight_ij = softmax(score_ij)
|
||||
output_i = Σ weight_ij * v_j
|
||||
```
|
||||
|
||||
NGRA-Tensor 的差异:
|
||||
|
||||
- Relation 是显式、长期存在、可带类型的;
|
||||
- 候选关系通常是稀疏的;
|
||||
- 不一定对全部 node 做 dense attention;
|
||||
- Relation 可以有自己的参数、状态、类型和 gate;
|
||||
- Attention 可以只是某种 Relation Eval backend。
|
||||
|
||||
---
|
||||
|
||||
## 15. 稀疏关系,而不是全连接爆破
|
||||
|
||||
Transformer 的典型关系空间:
|
||||
|
||||
```text
|
||||
L × L
|
||||
```
|
||||
|
||||
NGRA-Tensor 的目标:
|
||||
|
||||
```text
|
||||
E << N²
|
||||
```
|
||||
|
||||
即只计算显式 Relation。
|
||||
|
||||
但必须承认:
|
||||
|
||||
- 稀疏图带来 gather;
|
||||
- 不规则访问会伤害 GPU;
|
||||
- 小算子过多会降低吞吐。
|
||||
|
||||
因此不能简单逐 Relation 启动 kernel。
|
||||
|
||||
---
|
||||
|
||||
## 16. Relation Tile / 脑区式分块
|
||||
|
||||
后续 GPU backend 应研究:
|
||||
|
||||
```text
|
||||
Semantic Relation Graph
|
||||
↓
|
||||
Graph Partition
|
||||
↓
|
||||
Relation Tiles
|
||||
↓
|
||||
Block-sparse / Dense subgraph execution
|
||||
```
|
||||
|
||||
目标:
|
||||
|
||||
> 块内允许不规则访问,块间尽量转成规则张量传输。
|
||||
|
||||
示意:
|
||||
|
||||
```text
|
||||
Dense / semi-dense local region
|
||||
↓
|
||||
少量跨区 Relation
|
||||
↓
|
||||
another dense / semi-dense region
|
||||
```
|
||||
|
||||
类似“脑区”:
|
||||
|
||||
- 区内关系较密;
|
||||
- 区间关系较稀;
|
||||
- 每个区可以编译成中型 tensor operator;
|
||||
- 跨区使用显式 Relation。
|
||||
|
||||
这可能成为 NGRA-Tensor 与传统 Transformer 之间的重要折中。
|
||||
|
||||
---
|
||||
|
||||
## 17. 语义稀疏 vs 物理稠密
|
||||
|
||||
前端语义可以是:
|
||||
|
||||
```text
|
||||
Sparse Relation Graph
|
||||
```
|
||||
|
||||
GPU backend 可以通过:
|
||||
|
||||
```text
|
||||
node reorder
|
||||
feature reorder
|
||||
relation reorder
|
||||
graph clustering
|
||||
block detection
|
||||
tile packing
|
||||
kernel fusion
|
||||
shared-memory staging
|
||||
```
|
||||
|
||||
将其转为:
|
||||
|
||||
```text
|
||||
Dense blocks
|
||||
+
|
||||
Block-sparse blocks
|
||||
+
|
||||
Irregular sparse tails
|
||||
```
|
||||
|
||||
原则:
|
||||
|
||||
> 模型的图结构不等于 GPU 必须逐边解释。
|
||||
|
||||
---
|
||||
|
||||
## 18. Node State 可以是高维 Tensor
|
||||
|
||||
NGRA-Tensor 不等于“每个 Node 只有一个 scalar”。
|
||||
|
||||
Node 可以持有:
|
||||
|
||||
```text
|
||||
x_i = [f0, f1, ..., fN]
|
||||
```
|
||||
|
||||
但 Relation 不一定读取全部 feature。
|
||||
|
||||
因此可能存在两层稀疏:
|
||||
|
||||
```text
|
||||
Topology sparsity
|
||||
Feature sparsity
|
||||
```
|
||||
|
||||
Relation 可以只对特定 feature/subspace 感兴趣。
|
||||
|
||||
例如:
|
||||
|
||||
```text
|
||||
Relation A:
|
||||
Q features = [1, 2, 7]
|
||||
K features = [4, 8]
|
||||
V features = [16, 17, 18]
|
||||
```
|
||||
|
||||
语义上只取这些 feature。
|
||||
|
||||
物理执行时 GPU compiler 可以为了带宽一次读入更大的连续 block。
|
||||
|
||||
---
|
||||
|
||||
## 19. Tensor 观念
|
||||
|
||||
后续 NGRA-Tensor 设计不要把 Tensor 只理解为“算子”。
|
||||
|
||||
Tensor 可以同时是:
|
||||
|
||||
```text
|
||||
状态
|
||||
参数
|
||||
关系
|
||||
mask
|
||||
中间结果
|
||||
数据集合
|
||||
算子
|
||||
```
|
||||
|
||||
“算子”只是某个 Tensor 在某次 contraction 中扮演的角色。
|
||||
|
||||
建议以“语义轴”理解 Tensor,例如:
|
||||
|
||||
```text
|
||||
Node × Feature
|
||||
Destination × Source
|
||||
Relation × Feature
|
||||
Time × Node × Feature
|
||||
```
|
||||
|
||||
优先思考:
|
||||
|
||||
- 哪些轴保留;
|
||||
- 哪些轴被收缩;
|
||||
- 哪些轴表示关系;
|
||||
- 哪些轴表示 feature;
|
||||
- 哪些轴可以 sparse;
|
||||
- 哪些轴可以 dense。
|
||||
|
||||
---
|
||||
|
||||
## 20. 训练 / 运行时关系发现
|
||||
|
||||
NGRA-Tensor 长期设想允许结构学习。
|
||||
|
||||
训练阶段:
|
||||
|
||||
```text
|
||||
短程强关系
|
||||
↓
|
||||
词 / Token 局部关系
|
||||
↓
|
||||
词组 / 短句
|
||||
↓
|
||||
长句 / 语义
|
||||
↓
|
||||
知识网络 / 长程弱关系
|
||||
```
|
||||
|
||||
运行时:
|
||||
|
||||
当现有网络:
|
||||
|
||||
```text
|
||||
NoActivePath
|
||||
LowConfidence
|
||||
HighAmbiguity
|
||||
Unknown
|
||||
```
|
||||
|
||||
可进入较慢的 Relation Discovery:
|
||||
|
||||
```text
|
||||
Stable Graph
|
||||
↓
|
||||
Unknown
|
||||
↓
|
||||
Candidate Relation Search
|
||||
↓
|
||||
Candidate Graph
|
||||
↓
|
||||
验证
|
||||
↓
|
||||
可能固化进入 Stable Graph
|
||||
```
|
||||
|
||||
不建议运行时每一步都修改主图。
|
||||
|
||||
应区分:
|
||||
|
||||
```text
|
||||
Fast inference time
|
||||
Slow structural learning time
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# 近期开发优先级
|
||||
|
||||
## Phase 1 — ε-NFA Reference Runtime
|
||||
|
||||
实现:
|
||||
|
||||
```text
|
||||
EpsilonNfa<TInput>
|
||||
StateId
|
||||
Transition
|
||||
ε-closure
|
||||
Consume
|
||||
Accept
|
||||
```
|
||||
|
||||
目标:
|
||||
|
||||
- 语义简单;
|
||||
- 正确性优先;
|
||||
- 不做复杂性能优化;
|
||||
- 不绑定 `char`。
|
||||
|
||||
## Phase 2 — Pattern → NFA Lowering
|
||||
|
||||
从现有 AST / Pattern IR 构造 NFA:
|
||||
|
||||
```text
|
||||
literal
|
||||
predicate
|
||||
concat
|
||||
alt
|
||||
repeat
|
||||
```
|
||||
|
||||
## Phase 3 — NFA Merge
|
||||
|
||||
实现:
|
||||
|
||||
```text
|
||||
multi-pattern merge
|
||||
PatternId
|
||||
shared start
|
||||
accept payload
|
||||
```
|
||||
|
||||
然后测试大量 Pattern 同时运行。
|
||||
|
||||
## Phase 4 — Token Stream
|
||||
|
||||
将输入切换到:
|
||||
|
||||
```text
|
||||
Token
|
||||
```
|
||||
|
||||
支持:
|
||||
|
||||
```text
|
||||
Token kind
|
||||
Token value
|
||||
Token metadata
|
||||
simple binding
|
||||
context predicate
|
||||
```
|
||||
|
||||
目标:作为编译器前端 / Token Regex 原型。
|
||||
|
||||
## Phase 5 — Hierarchical Automata
|
||||
|
||||
允许低层自动机输出高层节点,并再次进入更高层自动机。
|
||||
|
||||
## Phase 6 — Weighted Automaton
|
||||
|
||||
将:
|
||||
|
||||
```text
|
||||
active : bool
|
||||
```
|
||||
|
||||
推广为:
|
||||
|
||||
```text
|
||||
activation : float
|
||||
```
|
||||
|
||||
先不要急于引入复杂 Tensor。
|
||||
|
||||
## Phase 7 — NGRA-Tensor Prototype
|
||||
|
||||
研究:
|
||||
|
||||
```text
|
||||
Node state tensor
|
||||
Typed Relation
|
||||
Eligibility
|
||||
Compatibility
|
||||
Transfer
|
||||
Blend
|
||||
Sparse / block-sparse execution
|
||||
```
|
||||
|
||||
QKV/Attention 可作为 baseline Relation operator。
|
||||
|
||||
---
|
||||
|
||||
# 非目标
|
||||
|
||||
近期不要优先投入:
|
||||
|
||||
```text
|
||||
完整 .NET Regex 兼容
|
||||
极端 Span<char> 优化
|
||||
复杂 Source Generator matcher
|
||||
完整 capture rollback
|
||||
所有 lookaround/backreference
|
||||
为了 Regex 完整性设计核心 IR
|
||||
```
|
||||
|
||||
这些功能如确实需要,可以:
|
||||
|
||||
```text
|
||||
System.Text.RegularExpressions
|
||||
```
|
||||
|
||||
作为字符串兼容层。
|
||||
|
||||
这不是架构退让,而是目标调整。
|
||||
|
||||
---
|
||||
|
||||
# 一句话定位
|
||||
|
||||
NGRA 的新定位:
|
||||
|
||||
> **以 NFA 为基础的广义序列模式与分层自动机框架,并逐步推广到加权 Relation Network 与 NGRA-Tensor。**
|
||||
|
||||
Text Regex:
|
||||
|
||||
> 一个兼容前端 / 参考实现,不再定义 NGRA 的核心架构。
|
||||
|
||||
NGRA-Tensor:
|
||||
|
||||
> 把 Boolean NFA 的多状态传播推广为显式、稀疏、可加权、可训练、可分块执行的 Relation Network。
|
||||
@@ -5,7 +5,6 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
// -----------------------------------------------------------------------------
|
||||
|
||||
using Nitload.RegularAnalysis.Text;
|
||||
using System;
|
||||
using System.Collections;
|
||||
using System.Collections.Generic;
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
# Frozen Text Regex implementation
|
||||
|
||||
Text Regex is archived following the [NGRA transformation plan](../../../future/NGRA_Transformation_Plan.md).
|
||||
It is no longer NGRA's execution architecture or an active development target.
|
||||
|
||||
## Retained results and future string matching
|
||||
|
||||
Freezing development does not diminish the value of the work already completed. The project
|
||||
produced reusable AST/token infrastructure, the five-link tree model and its normalization,
|
||||
rotation, and merging techniques, as well as tested fixed-width COUNT retry logic. The archived
|
||||
generator also records useful engineering lessons about continuations, group boundaries, label
|
||||
ownership, and rollback. These are reference material for the new architecture, not a claim that
|
||||
the old matcher is feature-complete.
|
||||
|
||||
Future string-regex functionality will use .NET `GeneratedRegex`. Further research into custom
|
||||
`Span<char>` and SIMD optimizations for the Text Regex matcher is discontinued. This boundary
|
||||
applies to the string-regex implementation, not to SIMD work elsewhere in Nitload.
|
||||
|
||||
## Contents
|
||||
|
||||
- `Text/`: the text tokenizer/parser, regex-specific AST and optimizers, matcher contract,
|
||||
attribute, and all source-generator files.
|
||||
- `GroupIndex.cs`: the former group index/name record.
|
||||
- `SourceGenerate.cs`: the unused source-generation entry stub.
|
||||
- [Archived debug routines](../../../test/StgSharpDebug/Legacy/TextRegex): the original
|
||||
AST inspection and generated-source preview programs.
|
||||
|
||||
Source contents and namespaces are preserved. The generic infrastructure in `../Abstraction/`
|
||||
remains active, including the five-link node interface, generic AST, token abstractions,
|
||||
compilation stack, and sequence emitter. Text-specific tree transformations remain here
|
||||
as source material for future Pattern-to-NFA lowering.
|
||||
|
||||
## Build boundary
|
||||
|
||||
The project excludes `Legacy/**/*.cs` from `Compile` and retains the files as `None` items.
|
||||
They are not copied to build/publish output or packed. The active project no longer references
|
||||
Roslyn or packages its assembly as a source generator. Debug routines have the same exclusion
|
||||
in their own project and are not called from `Main`.
|
||||
|
||||
To investigate this implementation again, use a separate experimental project that explicitly
|
||||
includes the archived files and supplies the former Roslyn dependencies:
|
||||
`Microsoft.CodeAnalysis.CSharp` 4.13.0 and `Microsoft.CodeAnalysis.Analyzers` 3.11.0.
|
||||
The debug preview also required Roslyn at runtime and the `StgSharpDebug` friend assembly.
|
||||
Re-enabling files is not sufficient to make the unfinished matcher complete.
|
||||
|
||||
Historical notes remain in [IR_RETIRED.md](../IR_RETIRED.md) and
|
||||
[regex_node_transaction.md](../regex_node_transaction.md). They describe earlier experiments,
|
||||
not the new NFA execution contract. Older `.discard` material is unchanged.
|
||||
|
||||
The next runtime is planned around generic sequence input and Boolean ε-NFA execution;
|
||||
weighted automata and NGRA-Tensor remain later development/research stages.
|
||||
@@ -1,26 +1,14 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<IsRoslynComponent>true</IsRoslynComponent>
|
||||
<EnforceExtendedAnalyzerRules>true</EnforceExtendedAnalyzerRules>
|
||||
<EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles>
|
||||
<CompilerGeneratedFilesOutputPath>$(BaseIntermediateOutputPath)Generated</CompilerGeneratedFilesOutputPath>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\Nitload.Common\Nitload.Common.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.11.0" PrivateAssets="all" />
|
||||
<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.13.0" PrivateAssets="all" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<None Include="$(OutputPath)$(AssemblyName).dll" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false" />
|
||||
<None Include="$(OutputPath)$(AssemblyName).pdb" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false" Condition="Exists('$(OutputPath)$(AssemblyName).pdb')" />
|
||||
<None Include="$(OutputPath)Nitload.Common.dll" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false" Condition="Exists('$(OutputPath)Nitload.Common.dll')" />
|
||||
<None Include="$(OutputPath)Nitload.Common.pdb" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false" Condition="Exists('$(OutputPath)Nitload.Common.pdb')" />
|
||||
<!-- Keep the frozen Text Regex sources visible, but out of builds, packages and deployment. -->
|
||||
<Compile Remove="Legacy\**\*.cs" />
|
||||
<None Include="Legacy\**\*.cs" />
|
||||
<None Update="Legacy\**\*" Pack="false" CopyToOutputDirectory="Never" CopyToPublishDirectory="Never" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
||||
+1
-1
@@ -10,7 +10,7 @@ This directory contains the production source code, native runtime, supporting a
|
||||
| `Nitload.Mathematics` | Numeric types, vectors, matrices, geometry, and managed access to accelerated math operations. |
|
||||
| `Nitload.Graphics` | OpenGL bindings, rendering infrastructure, shaders, textures, windows, and input integration. |
|
||||
| `Nitload.Native` | C native runtime, GLFW/STBI integration, allocators, platform code, and optimized SIMD/matrix kernels. See its own `README.md` for native build details. |
|
||||
| `Nitload.RegularAnalysis` | NGRA regular-language analysis, regex processing, intermediate representations, and source-generation infrastructure. |
|
||||
| `Nitload.RegularAnalysis` | NGRA generic AST/token infrastructure, transitioning toward ε-NFA sequence analysis. Text Regex and its source generator are archived under `Legacy` and excluded from compilation. |
|
||||
| `Nitload.UserInterface` | State-oriented user-interface framework and its incremental source generator. |
|
||||
| `Nitload.TerminalDialogue` | Terminal user-interface components and an interactive terminal application. |
|
||||
| `Nitload.Benchmark` | BenchmarkDotNet workloads for measuring selected runtime and allocation behavior. |
|
||||
|
||||
+18
-26
@@ -16,36 +16,28 @@ Run the active debug harness directly when investigating a subsystem:
|
||||
dotnet run --project test/StgSharpDebug/StgSharpDebug.csproj
|
||||
```
|
||||
|
||||
## NGRA generation preview
|
||||
## Archived Text Regex diagnostics
|
||||
|
||||
`StgSharpDebug/TestRegexGenerate.cs` contains a standalone debugging routine that calls the
|
||||
existing NGRA emitter without Roslyn attribute binding. Select it from the debug `Main`:
|
||||
The Text Regex analyzer and source-generator preview are frozen. Their original source files
|
||||
are retained under [StgSharpDebug/Legacy/TextRegex](StgSharpDebug/Legacy/TextRegex) and excluded
|
||||
from compilation, packaging, and build/publish copying. `Main` no longer calls them, and
|
||||
`--regex-preview` is not an active entry point.
|
||||
|
||||
```powershell
|
||||
dotnet run --project test/StgSharpDebug/StgSharpDebug.csproj -- --regex-preview
|
||||
```
|
||||
The corresponding runtime, text-specific AST/parser, and generator are archived under
|
||||
[src/Nitload.RegularAnalysis/Legacy](../src/Nitload.RegularAnalysis/Legacy/README.md).
|
||||
Generic AST and token infrastructure remain active. See the
|
||||
[NGRA transformation plan](../future/NGRA_Transformation_Plan.md) for the new NFA direction.
|
||||
|
||||
This branch returns before `World` or native-memory initialization. Without the switch, the
|
||||
existing debug flow is unchanged. You can also call `TestRegexGenerate.Run()` directly from
|
||||
`Main` while debugging; pass a pattern to `Run("abc")` for a single preview.
|
||||
The switch only selects runtime behavior; building the debug project still builds its normal
|
||||
project references, including Graphics and Mathematics, and prepares the native library with
|
||||
CMake. `BuildNativeBeforeManaged` defaults to true; Debug selects `clang-debug`, and other
|
||||
configurations select `clang-release`. `RequireNativeLibrary` reports a missing DLL at build
|
||||
time instead of allowing the default runtime path to fail later.
|
||||
## Native builds
|
||||
|
||||
For managed-only preview work without a native toolchain or DLL, add
|
||||
`-p:BuildNativeBeforeManaged=false -p:RequireNativeLibrary=false` before `--` in the run command.
|
||||
Building the debug project still builds its normal project references, including Graphics and
|
||||
Mathematics, and prepares the native library with CMake. `BuildNativeBeforeManaged` defaults
|
||||
to true; Debug selects `clang-debug`, and other configurations select `clang-release`.
|
||||
`RequireNativeLibrary` reports a missing DLL at build time instead of allowing execution to
|
||||
fail later.
|
||||
|
||||
Enter raw regular expressions without C# string delimiters. The routine prints only the emitted
|
||||
code, without status headers or file output. Exceptions propagate directly to the caller/debugger.
|
||||
Use `:exit` or end-of-input to leave the loop. Empty input is an empty regex. For one-shot mode:
|
||||
|
||||
```powershell
|
||||
dotnet run --project test/StgSharpDebug/StgSharpDebug.csproj -- --regex-preview 'a{2,3}'
|
||||
```
|
||||
|
||||
Set breakpoints in `TestRegexGenerate.cs` or any existing `Generate*` method. The output is a
|
||||
method-body fragment; this routine does not compile or execute it.
|
||||
For a managed-only build without a native toolchain or DLL, pass
|
||||
`-p:BuildNativeBeforeManaged=false -p:RequireNativeLibrary=false`.
|
||||
This does not make debug routines that use native services runnable without the native library.
|
||||
|
||||
New repeatable tests should use a test project and avoid machine-specific paths, external mutable data, and dependencies on previous test execution. Keep one-off investigations in `SingleFile` or the manual debug harness until their expected behavior is stable enough for automation.
|
||||
|
||||
@@ -37,8 +37,7 @@ namespace Nitload
|
||||
// .UseModule<GraphicModule>()
|
||||
.InitializeModule();
|
||||
|
||||
TestRegexAnalyse.Run();
|
||||
_ = TestRegexGenerate.Run("abc1{1,2}23");
|
||||
// Text Regex diagnostics are frozen under Legacy/TextRegex and excluded from compilation.
|
||||
|
||||
// TestHlsf();
|
||||
|
||||
|
||||
@@ -14,8 +14,10 @@
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<!-- The debug host calls the emitter directly and needs Roslyn at runtime. -->
|
||||
<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.13.0" />
|
||||
<!-- Archived Text Regex diagnostics are retained as source, not executable tests. -->
|
||||
<Compile Remove="Legacy\**\*.cs" />
|
||||
<None Include="Legacy\**\*.cs" />
|
||||
<None Update="Legacy\**\*" Pack="false" CopyToOutputDirectory="Never" CopyToPublishDirectory="Never" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
|
||||
Reference in New Issue
Block a user