remove useless node from source gen pipeline. add source gen logic
This commit is contained in:
Youmu Liu
@@ -0,0 +1,159 @@
|
||||
# TLSF (Two-Layer Segregated Fit) Allocator
|
||||
|
||||
This document describes the current StgSharp native arena allocator, `StgSharp.HighPerformance.Memory.TwoLayerSegregatedFitAllocator`. The file name is historical; the implementation described below is TLSF rather than the older HLSF allocator.
|
||||
|
||||
## Overview
|
||||
|
||||
- Single contiguous native arena allocated up front
|
||||
- Headerless returned blocks; allocator metadata is stored in 32-byte nodes carved from the tail of the same arena
|
||||
- Requests rounded up to 64-byte granularity, with 64 bytes as the minimum block size
|
||||
- Free blocks grouped by coarse size level derived from the highest set bit of the block size
|
||||
- Exact-level allocation uses first-fit traversal inside the selected bucket; higher levels fall back to the head of the first non-empty bucket
|
||||
- `Free` performs bounded local coalescing; `Collect` performs full free-run coalescing across the arena
|
||||
- Single-threaded implementation; concurrent access requires external synchronization
|
||||
- No arena growth and no implicit fallback to `NativeMemory.AlignedAlloc` for oversized requests
|
||||
|
||||
## Quick Use
|
||||
|
||||
```csharp
|
||||
using StgSharp.HighPerformance.Memory;
|
||||
|
||||
using var tlsf = new TwoLayerSegregatedFitAllocator(64 * 1024 * 1024);
|
||||
var handle = tlsf.Alloc(1024);
|
||||
|
||||
Span<byte> buffer = handle.AsSpan();
|
||||
buffer[0] = 42;
|
||||
|
||||
tlsf.Free(handle);
|
||||
```
|
||||
|
||||
## Constructors
|
||||
|
||||
- `TwoLayerSegregatedFitAllocator(nuint byteSize)`
|
||||
- `TwoLayerSegregatedFitAllocator(nuint byteSize, uint maxBlockSize)`
|
||||
- `TwoLayerSegregatedFitAllocator(nuint byteSize, int align)`
|
||||
- `TwoLayerSegregatedFitAllocator(nuint byteSize, int align, nuint maxBlockSize)`
|
||||
|
||||
Behavior summary:
|
||||
|
||||
- `byteSize` defines the total arena size
|
||||
- `maxBlockSize` defaults to 32 MB and defines the largest single request accepted by `Alloc` and `TryAlloc`
|
||||
- `align` is passed to the backing `NativeMemory.AlignedAlloc` call for arena allocation; request sizes inside the arena are still quantized to 64-byte blocks
|
||||
|
||||
## Public API
|
||||
|
||||
### `Alloc(nuint size)`
|
||||
|
||||
- Rounds `size` up to the next 64-byte block, with a minimum of 64 bytes
|
||||
- Throws `OverflowException` if the normalized size exceeds `maxBlockSize` or if the arena cannot provide a block
|
||||
- Returns a `Handle` containing block address and block size
|
||||
- Splits a sufficiently large tail remainder into a new free node and re-buckets that node
|
||||
|
||||
### `TryAlloc(nuint size, out Handle handle)`
|
||||
|
||||
- Uses the same size normalization and maximum block limit as `Alloc`
|
||||
- Returns `false` when the request exceeds the configured limit or when the arena is exhausted
|
||||
- Does not fall back to the system allocator
|
||||
|
||||
### `Free(Handle handle, int collectScanWindow = 4)`
|
||||
|
||||
- Marks the block as free
|
||||
- Scans up to `collectScanWindow` neighbors toward lower addresses and then up to `collectScanWindow` neighbors toward higher addresses
|
||||
- Merges any contiguous free neighbors found during those scans
|
||||
- Reclassifies the merged block back into the coarse bucket set
|
||||
|
||||
### `Collect()`
|
||||
|
||||
- Walks the full position chain from low address to high address
|
||||
- Merges consecutive free runs without the local scan-window limit used by `Free`
|
||||
- Restores larger reusable spans and improves bucket coverage after fragmentation-heavy phases
|
||||
|
||||
### `Dispose()`
|
||||
|
||||
- Releases the backing arena through the stored free callback
|
||||
|
||||
### `Handle`
|
||||
|
||||
- Readonly value type carrying the allocator metadata reference, block address, and block size
|
||||
- `Address` exposes the native pointer
|
||||
- `Size` exposes the normalized block size
|
||||
- `AsSpan(int range = -1)` exposes a span view over the block
|
||||
- The indexer provides byte-level access by offset
|
||||
|
||||
## Allocation Model
|
||||
|
||||
### Level Classification
|
||||
|
||||
- `GetLevel(size)` maps the normalized size to `57 - BitOperations.LeadingZeroCount(size)` and clamps the result to `_maxLevel`
|
||||
- Each bucket corresponds to one coarse size class rather than an exact size
|
||||
- The target bucket can therefore contain blocks of different sizes within the same coarse level
|
||||
|
||||
### Allocation Path
|
||||
|
||||
1. Normalize the requested size to a 64-byte multiple.
|
||||
2. Traverse the target bucket and select the first free block whose size is at least the requested size.
|
||||
3. If the target bucket has no suitable block, pop the head of the first non-empty higher-level bucket.
|
||||
4. If all buckets are empty, carve a fresh block from the spare region tracked by the arena sentinel.
|
||||
5. Split the tail remainder when both remainder size and metadata availability allow it.
|
||||
|
||||
### Metadata Layout
|
||||
|
||||
- `Metadata` is a 32-byte node
|
||||
- `NextNear` and `PreviousNear` form the address-ordered position chain
|
||||
- `NextLevel` and `PreviousLevel` form either the bucket chain or the recycled-metadata chain, depending on node state
|
||||
- The spare sentinel at the end of the arena tracks remaining virgin space, recycled metadata nodes, and position-chain anchors
|
||||
- Returned blocks remain headerless from the caller's point of view
|
||||
|
||||
### Coalescing Model
|
||||
|
||||
- `Free` performs bounded local coalescing to keep the hot path short
|
||||
- `Collect` performs full coalescing of consecutive free runs across the entire position chain
|
||||
- Recycled metadata nodes are returned to the spare metadata list and reused by later splits or fresh arena chunks
|
||||
|
||||
## Complexity and Runtime Characteristics
|
||||
|
||||
- `Alloc`: O(1) bucket index calculation plus target-level first-fit traversal; higher-level fallback is O(1)
|
||||
- `Free`: O(`collectScanWindow`) local merge cost
|
||||
- `Collect`: O(number of metadata nodes in the position chain)
|
||||
- Best steady-state behavior appears on repeated small and medium native allocations inside a pre-allocated arena
|
||||
|
||||
## BenchmarkDotNet (TLSF vs Libc)
|
||||
|
||||
Environment: .NET 8.0.26 (PerformanceDebug), x64 RyuJIT AVX2, Windows 11, AMD Ryzen 7 6800H, BenchmarkDotNet v0.15.8, InProcessEmitToolchain, LaunchCount=1, WarmupCount=6, IterationCount=15, InvocationCount=3.
|
||||
|
||||
Workload summary:
|
||||
|
||||
- Arena size: 512 MB
|
||||
- Arena alignment: 64 bytes
|
||||
- Allocation count per batch: 2048
|
||||
- Batch count: 256
|
||||
- Request set before normalization: 64, 100, 200, 400, 1024, 4096 bytes
|
||||
- Free order: shuffled per batch
|
||||
- `Free` local merge window: 4
|
||||
|
||||
Best observed run under the current harness:
|
||||
|
||||
| Method | Mean | Error | StdDev | Ratio | RatioSD | Allocated | Alloc Ratio |
|
||||
|------- |-----:|------:|-------:|------:|--------:|----------:|------------:|
|
||||
| Tlsf | 79.11 ns | 5.944 ns | 5.560 ns | 0.65 | 0.09 | - | NA |
|
||||
| Libc | 123.64 ns | 15.955 ns | 14.144 ns | 1.01 | 0.16 | - | NA |
|
||||
|
||||
Worst observed run under the same harness:
|
||||
|
||||
| Method | Mean | Error | StdDev | Ratio | RatioSD | Allocated | Alloc Ratio |
|
||||
|------- |-----:|------:|-------:|------:|--------:|----------:|------------:|
|
||||
| Tlsf | 82.95 ns | 6.337 ns | 5.617 ns | 0.81 | 0.07 | - | NA |
|
||||
| Libc | 102.63 ns | 8.590 ns | 7.173 ns | 1.00 | 0.09 | - | NA |
|
||||
|
||||
Interpretation:
|
||||
|
||||
- The current single-pass bucket selection fast path materially reduces allocator-side latency
|
||||
- The reported ratio still drifts across repeated compile-and-run cycles because the libc baseline also moves under the current InProcess harness
|
||||
- Recent repeated runs place the transient best case near 0.65 and the longer-running steady band near 0.75 to 0.80
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- Maximum single-allocation size is fixed at construction time
|
||||
- Arena growth is not implemented
|
||||
- `Collect` is the global defragmentation pass; `Free` only performs bounded local coalescing
|
||||
- The allocator is intended for explicit native-memory workflows rather than as a drop-in general-purpose heap replacement
|
||||
@@ -0,0 +1,762 @@
|
||||
# L4 Software Cache Design Specification
|
||||
|
||||
> Project: StgSharp
|
||||
> Author: Nitload
|
||||
> Version: Draft v0.3
|
||||
> Date: 2026-03-10
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
### 1.1 Problem Background
|
||||
|
||||
StgSharp's matrix computation system is migrating from a "single global context table" to a
|
||||
"per-thread context" architecture to adapt to the scheduling characteristics of hybrid-architecture
|
||||
CPUs (e.g., Intel P-Core / E-Core). After the migration, each thread independently holds its own
|
||||
computation context, but matrix data may still be allocated by the HLSF allocator or global
|
||||
`malloc` on a remote NUMA node, resulting in significantly increased cross-node access latency.
|
||||
|
||||
At the same time, in matrix multiplication (FMA), the core row data of matrix A in the
|
||||
checkerboard-shaped kernel grid layout must be repacked (Packed) into a micro-kernel-friendly
|
||||
contiguous arrangement. In libraries such as OpenBLAS, this Pack operation is re-executed on
|
||||
every tile switch, whereas L4 can cache the Pack results to avoid repeated reordering.
|
||||
|
||||
Additionally, the segmented accumulation (segment) of matrix multiplication requires storing
|
||||
intermediate partial sums locally to avoid cross-NUMA writes to the target matrix on every
|
||||
segment. L4 can serve as a local accumulation buffer for partial sums, writing them back in a
|
||||
single operation after all segments complete locally.
|
||||
|
||||
### 1.2 Solution
|
||||
|
||||
The L4 cache is a purely software-implemented **data caching layer** logically positioned
|
||||
between the CPU hardware caches (L1/L2/L3) and main memory. Its core responsibilities:
|
||||
|
||||
1. **Pack result caching**: Cache the repack results of FMA core rows in matrix A to avoid repeated Pack operations
|
||||
2. **NUMA migration cache**: Migrate data from remote NUMA nodes to local memory and cache it, eliminating cross-node latency
|
||||
3. **Segment accumulation buffer**: Temporarily store partial sums from FMA segmented computation to reduce remote write operations
|
||||
4. **PLU intermediate data buffer**: Provide a local contiguous workspace for PLU decomposition
|
||||
|
||||
### 1.3 Design Goals
|
||||
|
||||
- Provide NUMA-friendly local data access for per-thread matrix computation contexts
|
||||
- Fixed capacity (64 MB), no runtime expansion, deterministic latency
|
||||
- Transparent pointer redirection: upper-layer code queries via original pointers; L4 returns local cache pointers
|
||||
- Cache eviction policy based on **RRPV mask + windowed OPT prediction**
|
||||
- Support both **StreamTransparent** and **CacheBuffer** working modes
|
||||
- Support **writable cache lines** that invoke registered callback functions for pre-processing and write-back upon eviction or explicit flush
|
||||
|
||||
---
|
||||
|
||||
## 2. Overall Architecture
|
||||
|
||||
### 2.1 Three-Component Model
|
||||
|
||||
```
|
||||
┌���─────────────────────────────────────────────────────────────────┐
|
||||
│ L4 │
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌────────────────────┐ ┌──────────────────┐ │
|
||||
│ │ CacheLine │ │ L4 Core │ │ Predictor │ │
|
||||
│ │ Handle │ │ Cache │ │ Prefetch │ │
|
||||
│ │ │ │ │ │ Predictor │ │
|
||||
│ │ • local addr │ │ • Swiss Table │ │ • 2048-entry buf │ │
|
||||
│ │ • ref count │ │ • Slab Allocator │ │ • threshold 1024 │ │
|
||||
│ │ • Dispose │ │ • RRPV mask │ │ • windowed OPT │ │
|
||||
│ │ drops ref │ │ • SIMD eviction │ │ • active key set │ │
|
||||
│ │ │ │ • writeback mgmt │ │ │ │
|
||||
│ └──────────────┘ └────────────────────┘ └──────────────────┘ │
|
||||
│ │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 2.2 Data Access Path
|
||||
|
||||
```
|
||||
Compute thread accessing data:
|
||||
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ Read-only path: L4.Map(originKey) │
|
||||
│ ├─ StreamTransparent + no repack needed? │
|
||||
│ │ └─ Yes → return original pointer directly │
|
||||
│ │ (zero overhead) │
|
||||
│ │ │
|
||||
│ └─ CacheBuffer mode │
|
||||
│ ├─ Swiss Table lookup (2.6ns, lock-free) │
|
||||
│ ├─ Hit → update RRPV=0, return CacheLine handle │
|
||||
│ └─ Miss → SpinWait for Predictor prefetch │
|
||||
├─────────────────────────────────────────────────────────┤
|
||||
│ Writable path: L4.MapWritable(originKey, preprocess, writeback) │
|
||||
│ ├─ Hit → return existing partial sum cache line │
|
||||
│ └─ Miss → allocate zero-initialized cache line, │
|
||||
│ register writeback callbacks │
|
||||
│ cache line marked dirty (dirty=1) │
|
||||
│ eviction invokes preprocess + writeback │
|
||||
├─────────────────────────────────────────────────────────┤
|
||||
│ After computation: │
|
||||
│ 3. CacheLine.Dispose() (refCount--) │
|
||||
│ 4. L4.StepComplete() │
|
||||
│ ├─ Predictor.AdvanceExecutionPoint() │
|
||||
│ └─ Predictor.NeedsPrefetch? │
|
||||
│ └─ Yes → CAS-acquire lock, execute prefetch │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 2.3 Key Parameters
|
||||
|
||||
| Parameter | Value | Description |
|
||||
|------------------------|--------------------------|------------------------------------------------------|
|
||||
| Total cache capacity | 64 MB | Per-NUMA-node contiguous memory |
|
||||
| Cache line size | 8192 Bytes | = 64 × `double MAT_KERNEL` |
|
||||
| Cache line count | 8192 | 64MB / 8192B |
|
||||
| Mapping table type | Swiss Table | 1024 buckets × 16 slots, SIMD-accelerated |
|
||||
| Cache line pool | Slab Allocator | Non-concurrent, fixed capacity, no auto-expansion |
|
||||
| Eviction policy | RRPV mask + windowed OPT | SoA layout + SSE2 batch scan |
|
||||
| Prefetch policy | Predictor windowed OPT | 2048-entry buffer, 1024-step lookahead |
|
||||
| Writeback policy | Callback-driven | Invokes registered callbacks on dirty eviction/flush |
|
||||
|
||||
---
|
||||
|
||||
## 3. Key Components
|
||||
|
||||
### 3.1 Swiss Table (Pointer Mapping Table)
|
||||
|
||||
**Responsibility**: Maintains the `original pointer → cache pointer` mapping.
|
||||
|
||||
- Key: original pointer `nint`, pointing to data allocated by HLSF/malloc on a remote node
|
||||
- Value: cache pointer `nint`, pointing to locally Slab-allocated cache line
|
||||
- Hash: hardware `CRC32C` instruction on the 64-bit pointer
|
||||
- Low 10 bits → bucket index (1024 buckets)
|
||||
- Bits 10-16 → H2 control byte (used for SIMD pre-filtering)
|
||||
- Scan: SSE2 `pmovmskb` instruction for parallel comparison of 16 control bytes
|
||||
- Tombstone mechanism: deleted entries set ctrl to `0xFE`, preserving probe chain integrity
|
||||
- Capacity: 1024 × 16 = 16,384 mappings, significantly larger than total cache line count (8192 lines),
|
||||
ensuring extremely low hash collision rate
|
||||
|
||||
**Measured Performance** (Ryzen 7 6800H, .NET 8, BenchmarkDotNet):
|
||||
|
||||
| Operation | Swiss Table | Dictionary | Speedup |
|
||||
|--------------------------|-------------|-------------|---------|
|
||||
| Zipfian Lookup (8192×) | 20.0 μs | 76.1 μs | 3.8× |
|
||||
| Uniform Lookup (8192×) | 18.6 μs | 50.6 μs | 2.7× |
|
||||
| Insert (4096 entries) | 18.5 μs | 46.5 μs | 2.5× |
|
||||
| SteadyState (4096 ops) | 25.3 μs | 68.3 μs | 2.7× |
|
||||
|
||||
Single lookup latency ≈ 2.6 ns, GC allocation = 0.
|
||||
|
||||
### 3.2 Slab Allocator (Cache Line Pool)
|
||||
|
||||
**Responsibility**: Manages allocation and reclamation of fixed-size (8192B) cache lines.
|
||||
|
||||
- Mode: **Non-concurrent, fixed capacity** (`FixedSlabAllocator<T>` mode),
|
||||
reusing the existing `SlabAllocator.FromBuffer<T>()` interface
|
||||
- No auto-expansion: capacity is fixed at initialization (64MB / 8192B = 8192 lines);
|
||||
allocation failure triggers eviction rather than expansion
|
||||
- Allocation complexity: O(1), pops from free stack
|
||||
- Reclamation complexity: O(1), pushes back to free stack
|
||||
- Alignment: all cache lines are 64B aligned, ensuring no cross-hardware-cache-line boundaries
|
||||
|
||||
### 3.3 Eviction Metadata (SoA Layout + RRPV Mask + Writeback Callbacks)
|
||||
|
||||
**Responsibility**: Tracks per-cache-line reference state, heat, and writeback information to determine eviction priority.
|
||||
|
||||
Replaces the four-level circular linked list from v0.1 with parallel arrays in Structure-of-Arrays layout:
|
||||
|
||||
```
|
||||
SoA Layout — Parallel Arrays
|
||||
|
||||
_refFlags[i] : byte Reference count snapshot (0=free, non-zero=held)
|
||||
_evictionFlags[i] : byte dirty(1bit) + RRPV(2bit) + costClass(1bit)
|
||||
_originKeys[i] : nuint Original pointer (reverse-deregisters Swiss Table entry on eviction)
|
||||
_preprocess[i] : nint Writeback pre-processing function pointer (called before dirty eviction)
|
||||
_writeback[i] : nint Writeback function pointer (called during dirty eviction)
|
||||
_callbackCtx[i] : nuint Callback context (passed to preprocess/writeback)
|
||||
```
|
||||
|
||||
**Eviction Flag Bit Layout** (`_evictionFlags[i]`):
|
||||
|
||||
The RRPV array uses byte storage for each cache line's usage state; the low 2 bits encode the RRPV value.
|
||||
|
||||
**RRPV Semantics**:
|
||||
|
||||
| RRPV | Meaning | Eviction Priority |
|
||||
|------|---------------------------------|---------------------------|
|
||||
| 0 | Will be re-accessed very soon | Lowest (do not evict) |
|
||||
| 1 | Will be accessed shortly | Low |
|
||||
| 2 | Will be accessed in a while | Medium |
|
||||
| 3 | May never be accessed again | Highest (evict first) |
|
||||
|
||||
**RRPV Update Events**:
|
||||
|
||||
| Event | Operation | Description |
|
||||
|--------------------|---------------------------------|------------------------------------------------------|
|
||||
| Load (insert) | RRPV = 0 | Freshly loaded data will be used soon |
|
||||
| Hit | `_evictionFlags[i] = 0` | Bits 1-2 cleared, RRPV reset to 0 |
|
||||
| Eviction scan | Find lines with RRPV=3 to evict | If none found, increment all RRPV +1 (aging) |
|
||||
|
||||
**Advantages over the v0.1 four-level circular linked list**:
|
||||
|
||||
| Comparison | v0.1 Circular List | v0.3 RRPV Mask + Callbacks |
|
||||
|----------------------|-------------------------------------|--------------------------------------|
|
||||
| Data structure | 4 doubly-linked lists + next ptrs | Byte array + function pointer arrays |
|
||||
| Warm-up operation | Unlink + relink (4 pointer writes) | `+= 1` (1-byte write) |
|
||||
| Aging operation | Per-node list migration | SIMD batch +1 (16 entries/cycle) |
|
||||
| Eviction scan | Iterate list head (pointer chasing) | SIMD pattern matching (16/cycle) |
|
||||
| Cache friendliness | Poor (pointer chasing) | Good (contiguous memory scan) |
|
||||
| Reference protection | None | Lines with refCount > 0 not evicted |
|
||||
| Writeback support | None | Callback-function driven |
|
||||
|
||||
### 3.4 Writeback Callback Mechanism
|
||||
|
||||
**Responsibility**: Provides flexible writeback strategies for writable cache lines, using native function pointers to avoid managed overhead.
|
||||
|
||||
**Design Principle**: All writable cache lines are treated as dirty by default. On eviction or explicit flush,
|
||||
L4 itself does not manage the writeback strategy; instead, the bound predictor is responsible for implementing it.
|
||||
|
||||
The predictor is defined via the following interface:
|
||||
|
||||
```
|
||||
public interface IPrefetchPredict
|
||||
{
|
||||
|
||||
int Predict(
|
||||
Span<CacheLinePrediction> node
|
||||
);
|
||||
|
||||
bool Prefetch(
|
||||
nuint origin,
|
||||
ulong policy,
|
||||
nuint cache
|
||||
);
|
||||
|
||||
void WriteBack(
|
||||
nuint origin,
|
||||
ulong policy,
|
||||
nuint cache
|
||||
);
|
||||
|
||||
}
|
||||
```
|
||||
|
||||
**Typical Callback Implementations**:
|
||||
|
||||
```
|
||||
Scenario 1: Segment accumulation (FMA partial sum)
|
||||
preprocess = null (no pre-processing needed; data is already a double array)
|
||||
writeback = AccumulateAdd (element-wise dst[i] += src[i], AVX2-vectorized)
|
||||
|
||||
Scenario 2: Writeback of packed intermediate results
|
||||
preprocess = UnpackKernelRow (reverse-repack from Pack format to row-major order)
|
||||
writeback = AccumulateAdd
|
||||
|
||||
Scenario 3: PLU decomposition intermediate data
|
||||
preprocess = null
|
||||
writeback = DirectCopy (direct overwrite memcpy)
|
||||
```
|
||||
|
||||
### 3.5 CacheLine (Cache Line Handle)
|
||||
|
||||
**Responsibility**: A cache line access handle held by the compute thread; prevents in-use lines from being evicted via reference counting.
|
||||
|
||||
```csharp
|
||||
public ref struct CacheLine : IDisposable
|
||||
{
|
||||
private ref int _activeCount; // L4 global active count
|
||||
private ref int _refCount; // Per-line reference count
|
||||
private nuint _handle; // Local cache data address
|
||||
|
||||
// Construction: Interlocked.Increment(ref _activeCount)
|
||||
// Interlocked.Increment(ref _refCount)
|
||||
// Dispose: Interlocked.Decrement on both counters
|
||||
}
|
||||
```
|
||||
|
||||
- `ref struct` ensures it cannot escape to the heap
|
||||
- Reference counts use `Interlocked` atomic operations for thread safety
|
||||
- Lines with `_refCount > 0` are skipped during eviction scans
|
||||
- StreamTransparent mode returns a CacheLine that does not occupy a cache slot
|
||||
- Both read-only and writable paths return the same CacheLine type,
|
||||
callers decide read/write behavior via `.Address`
|
||||
|
||||
### 3.6 Predictor (Prefetch Predictor)
|
||||
|
||||
**Responsibility**: Provides deterministic prefetch predictions for L4 based on the windowed OPT strategy.
|
||||
|
||||
**Core Principle**: Matrix computation task sequences are deterministic — the scheduler pre-allocates
|
||||
task blocks `(kr, kc, zz)` for each thread, allowing precise computation of the cache line keys
|
||||
required at each future step. This gives L4 a predictive capability that hardware caches cannot
|
||||
possess: **direct visibility into the future**, rather than inferring the future from history as
|
||||
strategies like Hawkeye do.
|
||||
|
||||
```
|
||||
Predictor Sliding Window:
|
||||
|
||||
Task sequence: ... [T_now] [T+1] [T+2] ... [T+1023] ... [T+2047]
|
||||
↑ execution point ↑ buffer tail
|
||||
|
||||
Window capacity = 2048 prediction entries
|
||||
Prefetch trigger threshold = trigger refill when prefetch frontier
|
||||
leads execution point by fewer than 1024 steps
|
||||
Refill batch size = 64 entries (avoids excessive single-refill latency)
|
||||
```
|
||||
|
||||
**数据结构**：
|
||||
|
||||
```csharp
|
||||
internal class Predictor
|
||||
{
|
||||
private const int WindowCapacity = 2048;
|
||||
private const int RefillThreshold = 1024;
|
||||
|
||||
private PrefetchEntry[] _buffer; // Circular prediction buffer
|
||||
private int _head; // Prefetch frontier (prefetched up to here)
|
||||
private int _tail; // Production position (scheduler fills up to here)
|
||||
private int _executed; // Execution point (compute thread has completed up to here)
|
||||
|
||||
private HashSet<nuint> _activeKeys; // All active keys within the window (deduplicated)
|
||||
}
|
||||
|
||||
struct PrefetchEntry
|
||||
{
|
||||
public nuint Key; // Cache line identifier
|
||||
public nuint SourceAddress; // Source data address
|
||||
public bool NeedsRepack; // Whether non-contiguous repacking is needed
|
||||
}
|
||||
```
|
||||
|
||||
**Cooperation with Eviction Policy**:
|
||||
|
||||
```
|
||||
Eviction priority (from highest to lowest priority):
|
||||
|
||||
1. refCount==0, not in window, RRPV==3, dirty==0 ← First (zero-cost eviction)
|
||||
2. refCount==0, not in window, RRPV==3, dirty==1 ← Second (needs writeback but cold)
|
||||
3. refCount==0, not in window, RRPV<3, dirty==0 ← Ordered by RRPV
|
||||
4. refCount==0, not in window, RRPV<3, dirty==1 ← Ordered by RRPV
|
||||
5. refCount==0, in window ← Avoid evicting if possible
|
||||
6. refCount>0 ← Never evict
|
||||
```
|
||||
|
||||
Windowed OPT means most eviction decisions do not depend on RRPV — "not in window" alone is
|
||||
sufficient to determine evictability. RRPV provides fine-grained differentiation among
|
||||
out-of-window lines; the dirty bit preferentially evicts clean lines at equal RRPV to avoid writeback overhead.
|
||||
|
||||
### 3.7 Concurrency Model
|
||||
|
||||
L4 uses a **no-extra-threads** design; all operations are performed by compute threads themselves:
|
||||
|
||||
| Operation | Executor | Concurrency |
|
||||
|------------------------------------|---------------------------------|--------------------------------------|
|
||||
| Lookup (Swiss Table read) | Any compute thread | ✅ Fully parallel, lock-free |
|
||||
| HIT RRPV update | Any compute thread | ✅ Single-byte write, no sync needed |
|
||||
| refCount increment/decrement | Any compute thread | ✅ Interlocked atomic operations |
|
||||
| Prefetch refresh (eviction + load) | CAS-winner thread | Serial (only one thread enters) |
|
||||
| Dirty writeback | CAS-winner thread | Serial (within eviction flow) |
|
||||
| FlushAllDirty | Single thread after computation | Serial (no concurrency) |
|
||||
|
||||
**Prefetch Refresh CAS Lock**:
|
||||
|
||||
```csharp
|
||||
private int _refreshLock; // 0=free, 1=locked
|
||||
|
||||
private void TryRefreshPrefetch()
|
||||
{
|
||||
if (Interlocked.CompareExchange(ref _refreshLock, 1, 0) != 0)
|
||||
return; // Another thread is doing it; current thread continues computing
|
||||
try { /* eviction + prefetch */ }
|
||||
finally { Volatile.Write(ref _refreshLock, 0); }
|
||||
}
|
||||
```
|
||||
|
||||
**Swiss Table Concurrent Read Safety Guarantee**: The prefetch refresh thread follows strict write ordering
|
||||
on insertion: write Slab data first, then write eviction metadata and callback pointers, `sfence`, then
|
||||
write the Swiss Table control byte and key/value, ensuring that when a compute thread reads the control
|
||||
byte, the data is already fully visible.
|
||||
|
||||
---
|
||||
|
||||
## 4. Operation Flows
|
||||
|
||||
### 4.1 Cache Lookup (Read-Only Hit)
|
||||
|
||||
1. Compute CRC32C hash of the original pointer
|
||||
2. Swiss Table lookup: SIMD scan ctrl → compare key → return cache pointer
|
||||
3. Update RRPV: `_evictionFlags[slot] &= 0xF9` (RRPV zeroed, dirty bit unchanged)
|
||||
4. Construct CacheLine handle (refCount++)
|
||||
5. Return CacheLine; upper-layer code accesses local data via `.Address`
|
||||
|
||||
### 4.2 Cache Lookup (Writable Hit)
|
||||
|
||||
1. Swiss Table lookup hits
|
||||
2. Update RRPV: `_evictionFlags[slot] &= 0xF9`
|
||||
3. Construct CacheLine handle (refCount++)
|
||||
4. Return CacheLine; upper-layer code reads/writes the cache line via `.Address` (e.g., partial sum +=)
|
||||
|
||||
### 4.3 Cache Miss (Read-Only)
|
||||
|
||||
1. Swiss Table lookup misses
|
||||
2. Compute thread enters `Thread.Sleep(0)` spin loop
|
||||
3. Wait for Predictor-driven prefetch refresh to load the line into L4
|
||||
4. Exit wait loop when Swiss Table shows a hit
|
||||
5. Construct CacheLine handle and return
|
||||
|
||||
Under normal operation, the Predictor always leads the execution point by 1024+ steps,
|
||||
misses occur only during cold start or extreme bursts.
|
||||
|
||||
### 4.4 Cache Miss (Writable)
|
||||
|
||||
1. Swiss Table lookup misses
|
||||
2. Allocate a cache line from Slab
|
||||
3. If Slab has no free slots → trigger eviction (see 4.6)
|
||||
4. Zero-initialize the cache line (`Unsafe.InitBlock(dest, 0, 8192)`)
|
||||
5. Register writeback callbacks: `_preprocess[slot]`, `_writeback[slot]`, `_callbackCtx[slot]`
|
||||
6. Set eviction flags: `dirty=1, RRPV=0`
|
||||
7. Write `_originKeys[slot]`
|
||||
8. `sfence`
|
||||
9. Write Swiss Table mapping
|
||||
10. Construct CacheLine handle and return
|
||||
|
||||
### 4.5 Prefetch Refresh
|
||||
|
||||
Triggered by compute threads in `StepComplete()`; CAS lock ensures single-thread execution:
|
||||
|
||||
1. Synchronize refFlags snapshot (`_refCounts[i] > 0 → _refFlags[i] = 1`)
|
||||
2. Retrieve pending prefetch entries from Predictor (up to 64 at a time)
|
||||
3. For each entry:
|
||||
- Swiss Table hit → mark RRPV=0 (hot-line protection)
|
||||
- Swiss Table miss → execute load:
|
||||
a. Allocate cache line from Slab
|
||||
b. If Slab has no free slots → trigger eviction (see 4.6)
|
||||
c. Execute Pack reorder or memcpy based on `NeedsRepack` flag
|
||||
d. Write eviction metadata (originKey, RRPV=0, dirty=0, callbacks=null)
|
||||
e. `sfence`
|
||||
f. Write Swiss Table mapping
|
||||
|
||||
### 4.6 Cache Eviction (SIMD Scan + Writeback)
|
||||
|
||||
```
|
||||
Eviction scan pseudocode:
|
||||
|
||||
Pass 1: Find refCount==0, RRPV==3, dirty==0 (zero-cost eviction)
|
||||
for i in range(scanCursor, CacheLineCount), step=16:
|
||||
refChunk = SSE2.Load(refFlags + i)
|
||||
refFreeMask = SSE2.MoveMask(SSE2.CompareEqual(refChunk, zero))
|
||||
|
||||
evChunk = SSE2.Load(evictionFlags + i)
|
||||
// RRPV==3 and dirty==0 → byte & 0x07 == 0x06
|
||||
masked = SSE2.And(evChunk, 0x07)
|
||||
coldCleanMask = SSE2.MoveMask(SSE2.CompareEqual(masked, 0x06))
|
||||
|
||||
candidates = refFreeMask & coldCleanMask
|
||||
while candidates != 0:
|
||||
bit = TrailingZeroCount(candidates)
|
||||
idx = i + bit
|
||||
if originKeys[idx] not in Predictor.ActiveKeys:
|
||||
evict(idx) // No writeback, discard directly
|
||||
return
|
||||
candidates &= candidates - 1
|
||||
|
||||
Pass 2: Find refCount==0, RRPV==3, dirty==1 (needs writeback)
|
||||
Same as above, but match byte & 0x07 == 0x07
|
||||
evict triggers writeback callbacks
|
||||
|
||||
Pass 3: Increment all RRPV+1 (aging), retry
|
||||
Up to 3 rounds
|
||||
```
|
||||
|
||||
**逐出执行**：
|
||||
|
||||
```csharp
|
||||
private void Evict(int slot)
|
||||
{
|
||||
byte flags = _evictionFlags[slot];
|
||||
|
||||
// Dirty writeback
|
||||
if ((flags & 0x01) != 0)
|
||||
{
|
||||
nuint origin = _originKeys[slot];
|
||||
nuint cached = SlotAddressFromIndex(slot);
|
||||
nuint ctx = _callbackCtx[slot];
|
||||
|
||||
// 1. Pre-processing (optional)
|
||||
var preprocess = (delegate* unmanaged<nuint, nuint, void>)_preprocess[slot];
|
||||
if (preprocess != null)
|
||||
preprocess(cached, ctx);
|
||||
|
||||
// 2. Writeback
|
||||
var writeback = (delegate* unmanaged<nuint, nuint, nuint, void>)_writeback[slot];
|
||||
if (writeback != null)
|
||||
writeback(origin, cached, ctx);
|
||||
}
|
||||
|
||||
// Standard eviction
|
||||
_map.TryRemove(_originKeys[slot], out _);
|
||||
_slab.Free(SlotAddressFromIndex(slot));
|
||||
_evictionFlags[slot] = 0;
|
||||
_originKeys[slot] = 0;
|
||||
_preprocess[slot] = 0;
|
||||
_writeback[slot] = 0;
|
||||
_callbackCtx[slot] = 0;
|
||||
}
|
||||
```
|
||||
|
||||
### 4.7 显式刷新
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Explicitly writes back the specified dirty cache line. Called after all segments complete.
|
||||
/// </summary>
|
||||
public void FlushDirty(nuint originKey)
|
||||
{
|
||||
if (_map.TryGet(originKey, out nuint addr))
|
||||
{
|
||||
int slot = SlotIndex(addr);
|
||||
if ((_evictionFlags[slot] & 0x01) != 0)
|
||||
{
|
||||
// Execute pre-processing + writeback
|
||||
InvokeWriteBack(slot);
|
||||
_evictionFlags[slot] &= 0xFE; // Clear dirty bit
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Writes back all dirty cache lines in bulk. Called after entire matrix computation completes.
|
||||
/// </summary>
|
||||
public void FlushAllDirty()
|
||||
{
|
||||
for (int i = 0; i < CacheLineCount; i++)
|
||||
{
|
||||
if ((_evictionFlags[i] & 0x01) != 0 && _originKeys[i] != 0)
|
||||
{
|
||||
InvokeWriteBack(i);
|
||||
_evictionFlags[i] &= 0xFE;
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4.8 Working Mode Switch
|
||||
|
||||
```csharp
|
||||
public enum L4Mode
|
||||
{
|
||||
/// <summary>
|
||||
/// Pass-through mode: Map() returns the original pointer directly; L4 does not intervene.
|
||||
/// Suitable for single-NUMA + no-repack scenarios (e.g., vector addition, scalar multiplication).
|
||||
/// </summary>
|
||||
StreamTransparent,
|
||||
|
||||
/// <summary>
|
||||
/// Buffer mode: full lookup/Pack/cache/eviction pipeline.
|
||||
/// Suitable for FMA scenarios requiring repacking or multi-NUMA scenarios.
|
||||
/// </summary>
|
||||
CacheBuffer
|
||||
}
|
||||
```
|
||||
|
||||
Mode is determined at construction time and does not change at runtime. Decision logic:
|
||||
|
||||
- Requires Pack reordering (FMA core rows of matrix A) → CacheBuffer
|
||||
- Multiple NUMA nodes → CacheBuffer
|
||||
- Requires segment accumulation → CacheBuffer
|
||||
- Single NUMA + no reordering + no segment → StreamTransparent
|
||||
|
||||
---
|
||||
|
||||
## 5. Segment Accumulation Scenario
|
||||
|
||||
### 5.1 Problem
|
||||
|
||||
For matrix multiplication `C = A × B` when the K dimension is large, segmented accumulation is required:
|
||||
|
||||
```
|
||||
segment 0: C_partial = A[kr, 0..KC] × B[0..KC, kc]
|
||||
segment 1: C_partial += A[kr, KC..2KC] × B[KC..2KC, kc]
|
||||
...
|
||||
segment N: C_partial += A[kr, N*KC..K] × B[N*KC..K, kc]
|
||||
|
||||
Final: C[kr, kc] = Σ all segment partial sums
|
||||
```
|
||||
|
||||
If each segment's result is `+=` directly to the C matrix,
|
||||
and C resides on a remote NUMA node, every segment incurs cross-node write latency.
|
||||
|
||||
### 5.2 L4 Solution
|
||||
|
||||
Temporarily store C's partial sums in L4's writable cache lines:
|
||||
|
||||
```
|
||||
segment 0: c = l4.MapWritable(C_key, null, &AccumulateAdd)
|
||||
// First miss → allocate zero-initialized cache line
|
||||
FMA(c.Address, a.Address, b.Address) // c += a*b
|
||||
c.Dispose()
|
||||
|
||||
segment 1: c = l4.MapWritable(C_key, null, &AccumulateAdd)
|
||||
// Hit! Returns existing partial sum cache line
|
||||
FMA(c.Address, a.Address, b.Address) // c += a*b
|
||||
c.Dispose()
|
||||
|
||||
... after all segments complete ...
|
||||
|
||||
l4.FlushDirty(C_key)
|
||||
// Calls AccumulateAdd(C_origin, cacheline, ctx)
|
||||
// → Adds local partial sum back to remote C matrix in one shot
|
||||
```
|
||||
|
||||
### 5.3 Benefits
|
||||
|
||||
```
|
||||
Without segment buffering (OpenBLAS approach):
|
||||
Each segment writes C matrix across NUMA
|
||||
Remote writes per output tile = Zk times
|
||||
Per-write latency = ~200-300ns (cross-node)
|
||||
|
||||
With L4 segment buffering:
|
||||
All segment += operations complete locally in L4 cache lines
|
||||
Partial sums likely remain in L1/L2 (8KB << L1 capacity)
|
||||
Remote writes per output tile = 1 (final FlushDirty)
|
||||
Writeback uses AVX2-vectorized stores: 8192B = 32 × 256-bit stores
|
||||
|
||||
Reduction ratio = Zk : 1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Uniqueness and Advantages
|
||||
|
||||
### 6.1 Deterministic Prefetch (Windowed OPT)
|
||||
|
||||
Essential difference from hardware cache replacement policies (LRU, RRIP, Hawkeye):
|
||||
|
||||
| Policy | Information Source | Prediction Method | Accuracy |
|
||||
|---------------------|-------------------------|-------------------------------|---------------------------|
|
||||
| LRU | Past access order | Least recently used | Moderate |
|
||||
| RRIP | Insert/hit markers | Predict reuse distance | Moderate-High |
|
||||
| Hawkeye | OPTgen retrospection | Learn past to infer future | High |
|
||||
| **L4 Predictor** | **Future task queue** | **Direct future visibility** | **Perfect** (within window) |
|
||||
|
||||
L4's Predictor is a **deterministic Oracle** — it does not predict; it knows.
|
||||
Because matrix computation task sequences are fully determined at scheduler allocation time.
|
||||
|
||||
### 6.2 NUMA Adaptability
|
||||
|
||||
The same code automatically gains NUMA benefits on different hardware configurations:
|
||||
|
||||
| Hardware Configuration | L4 Behavior |
|
||||
|-------------------------|-------------------------------------------------------------|
|
||||
| Single-socket CPU | Pack cache + segment accumulation buffer |
|
||||
| Dual-socket CPU | Pack cache + NUMA migration cache + segment accum. |
|
||||
| Hybrid-architecture CPU | Pack cache + cross-domain migration cache + segment accum. |
|
||||
|
||||
### 6.3 Callback-Driven Writeback Strategy
|
||||
|
||||
Using native function pointers for writeback avoids:
|
||||
|
||||
- Virtual method call overhead
|
||||
- Interface dispatch overhead
|
||||
- Delegate GC pressure
|
||||
- Hard-coding specific writeback logic
|
||||
|
||||
Callers register `preprocess` and `writeback` callbacks at `MapWritable` time,
|
||||
L4 invokes them directly through function pointers on eviction or Flush,
|
||||
**zero managed overhead, zero virtual dispatch, performance equivalent to C function calls**.
|
||||
|
||||
### 6.4 Reference Count Protection
|
||||
|
||||
Through `CacheLine` handle reference counting, cache lines actively in use by compute threads are
|
||||
absolutely never evicted (including writeback not being triggered). The SIMD eviction scan skips
|
||||
lines with `refCount > 0` immediately via the `_refFlags` array, without checking linked lists or acquiring locks.
|
||||
|
||||
### 6.5 Cache Line Size Aligned to Matrix Kernel
|
||||
|
||||
The 8192B cache line size is not arbitrary:
|
||||
|
||||
- 1 `double MAT_KERNEL` (4×4 double matrix) = 128 Bytes
|
||||
- 1 cache line = 64 `MAT_KERNEL` units
|
||||
- Matrix multiplication micro-kernels operate on data sizes perfectly aligned to cache lines
|
||||
- Compute kernel functions operate on cache lines as units; no cross-cache-line accesses occur
|
||||
|
||||
### 6.6 Fixed-Capacity Determinism
|
||||
|
||||
- Slab allocator never expands → memory footprint upper bound is known (64 MB)
|
||||
- SIMD eviction scan → latency upper bound is known (at most 3 aging rounds × 512 SIMD comparisons)
|
||||
- Swiss Table fixed 16,384 slots → lookup latency upper bound is known
|
||||
- Writeback callback latency is controlled by the caller → predictable
|
||||
- Worst-case L4 latency = lookup + eviction (incl. writeback) + allocation + memcpy, all statically computable
|
||||
|
||||
### 6.7 Reuse of Existing StgSharp Infrastructure
|
||||
|
||||
| L4 Component | Reused StgSharp Infrastructure |
|
||||
|-----------------------|-------------------------------------------------------|
|
||||
| Mapping table | `SwissTable` (implemented and benchmark-validated) |
|
||||
| Mapping table SIMD | `IBucketScanner` + `SIMDID` architecture detection |
|
||||
| Cache line alloc | `SlabAllocator.FromBuffer<T>()` |
|
||||
| Hash computation | Hardware `CRC32C` instruction |
|
||||
| Memory allocation | `NativeMemory.AlignedAlloc` |
|
||||
| Future multi-platform | `IBucketScanner` + NRGA compilation pipeline |
|
||||
|
||||
### 6.8 Originality Analysis
|
||||
|
||||
After surveying existing mainstream numerical computation libraries, L4's design has
|
||||
**no direct precedent or equivalent in known public implementations**.
|
||||
|
||||
#### NUMA Strategy Comparison with Existing Libraries
|
||||
|
||||
| Library / Framework | NUMA Strategy | Software Cache Layer | Mapping Table | Eviction Policy | Dirty Writeback |
|
||||
|---------------------|-------------------------------------|---------------------|-----------------|-------------------------|--------------------|
|
||||
| Intel MKL | Thread pinning + `numactl` | None | None | None | None |
|
||||
| OpenBLAS | Thread affinity + block alignment | None | None | None | None |
|
||||
| BLIS | Multi-level block alignment L1/L2/L3| None | None | None | None |
|
||||
| Eigen | No explicit NUMA handling | None | None | None | None |
|
||||
| cuBLAS / rocBLAS | GPU memory (hardware-solved) | None | None | None | None |
|
||||
| **StgSharp L4** | **Software-managed thread-local cache** | **Yes** | **Swiss Table** | **RRPV + windowed OPT** | **Callback-driven** |
|
||||
|
||||
---
|
||||
|
||||
## 7. 从 v0.1 到 v0.3 的变更摘要
|
||||
|
||||
| 项目 | v0.1 | v0.3 |
|
||||
|-----------------|-------------------------------|---------------------------------------|
|
||||
| 逐出数据结构 | 4 级环形双向链表 | RRPV 2-bit 掩码 + 8KB SoA 数组 |
|
||||
| 逐出扫描方式 | 链表头指针遍历 | SSE2 批量模式匹配（16 行/周期） |
|
||||
| 热���更新 | 链表节点摘插（4 指针写） | 单字节位运算 `&= 0x03` |
|
||||
| 老化操作 | 逐节点移链表 | SIMD 批量 +1 |
|
||||
| 引用保护 | 无 | refCount > 0 时 SIMD 扫描跳过 |
|
||||
| 预取策略 | 无 | Predictor 窗口 OPT（2048 条缓冲） |
|
||||
| 线程模型 | L4 服务线程 + Command Queue | 无额外线程，CAS 抢锁 |
|
||||
| 命令队列 | 链式 CommandCache 分段 | 不需要（预取刷新替代命令驱动） |
|
||||
| 缓存行元数据 | Slab 分配的 CacheLineHead 结构 | SoA 平行数组 + 函数指针数组 |
|
||||
| 工作模式 | 单一模式 | StreamTransparent / CacheBuffer 切换 |
|
||||
| 写回策略 | 无（只读缓存） | dirty 位 + preprocess/writeback 回调 |
|
||||
| 写回机制 | 无 | 本机函数指针，零托管开销 |
|
||||
|
||||
---
|
||||
|
||||
## 8. 未来工作
|
||||
|
||||
- [ ] L4 根类型重构（组合 Swiss Table、Slab、SoA 元数据、Predictor）
|
||||
- [ ] Predictor 与矩阵乘法调度器的集成
|
||||
- [ ] Segment 累加写回的 AVX2 向量化实现
|
||||
- [ ] SIMD 逐出扫描的 AVX2 升级（32 行/周期）
|
||||
- [ ] Swiss Table ARM NEON 实现
|
||||
- [ ] Swiss Table AVX-512 升级（64 槽扫描从 16 次降至 4 次）
|
||||
- [ ] 墓碑累积监控与自动 rehash 策略
|
||||
- [ ] 性能基准测试（NUMA 本地 vs 远端访问延迟对比）
|
||||
- [ ] Segment 累加 vs 直接写回的 benchmark 对比
|
||||
- [ ] 与逐线程矩阵上下文的集成验证
|
||||
- [ ] 双路 6242R 环境下的 NUMA 搬运性能验证
|
||||
- [ ] 远期：PPC64LE 平台的 VSX 原生 Swiss Table 扫描支持
|
||||
|
||||
---
|
||||
|
||||
## 9. 参考
|
||||
|
||||
- Google Abseil Swiss Tables Design Notes
|
||||
- HP Labs RRIP (Re-Reference Interval Prediction) 论文
|
||||
- Hawkeye: Efficient Cache Replacement via OPTgen (ISCA 2017)
|
||||
- StgSharp HLSF Allocator 设计文档
|
||||
- StgSharp SIMD Architecture (`SIMDID` 体系)
|
||||
- StgSharp NRGA Roadmap
|
||||
- StgSharp `SlabAllocator<T>` 实现
|
||||
- StgSharp `SwissTable` 实现及 Benchmark
|
||||
Reference in New Issue
Block a user