youmuliu/stgsharp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tlsf is ok, and fast

Browse Source
This commit is contained in:
Youmu Liu
2026-05-13 15:08:30 +08:00
parent bf9879f59f
commit ae57de4040
22 changed files with 409 additions and 4619 deletions
Download Patch File Download Diff File Expand all files Collapse all files
StgSharp.Common/HighPerformance/Memory/InroductionToHLSF.md
-137
View File
@@ -1,137 +0,0 @@
# HLSF (Hybrid Layer Segregated Fit) Allocator
A constant-time (O(1)) allocator for real-time, high-throughput scenarios. It uses layered size classes with segmented buckets and stores all metadata externally for clean user memory and better cache locality.
## Key Points
- O(1) allocate/free with 10 levels: 64B x 16MB; each level split into 1x/2x/3x segments
- External metadata (`Entry`, `BucketNode`) in dedicated SLABs; user blocks have no headers
- Configurable alignment (>=16B); maximum block size is configurable at construction time via the `maxBlockSize` parameter (default 32MB, 10 levels). The level array and bucket table are dynamically extended when a larger `maxBlockSize` is specified. There is no implicit fallback to `NativeMemory` for oversized requests.
- Single-threaded implementation (external sync required for multi-threaded use)
## Quick Use
```csharp
using var hlsf = new HybridLayerSegregatedFitAllocator(64 * 1024 * 1024);
var h = hlsf.Alloc(1024);
Span<byte> s = h.BufferHandle; s[0] = 42;
hlsf.Free(h);
```
## Core APIs
### Alloc(uint size)
- Purpose: Allocate a block of at least `size` bytes.
- Input: `size` in bytes. Must not exceed the `maxBlockSize` configured at construction time (default 32MB). If `size > _maxSize`, an `InvalidOperationException` is thrown.
- Output: `HybridLayerSegregatedFitAllocationHandle` with pointer access and `Span<byte>`.
- Behavior:
- Classify into level/segment and pop a matching bucket; escalate segments and levels if empty.
- Fallback to overflow bucket or carve 32MB from spare memory if all buckets are empty.
- Mark selected `Entry` as Alloc; slice any remainder into `Empty` entries and push to buckets.
- There is no implicit fallback to `NativeMemory.AlignedAlloc` for oversized requests — all allocations are served from the pre-allocated arena. The level array and bucket table are dynamically sized based on the `maxBlockSize` parameter.
### Free(HybridLayerSegregatedFitAllocationHandle handle)
- Purpose: Free a previously allocated block.
- Behavior:
- Mark the corresponding `Entry` as `Empty`.
- Merge with adjacent `Empty` entries in both directions (boundary and level constraints).
- Remove merged neighbors from buckets, update position chain, and optionally promote level.
- Re-enqueue the final `Entry` into the appropriate bucket.
### Collect()
- Purpose: Reclaim and rebalance free entries without touching allocated blocks.
- Behavior:
- Consolidate fragmented regions where possible and improve bucket coverage.
- Intended to be cheap; safe to call between benchmark phases.
### BenchmarkDotNet (HLSF vs Libc)
Environment: .NET 8.0.23 (Release), x64 RyuJIT AVX2, Windows 11, AMD Ryzen 7 6800H.
| Method | Mean | StdDev | Ratio | Allocated |
|------- |-----:|------:|------:|----------:|
| Hlsf | 838.4 µs | 40.6 µs | 1.00 | 64 B |
| Libc | 849.8 µs | 100.2 µs | 1.01 | 64 B |
Interpretation: HLSF matches libc allocation throughput under the same workload, with slightly tighter dispersion in this run.
> Note: BenchmarkDotNet also warned that the minimum iteration time was under 1ms; for more stable statistics, increase per-iteration work (e.g., higher `AllocationCount`).
## Detailed Design (summary)
-
- Size classes: The default level size array is [64, 256, 1024, 4096, 16384, 65536, 262144, 1048576, 4194304, 16777216] (10 levels, 64B × 4^level). When `maxBlockSize` is specified at construction and exceeds the default range, additional levels are appended (each 4× the previous) to cover the requested maximum block size.
- Segments per level: 1x, 2x, 3x
- Buckets: index = 3 * level + segment (overflow bucket for level > 9)
- Metadata: Entry (external SLAB, position chain), BucketNode (user region, bucket chain)
- Position chain: circular doubly linked list anchored by spare sentinel
- Allocation: search buckets by level/segment with escalation; if none, carve 32MB from spare; set Alloc; slice remainder into buckets
- Slicing: align to level boundaries, pack contiguous blocks forward/backward to minimize overhead
- Free/Merge: bidirectional merge under boundary and level constraints; update buckets; optional level promotion
- Alignment: default 16B; internal blocks aligned by slicing/edge calculation
- Complexity: average O(1) with bounded levels and constant-time bucket ops
- `Entry` (external SLAB): linkage in position chain, `Position`, `Size`, `Level`, `State`
- Invariants: near neighbors must form a consistent double-linked ring; `Level` setter validates size vs. level window
- `BucketNode` (in user region): `EntryRef`, `NextLevel`, `PreviousLevel`, `IsInBucket`
- Position chain: circular doubly linked list anchored by sentinel `_spareMemory` (`State.Spare`)
### Allocation Flow
1) Compute `level` and `segment` from requested `size`
2) Try pop bucket (`TryPopLevelForAllocation(level, segment, out entry)`) and escalate:
- Advance `segment` 0��1��2, then `level++`, until `MaxLevel`
- If still empty: try pop from overflow bucket (L=10, seg=1) or `TryAllocateNew32MBBlock`
3) On success: bound-check the slice offset; set `entry` to `Alloc` using `SetEntryLevel`
4) If remainder exists in the source region, call `SliceMemory(entry->NextNear, remainder)` to produce free chunks into buckets
### Slicing Logic (`SliceMemory`)
- Inputs: `next` entry position, `sizeToSlice`
- Compute `offset` and `end` relative to base buffer; pick starting level by trailing-zero count of `offset`:
- `i = (BitOperations.TrailingZeroCount(offset) - 6) / 2`, clamped to `�� MaxLevel`
- Three phases to cover [offset, end):
1) Forward levels i..N-1: for each level size `size` and `upper=size*4`, align to `edge=((offset+upper-1)/upper)*upper`, pack `count = ((min(edge,end) - offset) / size)` blocks
2) If `i > MaxLevel`: pack in largest upper window on tail
3) Backward i-1..0: fill remaining tail with smaller sizes
- For each packed region: allocate new `Entry`, insert before `next`, mark `Empty`, and push corresponding `BucketNode` into the `(level, segment)` bucket (`PushBucketToLevel(i, count-1, b)`)
### Free and Merge (`MergeAndRemoveFromBuckets`)
- Given an `Entry` to free (turned `Empty` by caller):
- Merge left while neighbor `p` is `Empty`, same `Level`, and within boundary (`AssertBoundary`)
- Remove `p` from bucket (`RemoveFromBucket`), expand current, fix `Position`, unlink `p`
- Merge right with neighbor `n` by the same rule
- Optionally promote `Level` if `Size �� levelSizeArray[originLevel] * 4`
- Ensure `BucketNode.EntryRef` points to the merged entry; caller re-enqueues into the proper bucket
### Buckets
- `PushBucketToLevel(level, segment, node)`: push to bucket head, set `IsInBucket`, maintain prev/next
- `TryPopLevel(level, segment, out node)`: pop head; if empty, return false
- `RemoveFromBucket(level, segment, node)`: unlink from middle or use `TryPopLevel` if it is the head
### Large Allocations and Spare Memory
- Requests exceeding the configured `maxBlockSize` are rejected with `InvalidOperationException`. All allocations are served from the pre-allocated arena.
- When all buckets are empty, `TryAllocateNew32MBBlock` carves a region (up to 32MB) from `_spareMemory` to replenish the bucket system, builds an `Empty` entry, and advances the spare pointer
### Alignment
- Default `_align = 16`; `AlignOfLevel(level)` returns `min(levelSizeArray[level], _align)`
- Internal blocks align to level boundaries by design of slicing/edge calculations
### Complexity and Invariants
- Allocate/free are O(1) on average with bounded levels (��10) and constant-time bucket ops
- Invariants guarded by exceptions (e.g., `HLSFPositionChainBreakException`) and pointer consistency checks
## Profiling (hotspots, current workloads)
- SliceMemory ~22% (self ~17%) �� slicing and bucket enqueue dominate allocation path
- MergeAndRemoveFromBuckets ~13% (self ~12%) �� merge + removal dominate free path
- Collect ~7.6% (self ~7.5%) �� low overall cost
Focus: reduce per-slice math and bucket ops; add fast-paths for aligned regions; batch/lazy removals after merges.
StgSharp.Common/HighPerformance/Memory/InroductionToTLSF.md
+159
View File
@@ -0,0 +1,159 @@
# TLSF (Two-Layer Segregated Fit) Allocator
This document describes the current StgSharp native arena allocator, `StgSharp.HighPerformance.Memory.TwoLayerSegregatedFitAllocator`. The file name is historical; the implementation described below is TLSF rather than the older HLSF allocator.
## Overview
- Single contiguous native arena allocated up front
- Headerless returned blocks; allocator metadata is stored in 32-byte nodes carved from the tail of the same arena
- Requests rounded up to 64-byte granularity, with 64 bytes as the minimum block size
- Free blocks grouped by coarse size level derived from the highest set bit of the block size
- Exact-level allocation uses first-fit traversal inside the selected bucket; higher levels fall back to the head of the first non-empty bucket
- `Free` performs bounded local coalescing; `Collect` performs full free-run coalescing across the arena
- Single-threaded implementation; concurrent access requires external synchronization
- No arena growth and no implicit fallback to `NativeMemory.AlignedAlloc` for oversized requests
## Quick Use
```csharp
using StgSharp.HighPerformance.Memory;
using var tlsf = new TwoLayerSegregatedFitAllocator(64 * 1024 * 1024);
var handle = tlsf.Alloc(1024);
Span<byte> buffer = handle.AsSpan();
buffer[0] = 42;
tlsf.Free(handle);
```
## Constructors
- `TwoLayerSegregatedFitAllocator(nuint byteSize)`
- `TwoLayerSegregatedFitAllocator(nuint byteSize, uint maxBlockSize)`
- `TwoLayerSegregatedFitAllocator(nuint byteSize, int align)`
- `TwoLayerSegregatedFitAllocator(nuint byteSize, int align, nuint maxBlockSize)`
Behavior summary:
- `byteSize` defines the total arena size
- `maxBlockSize` defaults to 32 MB and defines the largest single request accepted by `Alloc` and `TryAlloc`
- `align` is passed to the backing `NativeMemory.AlignedAlloc` call for arena allocation; request sizes inside the arena are still quantized to 64-byte blocks
## Public API
### `Alloc(nuint size)`
- Rounds `size` up to the next 64-byte block, with a minimum of 64 bytes
- Throws `OverflowException` if the normalized size exceeds `maxBlockSize` or if the arena cannot provide a block
- Returns a `Handle` containing block address and block size
- Splits a sufficiently large tail remainder into a new free node and re-buckets that node
### `TryAlloc(nuint size, out Handle handle)`
- Uses the same size normalization and maximum block limit as `Alloc`
- Returns `false` when the request exceeds the configured limit or when the arena is exhausted
- Does not fall back to the system allocator
### `Free(Handle handle, int collectScanWindow = 4)`
- Marks the block as free
- Scans up to `collectScanWindow` neighbors toward lower addresses and then up to `collectScanWindow` neighbors toward higher addresses
- Merges any contiguous free neighbors found during those scans
- Reclassifies the merged block back into the coarse bucket set
### `Collect()`
- Walks the full position chain from low address to high address
- Merges consecutive free runs without the local scan-window limit used by `Free`
- Restores larger reusable spans and improves bucket coverage after fragmentation-heavy phases
### `Dispose()`
- Releases the backing arena through the stored free callback
### `Handle`
- Readonly value type carrying the allocator metadata reference, block address, and block size
- `Address` exposes the native pointer
- `Size` exposes the normalized block size
- `AsSpan(int range = -1)` exposes a span view over the block
- The indexer provides byte-level access by offset
## Allocation Model
### Level Classification
- `GetLevel(size)` maps the normalized size to `57 - BitOperations.LeadingZeroCount(size)` and clamps the result to `_maxLevel`
- Each bucket corresponds to one coarse size class rather than an exact size
- The target bucket can therefore contain blocks of different sizes within the same coarse level
### Allocation Path
1. Normalize the requested size to a 64-byte multiple.
2. Traverse the target bucket and select the first free block whose size is at least the requested size.
3. If the target bucket has no suitable block, pop the head of the first non-empty higher-level bucket.
4. If all buckets are empty, carve a fresh block from the spare region tracked by the arena sentinel.
5. Split the tail remainder when both remainder size and metadata availability allow it.
### Metadata Layout
- `Metadata` is a 32-byte node
- `NextNear` and `PreviousNear` form the address-ordered position chain
- `NextLevel` and `PreviousLevel` form either the bucket chain or the recycled-metadata chain, depending on node state
- The spare sentinel at the end of the arena tracks remaining virgin space, recycled metadata nodes, and position-chain anchors
- Returned blocks remain headerless from the caller's point of view
### Coalescing Model
- `Free` performs bounded local coalescing to keep the hot path short
- `Collect` performs full coalescing of consecutive free runs across the entire position chain
- Recycled metadata nodes are returned to the spare metadata list and reused by later splits or fresh arena chunks
## Complexity and Runtime Characteristics
- `Alloc`: O(1) bucket index calculation plus target-level first-fit traversal; higher-level fallback is O(1)
- `Free`: O(`collectScanWindow`) local merge cost
- `Collect`: O(number of metadata nodes in the position chain)
- Best steady-state behavior appears on repeated small and medium native allocations inside a pre-allocated arena
## BenchmarkDotNet (TLSF vs Libc)
Environment: .NET 8.0.26 (PerformanceDebug), x64 RyuJIT AVX2, Windows 11, AMD Ryzen 7 6800H, BenchmarkDotNet v0.15.8, InProcessEmitToolchain, LaunchCount=1, WarmupCount=6, IterationCount=15, InvocationCount=3.
Workload summary:
- Arena size: 512 MB
- Arena alignment: 64 bytes
- Allocation count per batch: 2048
- Batch count: 256
- Request set before normalization: 64, 100, 200, 400, 1024, 4096 bytes
- Free order: shuffled per batch
- `Free` local merge window: 4
Best observed run under the current harness:
| Method | Mean | Error | StdDev | Ratio | RatioSD | Allocated | Alloc Ratio |
|------- |-----:|------:|-------:|------:|--------:|----------:|------------:|
| Tlsf | 79.11 ns | 5.944 ns | 5.560 ns | 0.65 | 0.09 | - | NA |
| Libc | 123.64 ns | 15.955 ns | 14.144 ns | 1.01 | 0.16 | - | NA |
Worst observed run under the same harness:
| Method | Mean | Error | StdDev | Ratio | RatioSD | Allocated | Alloc Ratio |
|------- |-----:|------:|-------:|------:|--------:|----------:|------------:|
| Tlsf | 82.95 ns | 6.337 ns | 5.617 ns | 0.81 | 0.07 | - | NA |
| Libc | 102.63 ns | 8.590 ns | 7.173 ns | 1.00 | 0.09 | - | NA |
Interpretation:
- The current single-pass bucket selection fast path materially reduces allocator-side latency
- The reported ratio still drifts across repeated compile-and-run cycles because the libc baseline also moves under the current InProcess harness
- Recent repeated runs place the transient best case near 0.65 and the longer-running steady band near 0.75 to 0.80
## Operational Notes
- Maximum single-allocation size is fixed at construction time
- Arena growth is not implemented
- `Collect` is the global defragmentation pass; `Free` only performs bounded local coalescing
- The allocator is intended for explicit native-memory workflows rather than as a drop-in general-purpose heap replacement
StgSharp.Common/HighPerformance/Memory/TwoLayerSegregatedFitAllocator.cs
+204 -90
View File
@@ -29,6 +29,7 @@ using StgSharp.HighPerformance.Memory;
using StgSharp.HighPerformance.ProcessorAbstraction;
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Drawing;
using System.Linq;
using System.Numerics;
@@ -36,6 +37,7 @@ using System.Reflection.Metadata;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading.Tasks;
using System.Xml;
using static StgSharp.HighPerformance.Memory.TwoLayerSegregatedFitAllocator;
namespace StgSharp.HighPerformance.Memory
@@ -162,64 +164,109 @@ namespace StgSharp.HighPerformance.Memory
Metadata* node
)
{
int nextIdx = _spareMemory->NextLevel;
int nodeIdx = (int)(_spareMemory - node);
int level = GetLevel(node->Size);
Metadata* next = &_spareMemory[-nextIdx];
int nextLevelIdx = node->NextLevel;
int prevLevelIdx = node->PreviousLevel;
if (nextIdx == 0)
{
_spareMemory->NextLevel = nodeIdx;
node->NextLevel = 0;
return;
} else
{
// add to free list
next->PreviousLevel = nodeIdx;
_spareMemory->NextLevel = nodeIdx;
node->NextLevel = nextIdx;
node->PreviousLevel = 0;
}
RemoveFromBucket(level, node, nodeIdx);
// remove from position chain
int prevNearIdx = node->PreviousNear;
int nextNearIdx = node->NextNear;
Metadata* prevNear = &_spareMemory[-prevNearIdx];
Metadata* nextNear = &_spareMemory[-nextNearIdx];
prevNear->NextNear = nextNearIdx;
nextNear->PreviousNear = prevNearIdx;
if (nextNearIdx != 0)
{
Metadata* nextNear = &_spareMemory[-nextNearIdx];
nextNear->PreviousNear = prevNearIdx;
} else
{
_spareMemory->PreviousNear = prevNearIdx;
}
node->PreviousNear = 0;
node->NextNear = 0;
node->Position = 0;
node->SizeForSpare = 0;
node->State = AllocationState.SpareOrCollect;
node->PreviousLevel = 0;
// remove from level bucket
if (_bucket[level] == nodeIdx)
int nextIdx = _spareMemory->NextLevel;
if (nextIdx != 0)
{
_bucket[level] = nextLevelIdx;
if (nextLevelIdx != 0)
{
Metadata* nextLevel = &_spareMemory[-nextLevelIdx];
nextLevel->PreviousLevel = 0;
}
} else
{
if (nextLevelIdx != 0)
{
Metadata* nextLevel = &_spareMemory[-nextLevelIdx];
nextLevel->PreviousLevel = prevLevelIdx;
}
if (prevLevelIdx != 0)
{
Metadata* prevLevel = &_spareMemory[-prevLevelIdx];
prevLevel->NextLevel = nextLevelIdx;
}
node->PreviousLevel = 0;
node->NextLevel = 0;
Metadata* next = &_spareMemory[-nextIdx];
next->PreviousLevel = nodeIdx;
}
node->NextLevel = nextIdx;
_spareMemory->NextLevel = nodeIdx;
}
[MethodImpl(MethodImplOptions.AggressiveInlining)]
private void RemoveFromBucket(
Metadata* node,
int nodeIdx
)
{
RemoveFromBucket(GetLevel(node->Size), node, nodeIdx);
}
private void RemoveFromBucket(
int level,
Metadata* node,
int nodeIdx
)
{
int currentLevel = level;
int currentNextLevelIdx = node->NextLevel;
int currentPrevLevelIdx = node->PreviousLevel;
if (_bucket[currentLevel] == nodeIdx)
{
_bucket[currentLevel] = currentNextLevelIdx;
} else if (currentPrevLevelIdx != 0)
{
Metadata* prevLevel = &_spareMemory[-currentPrevLevelIdx];
prevLevel->NextLevel = currentNextLevelIdx;
}
if (currentNextLevelIdx != 0)
{
Metadata* nextLevel = &_spareMemory[-currentNextLevelIdx];
nextLevel->PreviousLevel = currentPrevLevelIdx;
}
node->PreviousLevel = 0;
node->NextLevel = 0;
}
// Assuming node to remove is head f bucket
private void RemoveFromBucketHead(
int level,
Metadata* node,
int nodeIdx
)
{
int currentLevel = level;
int currentNextLevelIdx = node->NextLevel;
// int currentPrevLevelIdx = 0;
if (_bucket[currentLevel] == nodeIdx)
{
_bucket[currentLevel] = currentNextLevelIdx;
}
/*
else if (currentPrevLevelIdx != 0)
{
Metadata* prevLevel = &_spareMemory[-currentPrevLevelIdx];
prevLevel->NextLevel = currentNextLevelIdx;
}
/**/
if (currentNextLevelIdx != 0)
{
Metadata* nextLevel = &_spareMemory[-currentNextLevelIdx];
nextLevel->PreviousLevel = 0;
}
node->PreviousLevel = 0;
node->NextLevel = 0;
}
private bool TryAllocNewChunk(
@@ -228,34 +275,49 @@ namespace StgSharp.HighPerformance.Memory
)
{
int idx = _spareMemory->NextLevel;
ulong spareSize = _spareMemory->Size;
if (idx != 0)
{
if (spareSize < size)
{
meta = null;
return false;
}
meta = &_spareMemory[-idx];
_spareMemory->NextLevel = meta->NextLevel;
int nextSpareIdx = meta->NextLevel;
if (nextSpareIdx != 0)
{
Metadata* nextSpare = &_spareMemory[-nextSpareIdx];
nextSpare->PreviousLevel = 0;
}
_spareMemory->NextLevel = nextSpareIdx;
} else
{
if (spareSize < Metadata.SizeOf + size)
{
meta = null;
return false;
}
_metadataEnd++;
meta = &_spareMemory[-_metadataEnd];
// TODO: correctly set position and size for new chunk
idx = _metadataEnd;
meta = &_spareMemory[-idx];
spareSize -= Metadata.SizeOf;
}
ulong spareSize = _spareMemory->Size;
if (spareSize < Metadata.SizeOf + size)
{
meta = null;
return false;
}
spareSize -= Metadata.SizeOf + size;
spareSize -= size;
meta->Size = size;
meta->Position = _spareMemory->Position;
meta->State = AllocationState.Free;
meta->PreviousLevel = 0;
meta->NextLevel = 0;
_spareMemory->Size = spareSize;
_spareMemory->Position += size;
meta->Size = size;
// insert the new chunk to position chain
int nearIdx = _spareMemory->PreviousNear;
Metadata* near = &_spareMemory[-nearIdx];
near->NextNear = _metadataEnd;
near->NextNear = idx;
meta->PreviousNear = nearIdx;
_spareMemory->PreviousNear = _metadataEnd;
_spareMemory->PreviousNear = idx;
meta->NextNear = 0;
return true;
}
@@ -277,6 +339,44 @@ namespace StgSharp.HighPerformance.Memory
return true;
}
private bool TryTakeBucketEntry(
ulong minimumSize,
out Metadata* entry
)
{
int level = GetLevel(minimumSize);
// first try to find in the target level
{
int idx = _bucket[level];
while (idx != 0)
{
Metadata* candidate = &_spareMemory[-idx];
if (candidate->Size >= minimumSize)
{
RemoveFromBucket(level, candidate, idx);
entry = candidate;
return true;
}
idx = candidate->NextLevel;
}
}
for (int i = level + 1; i < _bucket.Length; i++)
{
int idx = _bucket[i];
if (idx != 0)
{
Metadata* candidate = &_spareMemory[-idx];
RemoveFromBucketHead(i, candidate, idx);
entry = candidate;
return true;
}
}
entry = null;
return false;
}
[StructLayout(LayoutKind.Explicit)]
public readonly struct Handle
{
@@ -324,8 +424,8 @@ namespace StgSharp.HighPerformance.Memory
internal struct Metadata
{
[FieldOffset(24)] private ulong PositionMask;
[FieldOffset(16)] private ulong SizeMask;
[FieldOffset(16)] private ulong PositionMask;
[FieldOffset(24)] private ulong SizeMask;
[FieldOffset(30)] internal AllocationState State;
[FieldOffset(31)] internal sbyte Level;
@@ -367,23 +467,15 @@ namespace StgSharp.HighPerformance.Memory
if (allocSize > _maxAllocSize) {
throw new OverflowException("Requested allocation size exceeds the maximum block size.");
}
int level = GetLevel(allocSize);
Metadata* entry = null;
for (int i = level; i < _bucket.Length; i++)
{
int idx = _bucket[i];
if (idx != 0)
{
entry = &_spareMemory[-idx];
_bucket[i] = entry->NextLevel;
break;
}
}
if (entry is null && !TryAllocNewChunk(allocSize, out entry)) {
if (!TryTakeBucketEntry(allocSize, out Metadata* entry) &&
!TryAllocNewChunk(allocSize, out entry)) {
throw new OverflowException("No available ");
}
int entryIdx = (int)(_spareMemory - entry);
ulong entrySize = entry->Size;
if (entrySize < allocSize) {
throw new OverflowException("Bucket returned an undersized block.");
}
ulong remain = entrySize - allocSize;
// slice tail
@@ -392,18 +484,17 @@ namespace StgSharp.HighPerformance.Memory
int newEntryIdx = (int)(_spareMemory - newEntry);
newEntry->Position = entry->Position + allocSize;
newEntry->Size = remain;
newEntry->State = AllocationState.Free;
int newLevel = GetLevel(remain);
int bucketHeadIdx = _bucket[newLevel];
if (bucketHeadIdx != 0)
{
// if there is already an entry in the bucket, link it to the new entry
Metadata* bucketHead = &_spareMemory[-bucketHeadIdx];
newEntry->NextLevel = bucketHeadIdx;
bucketHead->PreviousLevel = newEntryIdx;
} else
{
newEntry->NextLevel = 0;
}
newEntry->NextLevel = bucketHeadIdx;
newEntry->PreviousLevel = 0;
_bucket[newLevel] = newEntryIdx;
@@ -416,6 +507,9 @@ namespace StgSharp.HighPerformance.Memory
newEntry->PreviousNear = entryIdx;
}
entry->State = AllocationState.Alloc;
if (entry->Position - (ulong)_basePtr > _totalSize) {
throw new OverflowException("Allocated memory address exceeds the maximum block size.");
}
return new Handle(entry, _basePtr);
}
@@ -431,17 +525,34 @@ namespace StgSharp.HighPerformance.Memory
Metadata* current = &_spareMemory[-currentIdx];
if (current->State == AllocationState.Free)
{
// scan next
Ptr64 size = 0;
int nextIdx = current->NextNear;
Metadata* next = &_spareMemory[-nextIdx];
while (nextIdx != 0 && next->State != AllocationState.Free)
while (nextIdx != 0)
{
Metadata* next = &_spareMemory[-nextIdx];
if (next->State != AllocationState.Free)
{
break;
}
size += next->Size;
RecycleNode(next);
nextIdx = next->NextNear;
nextIdx = current->NextNear;
}
if (size > 0) {
if (size > 0)
{
current->Size += size;
RemoveFromBucket(current, currentIdx);
int newLevel = GetLevel(current->Size);
int bucketHeadIdx = _bucket[newLevel];
if (bucketHeadIdx != 0)
{
Metadata* bucketHead = &_spareMemory[-bucketHeadIdx];
bucketHead->PreviousLevel = currentIdx;
}
current->NextLevel = bucketHeadIdx;
current->PreviousLevel = 0;
_bucket[newLevel] = currentIdx;
}
}
currentIdx = current->NextNear;
@@ -493,8 +604,10 @@ namespace StgSharp.HighPerformance.Memory
// recycle to level;
head->Size = size;
head->State = AllocationState.Free;
head->PreviousLevel = 0;
int level = GetLevel(head->Size);
int bucketHeadIdx = _bucket[level];
head->NextLevel = 0;
if (bucketHeadIdx != 0)
{
Metadata* bucketHead = &_spareMemory[-bucketHeadIdx];
@@ -516,25 +629,19 @@ namespace StgSharp.HighPerformance.Memory
h = default;
return false;
}
int level = GetLevel(allocSize);
Metadata* entry = null;
for (int i = level; i < _bucket.Length; i++)
{
int idx = _bucket[i];
if (idx != 0)
{
entry = &_spareMemory[-idx];
_bucket[i] = entry->NextLevel;
break;
}
}
if (entry is null && !TryAllocNewChunk(allocSize, out entry))
if (!TryTakeBucketEntry(allocSize, out Metadata* entry) &&
!TryAllocNewChunk(allocSize, out entry))
{
h = default;
return false;
}
int entryIdx = (int)(_spareMemory - entry);
ulong entrySize = entry->Size;
if (entrySize < allocSize)
{
h = default;
return false;
}
ulong remain = entrySize - allocSize;
// slice tail
@@ -543,6 +650,7 @@ namespace StgSharp.HighPerformance.Memory
int newEntryIdx = (int)(_spareMemory - newEntry);
newEntry->Position = entry->Position + allocSize;
newEntry->Size = remain;
newEntry->State = AllocationState.Free;
int newLevel = GetLevel(remain);
int bucketHeadIdx = _bucket[newLevel];
if (bucketHeadIdx != 0)
@@ -566,6 +674,12 @@ namespace StgSharp.HighPerformance.Memory
entry->NextNear = newEntryIdx;
newEntry->PreviousNear = entryIdx;
}
entry->State = AllocationState.Alloc;
if (entry->Position - (ulong)_basePtr > _totalSize)
{
h = default;
throw new OverflowException("Allocated memory address exceeds the maximum block size.");
}
h = new Handle(entry, _basePtr);
return true;
}