Skip to content

fix(mem): buddy leak, split/coalesce race and fragmentation on MCUs - #103

Merged
gg582 merged 5 commits into
mainfrom
fix/buddy-fragmentation
Oct 26, 2026
Merged

gg582 merged 5 commits into
mainfrom
fix/buddy-fragmentation

Conversation

@gg582

@gg582 gg582 commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

Summary

#102 had to shrink test_sharded_sched to 16 outstanding futures on MCUs, because the 64 KiB buddy pool ran out at 64. I wrote that off as fragmentation, but it turned out to be mostly a leak plus a race, with per-object overhead and LIFO reuse on top. All four are fixed here, and the test holds 64 futures on hardware again.

Commit Problem Fix
fix(buddy) ttak_mem_buddy_free() deferred through ttak_epoch_retire(), which frees only on ttak_epoch_reclaim(), and nothing calls that on bare metal. Freed buddy blocks never came back. Each retire node also took a 128 B pool block. A 200 B alloc/free loop ran out after 102 rounds. Free at once, as ttak_dangerous_free() already does on hosted builds (free()).
fix(buddy) Freeing at once exposed a race. There was one lock per order tier, but split and coalesce touch two orders. A block being split keeps in_use == 0 until the split ends, so a concurrent free could merge it and hand the same memory out twice. Under FreeRTOS this showed up as a configASSERT on a future's mutex handle that had been overwritten. Take one lock for alloc, free and defrag.
fix(epoch) Every thread that entered an epoch allocated a state and a list node that were never freed (the known leftover from #102). They piled up for every thread ever created and pinned blocks across the pool. When a thread exits, its node is marked free and the next thread to register reuses it. Memory now follows the peak number of live threads.
feat(mem) ttak_mem_header_t was 64 B aligned, so 128 B. Add the buddy header and a 20 B future took a 256 B block. The VMA block header (20 B) also sat in a slot sized for the 128 B mem header. On bare metal, headers and payloads are 8 B aligned via TTAK_MEM_HEADER_ALIGN: the mem header shrinks to 88 B and the buddy header to 24 B. Hosted builds keep 64. VMA uses its real block-header size.
fix(mem) (TinyCC) TinyCC 0.9.27 compiles as C99, so <stdalign.h> turns _Alignas into an attribute, which TinyCC ignores on struct members. This was already broken: the buddy header was 32 B and ttak_mem_header_t lost its alignment, so payloads were misaligned. The new alignment test caught it. Pad the buddy header by hand, align segment starts inside the allocator, and use TinyCC's built-in _Alignas keyword.
feat(buddy) LIFO free lists scattered small objects. When the 1 KiB allocation failed, 40 KiB was free but only in 256 B and 512 B pieces. Static pools (anti_fragmentation) keep free lists sorted by address. Add ttak_mem_buddy_get_stats() (capacity, bytes in use, largest free block).

Behaviour changes

  • On bare metal, ttak_mem_alloc() now returns pointers aligned to 8 B, not 64 B. For DMA buffers on Cortex-M7 cache lines, build with -DTTAK_MEM_HEADER_ALIGN=32 (documented in the README; tested with 32/32).
  • Hosted builds don't use the buddy unless EMBEDDED=1. Their header size, alignment and allocation paths are unchanged.

Measurements (QEMU FreeRTOS, test_sharded_sched with 64 futures, 20 runs each)

Stage Passed
Leak fix only 19/40, plus intermittent configASSERT failures and timeouts (the race)
+ single lock 0 asserts or timeouts in 40 runs; the remaining failures were ENOMEM
+ address-ordered free lists 15/20
+ epoch node reuse 17/20
+ 8 B headers 20/20

Testing

Target Result
STM32F769I-DISCO + FreeRTOS 38/38, plus 5 repeats of test_sharded_sched and test_buddy
STM32F769I-DISCO bare metal 32/32
QEMU mps2-an500 FreeRTOS / bare metal 38/38 / 32/32. 20 runs each of test_sharded_sched (64 futures) and test_buddy, all passing
QEMU bare metal, TTAK_MEM_HEADER_ALIGN=32 32/32
Hosted gcc / clang / tcc 44/44 each (tcc after the TinyCC fix)
Hosted ASan+UBSan / TSan clean for test_buddy, test_future_release, test_sharded_sched, test_thread, test_epoch_grace, test_pool (TSan: test_buddy, test_epoch_grace)
Every commit builds hosted and bare metal

test_buddy is new. It covers the leak, coalescing, payload alignment, a concurrent split/coalesce churn, and epoch-state reuse. Run against the old allocator, it fails at allocation 128 with the old free, and segfaults 5 times out of 5 with per-tier locks.

gg582 added 5 commits October 26, 2026 20:14
ttak_mem_buddy_free() deferred every block through ttak_epoch_retire(),
which only frees on ttak_epoch_reclaim(). Nothing calls that on bare
metal, so freed blocks never came back, and each retire node took a
pool block of its own: a 200-byte alloc/free loop ran the 64 KiB pool
dry after ~100 rounds. Free at once, as ttak_dangerous_free() already
does on hosted builds.

Freeing at once exposed a race the leak had hidden. The free lists were
guarded by one lock per order tier, but splitting a block and coalescing
its buddy touch two orders. A block being split keeps in_use == 0 until
the split ends, so a concurrent free could merge it as a free buddy and
hand the same memory out twice (FreeRTOS asserted on a future mutex
handle that had been overwritten). Take one lock for alloc, free and
defragmentation.
Every thread that entered an epoch allocated a state and a list node
that were never freed, since reclaim walks the list without a lock. On a
static pool these piled up for each thread ever created and pinned
blocks across the pool. Mark a node free when its thread deregisters,
or, without native TLS, when the thread exits, and let the next thread
to register take it over. Memory now follows the peak number of live
threads.
ttak_mem_header_t was 64-byte aligned everywhere, padding it to 128
bytes; with the 64-byte buddy header a 20-byte future took a 256-byte
block. Cache-line alignment buys nothing on a Cortex-M, so bare-metal
builds now align headers (and payloads) to 8 bytes via
TTAK_MEM_HEADER_ALIGN: the header shrinks to 88 bytes and the buddy
header to 24. Hosted builds keep 64.

The VMA region also placed its 20-byte block header in a slot sized for
ttak_mem_header_t, wasting 64-108 bytes per allocation.
LIFO free lists reused whichever block was freed last, scattering small
objects across the pool until no large block could coalesce: with 64
futures outstanding, test_sharded_sched failed a 1 KiB allocation while
~40 KiB sat free in 256- and 512-byte pieces. Static pools
(anti_fragmentation) now keep each list sorted by address, packing
allocations toward the low end.

ttak_mem_buddy_get_stats() reports capacity, bytes in use and the
largest free block. test_buddy covers the leak, coalescing, payload
alignment, concurrent split/coalesce and epoch-state reuse;
test_sharded_sched holds 64 futures on bare metal again.
TinyCC 0.9.27 compiles as C99, so <stdalign.h> turns _Alignas into an
aligned attribute that it ignores on struct members and static arrays.
The buddy header shrank to 32 bytes and ttak_mem_header_t lost its
alignment, so buddy and ttak_mem_alloc() payloads were misaligned (the
new test_payload_alignment caught it). Pad the buddy header by hand,
align each segment start inside the allocator, and drop the
<stdalign.h> macro under TinyCC so its built-in _Alignas applies.
@gg582
gg582 merged commit 613ed8f into main Oct 26, 2026
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant