Repository navigation
mm: scale kmalloc with bounded per-CPU magazines - #112
Open
JiTianYu391 wants to merge 1 commit into
Open
JiTianYu391 wants to merge 1 commit into
JiTianYu391 wants to merge 1 commit into
Conversation
On a 5-vCPU KVM guest, malloc(64)+free scales from about 115 ns alone to
2,130 ns with five workers. Prior contention sampling found 70% of slab
cache lock acquisitions contended; pointer_owner also took heap.page_lock
on every free. Reproduce with kbench=3 and five alternating A/B boot pairs.
Give each of the 17 kmalloc size classes a per-CPU magazine capped at
32 objects / 16 KiB. Refill and spill in batches under the slab cache lock,
keep cached objects counted in slab->inuse so shrink cannot reclaim them,
and retain a CAS transition from ALLOCATED to CACHED to reject double free.
Typed/dynamic caches keep their existing constructor/destructor path.
Use stable buddy ownership tags for valid live pointers without taking the
page lock, and aggregate usage counters from the per-CPU front ends.
The new early malloc caller also exposed smp_init publishing cpu_count
before allocating cpus. GDB found smp_init -> aligned_alloc -> malloc ->
heap_cpu_cache -> get_current_cpu_id with cpu_count=5 and cpus=NULL, faulting
at cpus[0].lapic_id (address 8); panic then faults in send_ipi_all. Extend
the BSP fallback until the CPU table exists.
Validation: five alternating A/B boot pairs, three rounds per boot,
median within each boot then median across boots (ns/op):
A B
malloc/64/solo 115.4 99.4
malloc/64/all-cpus 2130.3 107.8
malloc/4096/solo 171.5 159.0
malloc/4096/all 2992.1 184.1
malloc/65536/solo 167.5 145.1
malloc/65536/all 2793.1 2382.0
yield/solo 92.0 92.5
Only the >20% changes (64 B and 4 KiB multi-CPU) establish a performance
gain; smaller differences are treated as noise. All ten boots complete.
Every boot passes heap_validate, all size classes with batch spills,
realloc data checks, dynamic slab busy/shrink/destroy, free poisoning and
exactly one expected duplicate-free rejection, with zero failures.
The comparison images use the same scheduler/kbench/pinning/strcpy test
overlays and SMP bootstrap guard; only the allocator differs. Temporary
integrity checks remain outside this PR. Standalone origin/develop build
uses CONFIG_VIRTIO_GPU off and -Wl,-u,strcpy for the existing LTO defect.
The rejected two-task steal minimum and idle polling are not included.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
malloc(64)+free 单核约 115 ns，五核同时约 2130 ns/op。此前锁采样中 cache->lock 有 70% 获取遇占用；free 的 pointer_owner 还逐次获取全局 page_lock。
为 17 个 kmalloc 尺寸类添加每 CPU magazine（每类至多 32 个对象/16 KiB），批量补充/归还；usage 计数也分散到每 CPU。有效分配块的 buddy tag 在其存活期间稳定，pointer_owner 改为原子读取 tag。保留对象 cookie、毒化和 ALLOCATED→CACHED 的 CAS，缓存对象继续计入 slab->inuse，防止 shrink 提前释放。动态 typed cache 不使用 magazine，保留 ctor/dtor 路径。
启动问题已先复现并用
qemu -gdb tcp::12345/ GDB 沿 RBP 定位：smp_init → aligned_alloc → malloc → heap_cpu_cache → get_current_cpu_id；cpu_count=5 而 cpus=NULL，原始 RIP 是 get_current_cpu_id+96，读取 cpus[0].lapic_id（地址 8）；panic 又在 send_ipi_all 缺页，掩盖报告。给 CPU ID 的 BSP 早期回退补上 cpus 未分配的条件。A/B：两边同一覆盖层和 SMP 早期回退，仅 allocator 不同；均
kbench=3，交替 5 对，先取每次启动的中位数，再取 5 次启动中位数。日志为build/kbench/heap-verified-{A,B}-{1..5}.log。10 次启动、30 轮均完成，无 panic/stale wait。每次启动的临时完整性检查覆盖全部 17 类、96 对象批次和溢出、数据校验、realloc、动态 cache busy/shrink/destroy、free 毒化、heap_validate；均 0 失败，且恰好收到一次预期的重复释放错误。检查代码留在被忽略的测试目录，没有进入 PR。
参考 Linux per-CPU 对象缓存：缓存中的对象使所属 slab 保持占用。缓存有固定上限；本 PR 不引入 CPU 热拔插或内存压力回收机制。
验证环境：5 vCPU、KVM、2 GiB、每次 QEMU 超时 60 秒。所有独立分支直接从
origin/develop@153578b5开出，目标均为develop；每个分支均独立完成 GCC 16.2 / LTO 构建，构建源只取git ls-files '*.c'，切分支后清除对应.o/.d。按已知基线问题临时关闭CONFIG_VIRTIO_GPU。除 strcpy PR 自身外，独立构建在链接命令临时加-Wl,-u,strcpy，绕过原始 develop 的已复现 LTO 缺陷；没有把其他 PR 的源码带进分支。运行基准时明确使用独立 sched 修复、strcpy 修复、kbench 和 kthread 绑定的共同测试覆盖层，以免把已知调度崩溃或实际 CPU 不符误算成待测修改的效果。覆盖层仅用于测试镜像，不构成提交依赖。
已否决方案：把偷取限制为源 rq 至少两个任务，曾使 ping-pong 约 3 µs 恶化到 7.8 µs；轮询 idle 无明显收益。这些改动均未采用。宿主负载高，仅把交替多次中位数中超过 20% 的差异视为性能结论。
🤖 Generated with ClaudeCode