Skip to content

refactor: centralizing shared-allocation accounting for Arc DFHeapSize impls#23349

Open
saadtajwar wants to merge 4 commits into
apache:mainfrom
saadtajwar:saadt/centralize-arc-dfheapsize-impls
Open

refactor: centralizing shared-allocation accounting for Arc DFHeapSize impls#23349
saadtajwar wants to merge 4 commits into
apache:mainfrom
saadtajwar:saadt/centralize-arc-dfheapsize-impls

Conversation

@saadtajwar

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

DFHeapSize depends on a critical invariant: shared heap allocations must be counted once per traversal context. If Arc implementations drift, memory accounting becomes inconsistent and hard to reason about.

Consolidating the one-time-accounting logic improves:

Correctness durability (single source of truth for Arc dedup behavior)
Maintainability (less repeated logic)
Reviewability (future changes touch one helper)

What changes are included in this PR?

Helper functions for Arc allocation identity and deduplication, namely for providing pointer extraction & DFHeapSizeCtx set membership checks, with the DFHeapSize implementations on Arc backed types using the new helpers

Are these changes tested?

Yes, the below are passing:

cargo test -p datafusion-common heap_size --lib
cargo test -p datafusion-common --doc heap_size

Are there any user-facing changes?

No

@saadtajwar

Copy link
Copy Markdown
Contributor Author

cc @kosiew - thanks in advance for any feedback! :)

@github-actions github-actions Bot added the common Related to common crate label Jul 6, 2026

@mkleen mkleen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@saadtajwar Thanks for working on this! I left a comment.

Comment thread datafusion/common/src/heap_size.rs Outdated

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice cleanup overall. I don't see any correctness issues, and consolidating the shared Arc accounting logic makes this easier to maintain. I just have one small suggestion that could help document an important invariant.

Comment thread datafusion/common/src/heap_size.rs
@saadtajwar

Copy link
Copy Markdown
Contributor Author

@mkleen & @kosiew thanks so much for taking the time to review this! Pushed a commit to address your comments - please let me know what you think! 😁 🎉

@mkleen mkleen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@saadtajwar

Copy link
Copy Markdown
Contributor Author

Thank you! Any further steps to get this in? Should I tag a repo maintainer? 👀

@mkleen

mkleen commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Thank you! Any further steps to get this in? Should I tag a repo maintainer? 👀

No, the pr usually sits for a few more days so other people can have a look and will be eventually merged.

@saadtajwar

Copy link
Copy Markdown
Contributor Author

Thank you! Any further steps to get this in? Should I tag a repo maintainer? 👀

No, the pr usually sits for a few more days so other people can have a look and will be eventually merged.

Ah okay sounds good, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Centralize shared-allocation accounting for Arc DFHeapSize implementations

3 participants