Fix flaky test_compute_derivative_rows_mem_usage#23059
Open
Fix flaky test_compute_derivative_rows_mem_usage#23059
Conversation
Replace tracemalloc peak threshold check with direct assertion on _previous_statements contents. The old test was sensitive to Python version, GC timing, and ddtrace profiler overhead, causing it to exceed the 10 MB threshold on Python 3.13 in CI (measured ~105 MB). The new test directly verifies the property that matters: after calling compute_derivative_rows, the cache only contains metric columns (e.g. count, time) and not the full row data (e.g. the large query strings).
Contributor
Author
|
Added no-changelog since this is only tests. Will involve DBM in the review as well to make sure this is not breaking something I am missing. |
AAraKKe
commented
Mar 26, 2026
| .claude | ||
|
|
||
| # PR review artifacts | ||
| .agint-review/ |
Contributor
Author
There was a problem hiding this comment.
This is added to ignore any review we do with Claude in the repo, since worktrees are created it can become a pain to ensure none of this is considered.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files🚀 New features to boost your workflow:
|
eric-weaver
approved these changes
Mar 26, 2026
Contributor
eric-weaver
left a comment
There was a problem hiding this comment.
Lgtm 👍 this does change the goal of this test which was originally introduced here, but given the non-deterministic behavior of tracemalloc on CI this is a better test
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Replaces the
tracemallocpeak-memory threshold check intest_compute_derivative_rows_mem_usagewith a direct assertion on the contents of_previous_statements.Motivation
The test was failing on master (job link) with a measured peak of ~105 MB against a 10 MB threshold, on Python 3.13 with
ddtraceprofiling active.The root cause is that the old approach measured
tracemallocpeak memory, which is sensitive to:ddtraceprofiler keeping frame/object references alive longer than expectedWhy the new assertion is correct
The property we actually care about is that
_previous_statementsdoes not hold on to large non-metric data (e.g. full query strings) between check runs.compute_derivative_rowsachieves this by explicitly building the cache from only the metric columns:So after a call with
metrics = ['count', 'time'], the cache looks like:{'sig3767': {'count': 100, 'time': 2005}, 'sig4317': {'count': 200, 'time': 4010}, ...}The new test directly asserts this: each cached entry's keys are a subset of the requested metrics — meaning the 3000-char query strings,
errors,db,user, etc. are never retained. This is deterministic, independent of Python version and profiler state.Review checklist (to be filled by reviewers)
qa/skip-qalabel if the PR doesn't need to be tested during QA.backport/<branch-name>label to the PR and it will automatically open a backport PR once this one is merged