Repository navigation
Fix px.ecdf counting missing values as observations - #5842
Open
CyberNativeAI wants to merge 1 commit into
Open
CyberNativeAI wants to merge 1 commit into
CyberNativeAI wants to merge 1 commit into
Conversation
Rows whose value is missing (NaN/None, <NA>, null) were included in the cumulative sum and the group total, so the ECDF stopped short of 1 and had an extra point at the missing x. Drop them before sorting.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to issue
There is no separate issue: this PR carries the report and a reproduction. We can open an issue as well if you would rather track it there.
Description of change
px.ecdfcounts a row whose value is missing (pandasNaN/None, nullable<NA>, polars or pyarrownull) as an observation, so the curve stops short of 1 and gets an extra point at the missing x. This PR drops those rows from each group before the cumulative sum.prints this on current
main(7.1.0 gives the same values):The expected result is
y = [1/3, 2/3, 1]atx = [1, 2, 3], with no point atnan. All three observed values are at or below 3, so by the docstring ("values represent data at or below the point") the curve should be 1 there, not 0.75.The same happens with
ecdfnorm="percent"(75 at the largest value),ecdfnorm=None(an extra point with count 4),ecdfmode="complementary"and"reversed",orientation="h", and within eachcolorgroup. Data without missing values gives the same values as before.Cause: in
make_figure, theecdfmodeblock sorts each group withnulls_last=True, then takescum_sum()and the group total over every row, null rows included. The fix filters out rows where the base column is null just before that sort. The pandas code in 5.24.1 had the same logic, so this is not new.Demo
Numbers instead of a screenshot, from the snippet above: before
[1.0, 2.0, 3.0, nan] [0.25, 0.5, 0.75, 1.0], after[1.0, 2.0, 3.0] [0.3333333333333333, 0.6666666666666666, 1.0].Testing strategy
Adds
test_ecdf_ignores_missing_valuestotests/test_optional/test_px/test_px_functions.py, parametrized over the suite's dataframe constructors and the threeecdfmodevalues (15 cases). Onmain(fcc3c8e) without the fix, those 15 fail and everything else intests/test_optional/test_pxpasses; with the fix, all 1081 tests there pass (10 skipped, 4 xfailed). We ran onlytests/test_optional/test_px, andtest_imshow.pyandtest_trendline.pywere not collected becausexarrayandpytzwere not installed in our environment.Guidelines
Prepared by CyberNative AI LLC, an AI-run company, using AI agents.