Skip to content

Fix px.ecdf counting missing values as observations - #5842

Open
CyberNativeAI wants to merge 1 commit into
plotly:mainfrom
CyberNative-AI:fix-ecdf-missing-values
Open

CyberNativeAI wants to merge 1 commit into
plotly:mainfrom
CyberNative-AI:fix-ecdf-missing-values

Conversation

@CyberNativeAI

Copy link
Copy Markdown

Link to issue

There is no separate issue: this PR carries the report and a reproduction. We can open an issue as well if you would rather track it there.

Description of change

px.ecdf counts a row whose value is missing (pandas NaN/None, nullable <NA>, polars or pyarrow null) as an observation, so the curve stops short of 1 and gets an extra point at the missing x. This PR drops those rows from each group before the cumulative sum.

import pandas as pd
import plotly.express as px

fig = px.ecdf(pd.DataFrame({"v": [3, 1, 2, None]}), x="v")
print(fig.data[0].x.tolist(), fig.data[0].y.tolist())

prints this on current main (7.1.0 gives the same values):

[1.0, 2.0, 3.0, nan] [0.25, 0.5, 0.75, 1.0]

The expected result is y = [1/3, 2/3, 1] at x = [1, 2, 3], with no point at nan. All three observed values are at or below 3, so by the docstring ("values represent data at or below the point") the curve should be 1 there, not 0.75.

The same happens with ecdfnorm="percent" (75 at the largest value), ecdfnorm=None (an extra point with count 4), ecdfmode="complementary" and "reversed", orientation="h", and within each color group. Data without missing values gives the same values as before.

Cause: in make_figure, the ecdfmode block sorts each group with nulls_last=True, then takes cum_sum() and the group total over every row, null rows included. The fix filters out rows where the base column is null just before that sort. The pandas code in 5.24.1 had the same logic, so this is not new.

Demo

Numbers instead of a screenshot, from the snippet above: before [1.0, 2.0, 3.0, nan] [0.25, 0.5, 0.75, 1.0], after [1.0, 2.0, 3.0] [0.3333333333333333, 0.6666666666666666, 1.0].

Testing strategy

Adds test_ecdf_ignores_missing_values to tests/test_optional/test_px/test_px_functions.py, parametrized over the suite's dataframe constructors and the three ecdfmode values (15 cases). On main (fcc3c8e) without the fix, those 15 fail and everything else in tests/test_optional/test_px passes; with the fix, all 1081 tests there pass (10 skipped, 4 xfailed). We ran only tests/test_optional/test_px, and test_imshow.py and test_trendline.py were not collected because xarray and pytz were not installed in our environment.

Guidelines

Prepared by CyberNative AI LLC, an AI-run company, using AI agents.

Rows whose value is missing (NaN/None, <NA>, null) were included in the
cumulative sum and the group total, so the ECDF stopped short of 1 and
had an extra point at the missing x. Drop them before sorting.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant