Plans to expose deequ's VerificationResult.rowLevelResultsAsDataFrame?

[billpratt]

Is your feature request related to a problem? Please describe.
I'd like the ability to get the row level results of VerificationResult. This is critical for quarantining rows with dq issues. Deequ exposes this through VerificationResult.rowLevelResultsAsDataFrame. Are there plans to add this to pydeequ?

Describe the solution you'd like
pydeequ has a parity method with deequ for VerificationResult.rowLevelResultsAsDataFrame

Describe alternatives you've considered
The work around is to create a custom method to extract this via JVM bridge though issues like #234 highlight fragility

Additional context
There are a few open issues around this topic dating back to 2023.
#129
#223
#224
#225
#234

[github-actions]

Thanks for filing this — you're right that rowLevelResultsAsDataFrame is an important capability for quarantine/remediation workflows, and there are several related issues requesting it. We'll track this as an enhancement to add a VerificationResult.rowLevelResultsAsDataFrame wrapper in PyDeequ that mirrors the Deequ Scala API. Contributions are welcome if anyone wants to pick this up in the meantime — the implementation would live in pydeequ/verification.py as a new classmethod on VerificationResult.

This has also been flagged for our maintainer team to review.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: cf3780e5) — may not be fully accurate. Reply if this doesn't help.

[billpratt]

I'll be happy to look at contributing this feature.

[github-actions]

Great, thanks for volunteering! Here's a quick outline to get you started:

The implementation would go in pydeequ/verification.py as a new @classmethod on VerificationResult, following the same pattern as the existing checkResultsAsDataFrame and successMetricsAsDataFrame methods:

@classmethod
def rowLevelResultsAsDataFrame(cls, spark_session: SparkSession, verificationResult, forChecks=None, pandas: bool = False):
    if forChecks:
        raise NotImplementedError("forChecks have not been implemented yet.")
    forChecks = getattr(
        spark_session._jvm.com.amazon.deequ.VerificationResult, "rowLevelResultsAsDataFrame$default$3"
    )()

    df = spark_session._jvm.com.amazon.deequ.VerificationResult.rowLevelResultsAsDataFrame(
        spark_session._jsparkSession, verificationResult.verificationRun, forChecks
    )
    sql_ctx = SQLContext(
        sparkContext=spark_session._sc,
        sparkSession=spark_session,
        jsqlContext=spark_session._jsparkSession.sqlContext(),
    )
    return DataFrame(df, sql_ctx).toPandas() if pandas else DataFrame(df, sql_ctx)

A few things to keep in mind:

1. Deequ version: rowLevelResultsAsDataFrame was added in newer Deequ versions. You may want to check the method signature in the Deequ Scala source for the exact default arguments (the $default$ index may differ).
2. Row-level results require specific check configuration: In Deequ, row-level results are only produced when checks are configured to compute them (e.g., via .withRowLevelResults). You may want to verify whether the VerificationRunBuilder in PyDeequ needs any additional wiring to enable this.
3. Tests: Add a test in tests/test_checks.py or a new test file that verifies the DataFrame is returned with the expected row-level columns.
4. Contributing guide: Follow the setup in CONTRIBUTING.md — use Poetry for deps and ensure tests pass with pytest.

Feel free to open a draft PR early if you'd like feedback from maintainers along the way!

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 67928783) — may not be fully accurate. Reply if this doesn't help.

[billpratt]

Opened a PR for this: #262

A few corrections to the AI-generated suggestion above — the Scala method signature for rowLevelResultsAsDataFrame is (sparkSession, verificationResult, data) where data is the original input DataFrame. There is no forChecks parameter and no \\$ args to resolve. There is also no .withRowLevelResults builder method — row-level results are purely post-processing on the verification result.

The PR includes tests and a README update.

[github-actions]

I've reached the limit of what I can assist with on this issue. The maintainer team has been notified and will take over.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: n/a) — may not be fully accurate. Reply if this doesn't help.