Conversation
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> N/A ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Some PRs are being omitted from stale check because they were in a cache, and the workflow appears to not have permission to delete cache so they are forever stuck as unprocessed. For example in this run: https://github.com/apache/datafusion/actions/runs/24756695077/job/72431314533 Seeing this in logs: ``` [apache#20473] issue skipped due being processed during the previous run [apache#20460] pull request skipped due being processed during the previous run [apache#20448] issue skipped due being processed during the previous run [apache#20443] issue skipped due being processed during the previous run [apache#20435] issue skipped due being processed during the previous run [apache#20418] issue skipped due being processed during the previous run [apache#20417] pull request skipped due being processed during the previous run [apache#20416] pull request skipped due being processed during the previous run [apache#20403] pull request skipped due being processed during the previous run ``` And at the end we see this warning: ``` Warning: Error delete _state: [403] Resource not accessible by integration - https://docs.github.com/rest/actions/cache#delete-github-actions-caches-for-a-repository-using-a-cache-key ``` stale workflow uses a cache in case it hits the `operations-per-run` limit meant to prevent API rate limiting (we have default of 30), so it seems we previously hit this limit and some issues/PRs were cached, and have never been uncached since so are never processed again. See: https://github.com/actions/stale#operations-per-run ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Give permission to stale workflow to run github actions (like delete cache). See recommended permissions: https://github.com/actions/stale#recommended-permissions ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
## Summary - Adds `cosine_distance(array1, array2)` / `list_cosine_distance` — computes cosine distance (1 - cosine similarity) between two numeric arrays - Introduces shared `vector_math.rs` primitives (`dot_product_f64`, `magnitude_f64`, `convert_to_f64_array`) for reuse by follow-on vector functions - Returns NULL for zero-magnitude vectors; errors on mismatched lengths - Supports List, LargeList, and FixedSizeList with any numeric element type Part of apache#21536 — first in a series of split PRs (replacing apache#21371). ## Test plan - [x] Unit tests: identical, orthogonal, opposite, 45-degree, zero-magnitude, mismatched-length, NULL, multi-row - [x] sqllogictest: `cosine_distance.slt` covering all edge cases including empty arrays, LargeList, integer coercion, alias, return type - [x] Full slt suite (426/426 pass) - [x] `cargo clippy`, `cargo fmt`, `taplo`, `prettier`, `cargo machete` — all clean 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ueryAlias (apache#21491) ## Which issue does this PR close? - Closes apache#21490. ## Rationale for this change When `plan_to_sql` encounters a `Projection → Sort → Projection → SubqueryAlias` plan where the outer Projection excludes a Sort column defined as an alias in the inner Projection (e.g. `Z AS c`), the function `rewrite_plan_for_sort_on_non_projected_fields` flattens the two Projections into one but only keeps the outer Projection's columns. This drops the alias definition, leaving `ORDER BY c` referencing a column that no longer exists in the generated SQL. ## What changes are included in this PR? In `rewrite_plan_for_sort_on_non_projected_fields` (`datafusion/sql/src/unparser/rewrite.rs`), when the inner Projection is trimmed to only the outer Projection's expressions, sort expressions that reference dropped aliases are now inlined to the underlying physical expression. For example, `ORDER BY c` becomes `ORDER BY t."Z"` when `c` was defined as `Z AS c` in the dropped inner Projection. ## Are these changes tested? Yes. A new regression test `test_sort_on_aliased_column_dropped_by_outer_projection` in `datafusion/sql/tests/cases/plan_to_sql.rs` constructs the exact plan shape that triggers the bug and asserts the correct SQL output: ```sql SELECT t."X" AS a, t."Y" AS b FROM phys_table AS t ORDER BY t."Z" DESC NULLS FIRST LIMIT 1 ``` All existing `plan_to_sql` tests (118 tests) continue to pass. ## Are there any user-facing changes? No API changes. Previously invalid SQL is now generated correctly. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - From this discussion: apache#19711 (comment) ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> In above PR we didn't enable a test because upstream arrow-rs had a panic bug. This is now fixed: - apache/arrow-rs#9144 So now we can enable this test again to assert expected error. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#19782 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> We've landed various PRs required to support aggregating on list views (both upstream in arrow-rs and here in datafusion) so now this simple aggregation test succeeds; adding it to the slt suite. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Add aggregation test for listviews ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> No <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
…he#21773) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#19798 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Upstream issue fixed, re-enable test - apache/arrow-rs#9227 ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
…l_compression` pass in every config (apache#21504) ## Which issue does this PR close? - Closes apache#21503. ## Rationale for this change The spill manager assumes that all available compressions are actually available, which currently relies on feature unification with `datafusion-datasource-arrow`. ## What changes are included in this PR? Move `arrow-ipc/zstd` feature from `datafusion-datasource-arrow` to the workspace dependency, like lz4. ## Are these changes tested? Existing tests cover the functionality, tested individual crates locally. ## Are there any user-facing changes? Shifts some features around which might change behavior for users for DataFusion, not sure how the project reasons about this level of changes.
…he#21800) ## Which issue does this PR close? - Closes apache#21798 ## Rationale for this change The coercion should take place in the same way as is foreseen already for the string datatypes (Utf8, LargeUtf8, Null -> Utf8View) ## What changes are included in this PR? The coerced_from function in datafusion/expr/src/type_coercion/functions.rs was missing a match arm for (BinaryView, Binary | LargeBinary | Null). ## Are these changes tested? Additional unit test written : `test_binary_conversion` ## Are there any user-facing changes? No Co-authored-by: Bert Vermeiren <bert.vermeiren@datadobi.com>
## Which issue does this PR close? - Follow-up to apache#21544 ## Rationale for this change @Jefffrey [noted](apache#21544 (comment)) that some bench helpers created a special distribution of values, so we can revisit their usage. ## What changes are included in this PR? Some of the helpers are created for the purpose. For this reason, I initially avoided migrating most candidate helpers from other files (`array_has`, `array_sort` etc). - `array_remove`'s helpers (`create_f64_list_array` and `create_decimal64_list_array`) should be brought back because the probability of finding a needle depends on the distribution, which is set explicitly instead of the range of a template parameter - bringing them back in this PR - array_min_max is safe since it locates a min/max elements, which isn't affected by the value distribution - array_repeat doesn't check contents, so it is safe - array_to_string is not content-aware ## Are these changes tested? Run changed benchmarks ## Are there any user-facing changes? No
## Which issue does this PR close? - Closes apache#10669 Related arrow-rs PRs apache/arrow-rs#8960 and apache/arrow-rs#9004 ## Rationale for this change The CSV writer was missing support for `quote_style`, `ignore_leading_whitespace`, and `ignore_trailing_whitespace` options that are available on the underlying arrow `WriterBuilder`. This meant users couldn't control quoting behaviour or whitespace trimming when writing CSV files. ## What changes are included in this PR? Adds three new CSV writer options wired through the full stack: - `quote_style` — controls when fields are quoted (`Always`, `Necessary`, `NonNumeric`, `Never`). Modelled as a protobuf enum (`CsvQuoteStyle`). - `ignore_leading_whitespace` — trims leading whitespace from string values on write. - `ignore_trailing_whitespace` — trims trailing whitespace from string values on write. ## Are these changes tested? Yes — sqllogictest coverage added in `csv_files.slt` ## Are there any user-facing changes? Three new `format.*` options available in COPY TO and CREATE EXTERNAL TABLE for CSV: - `format.quote_style` (string: `Always`, `Necessary`, `NonNumeric`, `Never`) - `format.ignore_leading_whitespace` (boolean) - `format.ignore_trailing_whitespace` (boolean)
## Which issue does this PR close? - Closes apache#21804. ## Rationale for this change `case_conversion_ascii_array` operates directly on the underlying values buffer, but it neglects to ensure it only looks at bytes within the visible slice. For sliced arrays, this can lead to doing substantial unnecessary work. ## What changes are included in this PR? * Optimize `case_conversion_ascii_array` for sliced arrays * Add a unit test * Add a benchmark. We can make the "sliced array" case arbitrarily extreme, so the raw benchmark number here is less important; it is more important that this benchmark confirms that the work we do scales with the visible size of a sliced array, which it does. ## Are these changes tested? Yes. ## Are there any user-facing changes? No.
…apache#21789) ## Which issue does this PR close? - Part of apache#21684 ## Rationale for this change Introduce three new string array builders with bulk null tracking: - `StringArrayBuilder` (Utf8) - `LargeStringArrayBuilder` (LargeUtf8) - `StringViewArrayBuilder` (Utf8View) Each builder has the following API: - append_value(&str) -- add a non-NULL value (row) - append_placeholder() -- add a NULL row placeholder - finish(Option<NullBuffer>) -- finish the build, specify NULLs These are the counterpart of Arrow's `GenericStringBuilder` / `StringViewBuilder` but it skips per-row NULL buffer maintenance, which lets callers compute the NULL buffer in bulk when possible. This PR also switches `case_conversion` to use the new APIs, which is used to implement `lower`, `upper`, and the Spark equivalents. This improves `lower` / `upper` performance by 3-15% on microbenchmarks. More UDFs (~10) will be converted to use this API in future PRs. ## What changes are included in this PR? * Add new builders * Add unit tests * Adopt builders in `case_conversion` ## Are these changes tested? Yes. ## Are there any user-facing changes? No.
…st/datafusion-wasm-app (apache#21164) Bumps [picomatch](https://github.com/micromatch/picomatch) from 2.3.1 to 2.3.2. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/micromatch/picomatch/releases">picomatch's releases</a>.</em></p> <blockquote> <h2>2.3.2</h2> <p>This is a security release fixing several security relevant issues.</p> <h2>What's Changed</h2> <ul> <li>fix: exception when glob pattern contains constructor by <a href="https://github.com/Jason3S"><code>@Jason3S</code></a> in <a href="https://redirect.github.com/micromatch/picomatch/pull/144">micromatch/picomatch#144</a></li> <li>Fix for <a href="https://github.com/micromatch/picomatch/security/advisories/GHSA-c2c7-rcm5-vvqj">CVE-2026-33671</a></li> <li>Fix for <a href="https://github.com/micromatch/picomatch/security/advisories/GHSA-3v7f-55p6-f55p">CVE-2026-33672</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/micromatch/picomatch/compare/2.3.1...2.3.2">https://github.com/micromatch/picomatch/compare/2.3.1...2.3.2</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md">picomatch's changelog</a>.</em></p> <blockquote> <h1>Release history</h1> <p><strong>All notable changes to this project will be documented in this file.</strong></p> <p>The format is based on <a href="http://keepachangelog.com/en/1.0.0/">Keep a Changelog</a> and this project adheres to <a href="http://semver.org/spec/v2.0.0.html">Semantic Versioning</a>.</p> <!-- raw HTML omitted --> <ul> <li>Changelogs are for humans, not machines.</li> <li>There should be an entry for every single version.</li> <li>The same types of changes should be grouped.</li> <li>Versions and sections should be linkable.</li> <li>The latest version comes first.</li> <li>The release date of each versions is displayed.</li> <li>Mention whether you follow Semantic Versioning.</li> </ul> <!-- raw HTML omitted --> <!-- raw HTML omitted --> <p>Changelog entries are classified using the following labels <em>(from <a href="http://keepachangelog.com/">keep-a-changelog</a></em>):</p> <ul> <li><code>Added</code> for new features.</li> <li><code>Changed</code> for changes in existing functionality.</li> <li><code>Deprecated</code> for soon-to-be removed features.</li> <li><code>Removed</code> for now removed features.</li> <li><code>Fixed</code> for any bug fixes.</li> <li><code>Security</code> in case of vulnerabilities.</li> </ul> <!-- raw HTML omitted --> <h2>4.0.0 (2024-02-07)</h2> <h3>Fixes</h3> <ul> <li>Fix bad text values in parse <a href="https://redirect.github.com/micromatch/picomatch/issues/126">#126</a>, thanks to <a href="https://github.com/connor4312"><code>@connor4312</code></a></li> </ul> <h3>Changed</h3> <ul> <li>Remove process global to work outside of node <a href="https://redirect.github.com/micromatch/picomatch/issues/129">#129</a>, thanks to <a href="https://github.com/styfle"><code>@styfle</code></a></li> <li>Add sideEffects to package.json <a href="https://redirect.github.com/micromatch/picomatch/issues/128">#128</a>, thanks to <a href="https://github.com/frandiox"><code>@frandiox</code></a></li> <li>Removed <code>os</code>, make compatible browser environment. See <a href="https://redirect.github.com/micromatch/picomatch/issues/124">#124</a>, thanks to <a href="https://github.com/gwsbhqt"><code>@gwsbhqt</code></a></li> </ul> <h2>3.0.1</h2> <h3>Fixes</h3> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/micromatch/picomatch/commit/81cba8d4b767cab3cb29d26eb4f691eed75b73b2"><code>81cba8d</code></a> Publish 2.3.2</li> <li><a href="https://github.com/micromatch/picomatch/commit/fc1f6b69006e9435caf8fb40d8aff378bc0b7bce"><code>fc1f6b6</code></a> Merge commit from fork</li> <li><a href="https://github.com/micromatch/picomatch/commit/eec17aee5428a7249e9ca5adbb8a0d28fa29619b"><code>eec17ae</code></a> Merge commit from fork</li> <li><a href="https://github.com/micromatch/picomatch/commit/78f8ca4362d9e66cadea97b93e292f10096452ed"><code>78f8ca4</code></a> Merge pull request <a href="https://redirect.github.com/micromatch/picomatch/issues/156">#156</a> from micromatch/backport-144</li> <li><a href="https://github.com/micromatch/picomatch/commit/3f4f10eaa65bf3a52e8f2999674cd27e11fa3c9b"><code>3f4f10e</code></a> Merge pull request <a href="https://redirect.github.com/micromatch/picomatch/issues/144">#144</a> from Jason3S/jdent-object-properties</li> <li>See full diff in <a href="https://github.com/micromatch/picomatch/compare/2.3.1...2.3.2">compare view</a></li> </ul> </details> <br /> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Part of apache#20585. ## Rationale for this change `substr_index` is part of the Utf8View epic. This function was still missing the Utf8View optimization path, and it also did extra work when delimiter and count were constant scalars. This PR adds the same kind of optimization used in similar string work and includes benchmark coverage. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Return the input string type directly, including `Utf8View`. - Add a zero-copy `Utf8View` path for `substr_index`. - Add a scalar fast path when delimiter and count are constant. - Keep the hot path specialized after checking a cleaner shared-helper version and benchmarking it. - Add SQL coverage for `Utf8View` type and value results. - Add unit tests for the scalar fast path, sliced `Utf8View` arrays, and the unchanged original-view case. - Extend the benchmark to cover `Utf8`, `Utf8View`, array arguments, scalar delimiter/count, and positive and negative counts. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? No <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
## Which issue does this PR close? When `PushdownSort` removes a `SortExec` because a source returns `Exact` (guaranteeing ordering), any `fetch` (LIMIT) on the `SortExec` is silently dropped if the underlying plan does not support `with_fetch()`. For example, `ProjectionExec` supports `try_pushdown_sort` (delegating to its child) but does not implement `with_fetch()`. A plan like `SortExec(fetch=10) → ProjectionExec → source` that gets sort-eliminated loses the limit. ## What changes are included in this PR? In the `Exact` branch of `PushdownSort`, when the eliminated `SortExec` carried a `fetch`: 1. Try `with_fetch()` on the pushed-down source first 2. If `with_fetch()` returns `None`, fall back to wrapping with `GlobalLimitExec` ## Are these changes tested? Yes. Three new unit tests: - `test_sort_pushdown_exact_no_fetch_no_limit` — Exact elimination without fetch: no limit wrapper added - `test_sort_pushdown_exact_preserves_fetch_with_global_limit` — Exact elimination with fetch, source does NOT support `with_fetch()`: `GlobalLimitExec` wrapper added - `test_sort_pushdown_exact_preserves_fetch_with_source_support` — Exact elimination with fetch, source supports `with_fetch()`: limit pushed into source directly ## Are there any user-facing changes? No.
## Which issue does this PR close? - Closes apache#21844. ## Rationale for this change `uuid()` never emits NULLs, so this is a straightforward win. Benchmarks: ``` uuid (1024 rows): 21.618 µs → 20.133 µs, −6.90% (`p < 0.05`) ``` ## What changes are included in this PR? * `GenericStringBuilder` -> `GenericStringArrayBuilder` in `uuid()` implementation ## Are these changes tested? Yes. ## Are there any user-facing changes? No.
…ns (apache#21701) ## Which issue does this PR close? - Closes apache#21700. ## Rationale for this change When an Extension node has no expressions, `map_expressions` was still cloning all inputs and calling `with_exprs_and_inputs` to reconstruct the node — wasted work since there are no expressions to transform. This is common for Extension nodes like view matching candidates that carry multiple children but no expressions. ## What changes are included in this PR? 1. **Code change** (`datafusion/expr/src/logical_plan/tree_node.rs`): Add early return when `node.expressions()` is empty, skipping the clone + rebuild path. 2. **Micro-benchmark** (`datafusion/expr/benches/map_expressions.rs`): Criterion benchmark comparing `map_expressions` on Extension nodes with and without expressions, varying the number of children (1, 3, 5, 10). ## Are these changes tested? Yes — existing tests pass, and the new benchmark validates the optimization. Benchmark results: | Children | no_expr (optimized) | with_expr (rebuild) | Speedup | |----------|--------------------|--------------------|---------| | 1 | 24 ns | 167 ns | 7x | | 3 | 23 ns | 192 ns | 8x | | 5 | 23 ns | 181 ns | 8x | | 10 | 24 ns | 216 ns | 9x | The `no_expr` path is constant time regardless of children count. In a real optimizer pipeline (~15 rules × 5-child Extension), this saves ~2.4 us per optimization pass. ## Next step A more general optimization would be to change `UserDefinedLogicalNode::expressions()` to return references (`&[Expr]`) instead of cloned `Vec<Expr>`, and only clone + rebuild when the transform actually modifies an expression. This would avoid the clone + `with_exprs_and_inputs` rebuild even for non-empty expression lists when the transform is a no-op. Added a TODO comment in the code for this direction. This would be a larger API change, so the empty-expressions shortcut is a pragmatic first step. ## Are there any user-facing changes? No — purely internal optimization. No API changes.
## Which issue does this PR close? - Closes #. ## Rationale for this change Avoid deep-copying values behind Arc when the Arc is uniquely owned by using Arc::unwrap_or_clone. ## What changes are included in this PR? Replaces several owned Arc<T> pointee clones with Arc::unwrap_or_clone, including logical plans, schemas, fields, expressions, and statistics. ## Are these changes tested? ## Are there any user-facing changes? No.
…1869) ## Which issue does this PR close? N/A ## Rationale for this change this is done so we can query cargo metadata alone to find only user facing crates and to use cargo metadata to find user facing crates when searching for breaking changes in pr as suggested here: - apache#21499 (comment) ## What changes are included in this PR? set `publish = false` for: 1. `datafusion-benchmarks` 2. `datafusion-wasmtest` 3. `test-utils` ## Are these changes tested? I verified that these are not already published to crates.io and not in `dev/release/README.md` _Publish on Crates.io_ list ## Are there any user-facing changes? No
## Which issue does this PR close? - Part of apache#19241. ## Rationale for this change This PR is a preparatory refactor for [apache#19390](apache#19390). `InListExpr` currently mixes expression evaluation, constant-list filter construction, primitive-vs-fallback dispatch, and generic fallback execution in one file. This PR extracts the existing static-filter code into smaller internal modules while preserving the current behavior. ## Review guide This PR is intended to be a move/extract-only refactor. The main review question is whether the existing `InListExpr` static-filter behavior has been preserved while making the internal structure easier to follow. In particular: - `in_list.rs` still owns `InListExpr` construction and evaluation - `primitive_filter.rs` contains the existing primitive numeric static-filter implementations moved out of `in_list.rs` - `array_static_filter.rs` contains the existing `ArrayStaticFilter` fallback moved out of `in_list.rs` - `strategy.rs` contains the existing dispatch between primitive filters and `ArrayStaticFilter` - `static_filter.rs` defines the shared internal trait used by those extracted pieces No user-facing behavior or API changes are intended in this PR. ## Are these changes tested? Yes. I validated this PR with: - `cargo fmt --all` - `cargo clippy -p datafusion-physical-expr --all-targets --all-features -- -D warnings` - `cargo test -p datafusion-physical-expr --all-features` ## Are there any user-facing changes? No. This is an internal refactor only. Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close? - None. ## Rationale for this change Fix a handful of small typos in the contributor and user documentation. ## What changes are included in this PR? - Correct spelling mistakes in several docs pages - Replace a Cyrillic `С` with an ASCII `C` in aggregate function descriptions ## Are these changes tested? - `rustup run stable cargo fmt --all -- --check` - Attempted `PATH="$HOME/Library/Python/3.9/bin:$PATH" rustup run stable cargo clippy --all-targets --all-features -- -D warnings`, which hit an existing clippy failure in `datafusion/expr/src/logical_plan/plan.rs:3775` unrelated to this docs-only change ## Are there any user-facing changes? - Documentation text only.
Last few runs for stale workflow is running into limit of `operations-per-run` which by default is 30 - https://github.com/actions/stale#operations-per-run See: - https://github.com/apache/datafusion/actions/runs/24868809436 - https://github.com/apache/datafusion/actions/runs/24920208039 - https://github.com/apache/datafusion/actions/runs/24946087481 - https://github.com/apache/datafusion/actions/runs/24973415001 I don't think we're in danger of being rate limited by GitHub if we bump to 150 (from 30)
## Which issue does this PR close? - Closes apache#21862. ## Rationale for this change Similar to other recent changes, we can use the bulk-NULL string builders to avoid per-row NULL handling overhead. Benchmarks: - scalar/scalar_utf8: 227.4ns → 228.6ns (+0.53%) - scalar/scalar_utf8view: 227.9ns → 229.2ns (+0.57%) - size=1024 str_len=128/array_utf8: 276.5µs → 277.7µs (+0.43%) - size=1024 str_len=128/array_utf8view: 223.3µs → 222.4µs (−0.40%) - size=1024 str_len=16/array_utf8: 37.5µs → 37.4µs (−0.27%) - size=1024 str_len=16/array_utf8view: 41.9µs → 39.3µs (−6.21%) - size=4096 str_len=128/array_utf8: 1115.4µs → 1109.2µs (−0.56%) - size=4096 str_len=128/array_utf8view: 900.2µs → 895.2µs (−0.56%) - size=4096 str_len=16/array_utf8: 148.3µs → 148.4µs (+0.07%) - size=4096 str_len=16/array_utf8view: 164.2µs → 153.8µs (−6.33%) - size=8192 str_len=128/array_utf8: 2.2ms → 2.2ms (0.00%) - size=8192 str_len=128/array_utf8view: 1811.2µs → 1819.2µs (+0.44%) - size=8192 str_len=16/array_utf8: 299.0µs → 299.3µs (+0.10%) - size=8192 str_len=16/array_utf8view: 330.6µs → 310.2µs (−6.17%) - unicode size=1024/array_utf8: 772.2µs → 769.2µs (−0.39%) - unicode size=1024/array_utf8view: 777.8µs → 779.2µs (+0.18%) - unicode size=4096/array_utf8: 3.1ms → 3.1ms (0.00%) - unicode size=4096/array_utf8view: 3.1ms → 3.1ms (0.00%) - unicode size=8192/array_utf8: 6.2ms → 6.1ms (−1.61%) - unicode size=8192/array_utf8view: 6.2ms → 6.2ms (0.00%) ## What changes are included in this PR? * Switch to bulk-NULL string builders * Optimize `StringViewArrayBuilder::append_value()` by inlining the view construction for > 12 byte strings. Empirically this improved performance, probably because `make_view` can't be inlined. This should also help other PRs in this series. ## Are these changes tested? Yes, covered by existing tests. ## Are there any user-facing changes? No.
## Which issue does this PR close? - Closes apache#21574. ## Rationale for this change Previously, the CLI hint was being highlighted directly, which was causing an assertion to fail in `rustyline`. ## What changes are included in this PR? - Use `highlight_hint` instead of manually highlighting the text. ## Are these changes tested? Manually. ## Are there any user-facing changes? No.
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#21279 - Closes apache#21281 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Keep dependencies up to date. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> We used to use `Digest` from `blake2`, which was a common dependency used by `md5` and `sha2`; however `blake2` doesn't have a `0.11.0` release so we were blocked because of incompatible dependencies when trying to upgrade `sha2` or `md5`. Fix code to use their own `Digest` (e.g. `md5::Digest`, `sha2::Digest`) instead of relying on `blake2::Digest` which should prevent such issues from occurring again and allows us to bump their versions independently. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Existing tests. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> No. <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
## Which issue does this PR close? - None. This is a small documentation sync update. ## Rationale for this change The CLI help output shown in the user guide had drifted from the current `datafusion-cli --help` output, which makes the documentation misleading for users trying to discover supported flags. ## What changes are included in this PR? - Update the `datafusion-cli --help` snippet in `docs/source/user-guide/cli/usage.md` - Remove outdated options and defaults - Add newer flags and current help text formatting ## Are these changes tested? - Not with automated tests. This is a documentation-only change that updates the checked-in help output. ## Are there any user-facing changes? - Yes. The user guide now matches the current CLI usage output more closely. Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close? - Follow on to apache#21240 ## Rationale for this change While reviewing apache#21240 I found that the existing doc comments describing the relationship between ExecutionProps and TaskContext were vague ## What changes are included in this PR? - Update the doc comments on ExecutionProps and TaskContext - Clarify why the two structures remain separate ## Are these changes tested? Not with automated tests. This is a comment-only change. ## Are there any user-facing changes? Docs only
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#21747 ## Rationale for this change datafusion did not have a CI check for broken links in markdown content, docs workflows build and deploy docs, and dev checks formatting and spelling, but none of them validate link targets. This pr adds a dedicated link check for internal markdown links so broken references fail early in PRs. I kept the scope internal-only to avoid flaky CI failures from external websites and rate limits. Rust doc comments remain covered by the existing rustdoc CI job. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Added a new Dev workflow job, **Check Markdown Links**, in `dev.yml`. - Added `LYCHEE_VERSION` pin in `tool_versions.sh`. - Added `markdown_link_check.sh` to run lychee on the selected markdown paths. - Added `lychee.toml` with internal-link policy and exclusions. - Added check markdown links to required status checks in `.asf.yaml`. - Updated contributor testing docs with the new local command and scope note. - Fixed internal markdown links that failed under the new check in: - `roadmap.md` - `49.0.0.md` - `overview.md` - `dataframe.md` - `format_options.md` <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes, - `python3 ci/scripts/check_asf_yaml_status_checks.py` passed. - `bash -n ci/scripts/markdown_link_check.sh` passed. - `bash ci/scripts/markdown_link_check.sh` passed with 0 errors. - `cargo fmt --all --check` passed. OK: All 5 required_status_checks match existing GitHub Actions jobs. 🔍 12824 Total (in 0s) ✅ 490 OK 🚫 0 Errors 👻 12334 Excluded <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? No, There is one contributor-facing CI change: PRs now fail when internal markdown links break in the checked markdown files. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com>
## Which issue does this PR close? - Closes apache#21846. ## Rationale for this change Optimize `chr` by avoiding per-row NULL bitmap maintenance, and also split the hot loop to avoid taking a branch when no NULL bitmap is given. Benchmarks: ``` chr/array: 3.8768 µs → 3.1548 µs, −18.57% (`p < 0.05`) ``` ## What changes are included in this PR? * Optimize `chr` to reduce NULL-handling overhead ## Are these changes tested? Yes. ## Are there any user-facing changes? No. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…ache#21725) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#16894 ## What changes are included in this PR? - User-visible struct-UNNEST columns no longer expose `__unnest_placeholder`. - Internal Unnest planning still uses placeholders. - This adds a final Projection layer in explain plans wherever struct UNNEST output is published. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? Yes. The aliases of unnested columns will be changed. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Bumps [idna](https://github.com/kjd/idna) from 3.11 to 3.15. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/kjd/idna/blob/master/HISTORY.md">idna's changelog</a>.</em></p> <blockquote> <h2>3.15 (2026-05-12)</h2> <ul> <li>Enforce DNS-length cap on individual labels early in <code>check_label</code>, short-circuiting contextual-rule processing for oversized input while staying compatible with UTS 46 usage.</li> <li>Tidy core helpers: hoist bidi category sets to module-level frozensets (avoiding per-codepoint list construction), simplify length checks, and reuse the shared <code>_unicode_dots_re</code> from <code>idna.core</code> in the codec module.</li> <li>Use <code>raise ... from err</code> for proper exception chaining and switch internal string formatting to f-strings.</li> <li>Allow <code>flit_core</code> 4.x in the build backend.</li> <li>Expand the ruff lint set (flake8-bugbear, flake8-simplify, pyupgrade, perflint) and apply the surfaced fixes; pin lint CI to Python 3.14.</li> <li>Add Dependabot configuration for GitHub Actions.</li> <li>Convert README and HISTORY from reStructuredText to Markdown.</li> <li>Reference CVE-2026-45409 for the 3.14 advisory in place of the initial GHSA identifier.</li> </ul> <p>Thanks to Felix Yan, Stan Ulbrych, and metsw24-max for contributions to this release.</p> <h2>3.14 (2026-05-10)</h2> <ul> <li>Removed opportunity to process long inputs into quadratic time by rejecting oversize inputs up-front. Closes a bypass of the CVE-2024-3651 mitigation. [CVE-2026-45409]</li> </ul> <p>Thanks to Stan Ulbrych for reporting the issue.</p> <h2>3.13 (2026-04-22)</h2> <ul> <li>Correct classification error for codepoint U+A7F1</li> </ul> <h2>3.12 (2026-04-21)</h2> <ul> <li>Update to Unicode 17.0.0.</li> <li>Issue a deprecation warning for the transitional argument.</li> <li>Added lazy-loading to provide some performance improvements.</li> <li>Removed vestiges of code related to Python 2 support, including segmentation of data structures specific to Jython.</li> </ul> <p>Thanks to Rodrigo Nogueira for contributions to this release.</p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/kjd/idna/commit/af30a092e158181d0b35ac66dfa813788126bdd8"><code>af30a09</code></a> Release 3.15</li> <li><a href="https://github.com/kjd/idna/commit/30314d4628744ca14cf2b5820564e5127a9f86f2"><code>30314d4</code></a> Pre-release 3.15rc0</li> <li><a href="https://github.com/kjd/idna/commit/05d4b219aa9eddc47371fcbd2000f0301016f3e9"><code>05d4b21</code></a> Merge pull request <a href="https://redirect.github.com/kjd/idna/issues/237">#237</a> from kjd/convert-docs-to-markdown</li> <li><a href="https://github.com/kjd/idna/commit/2987fdba1962bbb2358399e0084ba062b98a0bee"><code>2987fdb</code></a> Convert README and HISTORY from reStructuredText to Markdown</li> <li><a href="https://github.com/kjd/idna/commit/59fa8002d514bf4a5ce7b58f67b9ec587d53fa9c"><code>59fa800</code></a> Merge pull request <a href="https://redirect.github.com/kjd/idna/issues/236">#236</a> from kjd/dependabot/github_actions/actions-f3e34333ea</li> <li><a href="https://github.com/kjd/idna/commit/def69834ced5d4b3c50439d8b99c4c856ec19ca2"><code>def6983</code></a> Merge branch 'master' into dependabot/github_actions/actions-f3e34333ea</li> <li><a href="https://github.com/kjd/idna/commit/bbd8004a797185d8c56bb555cd5c88fde05e0631"><code>bbd8004</code></a> Merge pull request <a href="https://redirect.github.com/kjd/idna/issues/234">#234</a> from StanFromIreland/patch-1</li> <li><a href="https://github.com/kjd/idna/commit/edd07c05024344a6ccb517414ccb36683aee99fc"><code>edd07c0</code></a> Bump github/codeql-action from 3.35.2 to 4.35.2 in the actions group</li> <li><a href="https://github.com/kjd/idna/commit/5557db030c11bdec50d62aa5f631d705d33ba123"><code>5557db0</code></a> Merge branch 'master' into patch-1</li> <li><a href="https://github.com/kjd/idna/commit/f11746cf4981d25123ef7830d3ee60f07de8ae3d"><code>f11746c</code></a> Merge pull request <a href="https://redirect.github.com/kjd/idna/issues/235">#235</a> from StanFromIreland/patch-2</li> <li>Additional commits viewable in <a href="https://github.com/kjd/idna/compare/v3.11...v3.15">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/datafusion/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
## Which issue does this PR close? - Partially addresses apache#14044 ## Rationale for this change Donate the `xxhash64` hash function from Comet so that other projects can benefit from it. The function was initially implemented in Comet by @advancedxy. This is a continuation of apache#19627, which has gone stale. To keep the change focused and easier to review, this PR adds only `xxhash64`. Murmur3 will follow in a separate PR. The first commit is the same as the version of apache#19627 that was previously approved. The second commit implements optmizations and bug fixes from the latest Comet version. The third commit is cleanup. ## What changes are included in this PR? - Add `xxhash64(expr1, expr2, ...)` to `datafusion-spark`. - Add Rust unit tests for primitives, boundary values, emoji/CJK strings, float `-0.0` normalization, dictionaries (with and without nulls), `FixedSizeBinary`, `Struct`, and `List`. - Add sqllogictest coverage in `datafusion/sqllogictest/test_files/spark/hash/xxhash64.slt` with values verified against Spark. ## Are these changes tested? Yes, both Rust unit tests and sqllogictest are included. ## Are there any user-facing changes? A new `xxhash64` scalar function is available in `datafusion-spark`.
## Which issue does this PR close? Follow up of apache#21679 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
…he#22166) ## Which issue does this PR close? * Part of apache#22163 ## Rationale for this change The `%c` conversion path in Spark `format_string` duplicated nearly identical logic across all signed and unsigned integer scalar variants. This duplication increased maintenance overhead and the risk of future `%c` fixes being applied inconsistently. This change centralizes numeric scalar-to-character conversion in a single helper while preserving existing behavior and error propagation semantics. ## What changes are included in this PR? * Added a local `integer_scalar_to_char` helper to centralize integer scalar `%c` conversion handling. * Routed all non-null integer `%c` conversion paths through the new helper. * Removed duplicated `%c` conversion branches from individual integer scalar match arms. * Preserved existing signed/unsigned conversion semantics by continuing to use the existing `signed_to_char` and `unsigned_to_char` helpers. * Left all non-`%c` formatting behavior unchanged. ## Are these changes tested? No new tests were added in this PR. This refactor is intended to preserve existing behavior, and existing `%c` formatting tests continue to validate valid and invalid code point handling. Suggested validation command: ```bash cargo test -p datafusion-spark format_char ``` ## Are there any user-facing changes? No. This is a maintainability refactor only and is not intended to change observable behavior or error messages. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed and tested.
… in /docs (apache#22378) Updates the requirements on [myst-parser](https://github.com/executablebooks/MyST-Parser) to permit the latest version. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/executablebooks/MyST-Parser/releases">myst-parser's releases</a>.</em></p> <blockquote> <h2>v5.1.0</h2> <h2>✨ New Features</h2> <ul> <li>✨ Add <code>"alert"</code> syntax extension for <a href="https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#alerts">GFM alerts</a> (e.g. <code>> [!NOTE]</code>), see <a href="https://myst-parser.readthedocs.io/en/latest/syntax/optional.html#alerts-github-style-callouts">docs</a> by <a href="https://github.com/chrisjsewell"><code>@chrisjsewell</code></a> in <a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1128">#1128</a></li> <li>✨ Add <code>"gfm_autolink"</code> syntax extension for <a href="https://github.github.com/gfm/#autolinks-extension-">GFM autolinks</a>, see <a href="https://myst-parser.readthedocs.io/en/latest/syntax/optional.html#gfm-autolinks">docs</a> by <a href="https://github.com/chrisjsewell"><code>@chrisjsewell</code></a> in <a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1128">#1128</a></li> <li>✨ Add <code>myst_strikethrough_single_tilde</code> <a href="https://myst-parser.readthedocs.io/en/latest/configuration.html">config option</a> to allow single tilde (<code>~</code>) for strikethrough by <a href="https://github.com/chrisjsewell"><code>@chrisjsewell</code></a> in <a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1128">#1128</a></li> <li>✨ Add <code>myst_colon_fence_exact_match</code> <a href="https://myst-parser.readthedocs.io/en/latest/configuration.html">config option</a> to require the closing colon fence to have exactly the same number of colons as the opening, see <a href="https://myst-parser.readthedocs.io/en/latest/syntax/optional.html#code-fences-using-colons">docs</a> by <a href="https://github.com/chrisjsewell"><code>@chrisjsewell</code></a> in <a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1128">#1128</a></li> </ul> <h2>👌 Improvements</h2> <ul> <li>👌 Update <code>myst_gfm_only</code> mode to use the unified <code>gfm_plugin</code>, which now includes GFM autolinks, alerts, and improved strikethrough/tasklist handling by <a href="https://github.com/chrisjsewell"><code>@chrisjsewell</code></a> in <a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1128">#1128</a></li> <li>👌 Improve MathJax 4 compatibility for Sphinx 9 by <a href="https://github.com/chrisjsewell"><code>@chrisjsewell</code></a> in <a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1110">#1110</a></li> <li>👌 Stop directive-option parsing at colon fences, fixing nested colon fence directives by <a href="https://github.com/chrisjsewell"><code>@chrisjsewell</code></a> in <a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1133">#1133</a></li> </ul> <h2>🐛 Bug Fixes</h2> <ul> <li>🐛 Use docname instead of source path in warning locations by <a href="https://github.com/chrisjsewell"><code>@chrisjsewell</code></a> in <a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1114">#1114</a></li> <li>🐛 Correctly encode <code>&</code> in Markdown URLs by not HTML-escaping <code>refuri</code> by <a href="https://github.com/chrisjsewell"><code>@chrisjsewell</code></a> in <a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1126">#1126</a></li> <li>🐛 Fix <code>RemovedInSphinx10Warning</code> for inventory item iteration by <a href="https://github.com/chrisjsewell"><code>@chrisjsewell</code></a> in <a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1129">#1129</a></li> <li>🐛 Pin <code>mdit-py-plugins>=0.6.1</code> for nested field list fix by <a href="https://github.com/chrisjsewell"><code>@chrisjsewell</code></a> in <a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1134">#1134</a></li> </ul> <h2>⬆️ Dependency Upgrades</h2> <ul> <li>⬆️ Upgrade to <code>markdown-it-py~=4.2</code> and <code>mdit-py-plugins~=0.6</code> by <a href="https://github.com/chrisjsewell"><code>@chrisjsewell</code></a> in <a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1128">#1128</a></li> <li>⬆️ Update pygments requirement from <code><2.20</code> to <code><2.21</code> by <a href="https://github.com/chrisjsewell"><code>@chrisjsewell</code></a> in <a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1117">#1117</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/mb"><code>@mb</code></a> made their first contribution in <a href="https://redirect.github.com/executablebooks/MyST-Parser/pull/1126">executablebooks/MyST-Parser#1126</a></li> <li><a href="https://github.com/Bizordec"><code>@Bizordec</code></a> made their first contribution in <a href="https://redirect.github.com/executablebooks/MyST-Parser/pull/1114">executablebooks/MyST-Parser#1114</a></li> <li><a href="https://github.com/ilia-kats"><code>@ilia-kats</code></a> made their first contribution in <a href="https://redirect.github.com/executablebooks/MyST-Parser/pull/1110">executablebooks/MyST-Parser#1110</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/executablebooks/MyST-Parser/compare/v5.0.0...v5.1.0">https://github.com/executablebooks/MyST-Parser/compare/v5.0.0...v5.1.0</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/executablebooks/MyST-Parser/blob/master/CHANGELOG.md">myst-parser's changelog</a>.</em></p> <blockquote> <h2>5.1.0 - 2026-05-13</h2> <h3>✨ New Features</h3> <ul> <li>✨ Add <code>"alert"</code> syntax extension for <a href="https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#alerts">GFM alerts</a> (e.g. <code>> [!NOTE]</code>), see <a href="https://github.com/executablebooks/MyST-Parser/blob/master/syntax/alerts"></a> by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1128">gh-pr:1128</a></li> <li>✨ Add <code>"gfm_autolink"</code> syntax extension for <a href="https://github.github.com/gfm/#autolinks-extension-">GFM autolinks</a>, see <a href="https://github.com/executablebooks/MyST-Parser/blob/master/syntax/gfm-autolink"></a> by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1128">gh-pr:1128</a></li> <li>✨ Add <code>myst_strikethrough_single_tilde</code> <a href="https://github.com/executablebooks/MyST-Parser/blob/master/sphinx/config-options">config option</a> to allow single tilde (<code>~</code>) for strikethrough by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1128">gh-pr:1128</a></li> <li>✨ Add <code>myst_colon_fence_exact_match</code> <a href="https://github.com/executablebooks/MyST-Parser/blob/master/sphinx/config-options">config option</a> to require the closing colon fence to have exactly the same number of colons as the opening, see <a href="https://github.com/executablebooks/MyST-Parser/blob/master/syntax/colon_fence"></a> by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1128">gh-pr:1128</a></li> </ul> <h3>👌 Improvements</h3> <ul> <li>👌 Update <a href="https://github.com/executablebooks/MyST-Parser/blob/master/sphinx/config-options"><code>myst_gfm_only</code></a> mode to use the unified <code>gfm_plugin</code>, which now includes GFM autolinks, alerts, and improved strikethrough/tasklist handling by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1128">gh-pr:1128</a></li> <li>👌 Improve MathJax 4 compatibility for Sphinx 9 by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1110">gh-pr:1110</a></li> <li>👌 Stop directive-option parsing at colon fences, fixing nested colon fence directives by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1133">gh-pr:1133</a></li> </ul> <h3>🐛 Bug Fixes</h3> <ul> <li>🐛 Use docname instead of source path in warning locations by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1114">gh-pr:1114</a></li> <li>🐛 Correctly encode <code>&</code> in Markdown URLs by not HTML-escaping <code>refuri</code> by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1126">gh-pr:1126</a></li> <li>🐛 Fix <code>RemovedInSphinx10Warning</code> for inventory item iteration by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1129">gh-pr:1129</a></li> <li>🐛 Pin <code>mdit-py-plugins>=0.6.1</code> for nested field list fix by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1134">gh-pr:1134</a></li> </ul> <h3>⬆️ Dependency Upgrades</h3> <ul> <li>⬆️ Upgrade to <code>markdown-it-py~=4.2</code> and <code>mdit-py-plugins~=0.6</code> by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1128">gh-pr:1128</a></li> <li>⬆️ Update pygments requirement from <code><2.20</code> to <code><2.21</code> by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1117">gh-pr:1117</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/executablebooks/MyST-Parser/compare/v5.0.0...v5.1.0">v5.0.0...v5.1.0</a></p> <h2>5.0.0 - 2026-01-15</h2> <p>This release significantly bumps the supported versions of core dependencies:</p> <h3>‼️ Breaking Changes</h3> <p>This release updates the minimum supported versions:</p> <ul> <li><strong>Python</strong>: <code>>=3.11</code> (dropped Python 3.10, tests up to 3.14)</li> <li><strong>Sphinx</strong>: <code>>=8,<10</code> (dropped Sphinx 7, added Sphinx 9)</li> <li><strong>Docutils</strong>: <code>>=0.20,<0.23</code> (dropped docutils 0.19, added docutils 0.22)</li> <li><strong>markdown-it-py</strong>: <code>~=4.0</code> (upgraded from v3)</li> </ul> <h3>⬆️ Dependency Upgrades</h3> <ul> <li>⬆️ Upgrade to markdown-it-py v4 by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1060">gh-pr:1060</a></li> <li>⬆️ Drop Python 3.10 and Sphinx 7 by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1059">gh-pr:1059</a></li> <li>⬆️ Drop docutils 0.19 by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1061">gh-pr:1061</a></li> <li>⬆️ Add support for Python 3.14 by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1075">gh-pr:1075</a></li> <li>⬆️ Support Sphinx v9 by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1076">gh-pr:1076</a></li> <li>⬆️ Allow docutils 0.22 by <a href="gh-user:chrisjsewell">gh-user:chrisjsewell</a> in <a href="gh-pr:1084">gh-pr:1084</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/executablebooks/MyST-Parser/commit/2871eb95750873ccec2c4ab1dac0568815b64ca5"><code>2871eb9</code></a> 🚀 Release v5.1.0 (<a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1135">#1135</a>)</li> <li><a href="https://github.com/executablebooks/MyST-Parser/commit/cc5db37fd06445d7d023f7f2e0c2c073730be9cf"><code>cc5db37</code></a> 🐛 FIX: Pin mdit-py-plugins>=0.6.1 for nested field list fix (<a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1134">#1134</a>)</li> <li><a href="https://github.com/executablebooks/MyST-Parser/commit/4ce57f94af31de53c8790ccfffa3107c64241d0d"><code>4ce57f9</code></a> 👌 Stop directive-option parsing at colon fences (<a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1133">#1133</a>)</li> <li><a href="https://github.com/executablebooks/MyST-Parser/commit/cfcc3278f9e8c7508aae4cea82f1dd9c5c111183"><code>cfcc327</code></a> ⬆️ Bump mypy from 2.0.0 to 2.1.0 (<a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1131">#1131</a>)</li> <li><a href="https://github.com/executablebooks/MyST-Parser/commit/691738c3d897f82577440e18079d8b990edb8e34"><code>691738c</code></a> ⬆️ Bump ruff from 0.15.10 to 0.15.12 (<a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1132">#1132</a>)</li> <li><a href="https://github.com/executablebooks/MyST-Parser/commit/0fb1ae983d7b0df68bd02a9f77b0bc45057edbaf"><code>0fb1ae9</code></a> 👌 IMPROVE: MathJax 4 compatibility (Sphinx 9) (<a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1110">#1110</a>)</li> <li><a href="https://github.com/executablebooks/MyST-Parser/commit/f153b4b8ae68cbe77b41942147cf5cb6464168f0"><code>f153b4b</code></a> ⬆️ Bump actions/setup-python from 5 to 6 (<a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1092">#1092</a>)</li> <li><a href="https://github.com/executablebooks/MyST-Parser/commit/93acf8dae502aba484b14a1d291366fc21f839e6"><code>93acf8d</code></a> [pre-commit.ci] pre-commit autoupdate (<a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1095">#1095</a>)</li> <li><a href="https://github.com/executablebooks/MyST-Parser/commit/a5f1d6963bc3ee361d25309a24ccc42e6860fd54"><code>a5f1d69</code></a> ⬆️ Update pygments requirement from <2.20 to <2.21 (<a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1117">#1117</a>)</li> <li><a href="https://github.com/executablebooks/MyST-Parser/commit/838129687219517e31a395736397c957d93532dd"><code>8381296</code></a> 🐛 FIX: Use docname instead of source path in warning locations (<a href="https://redirect.github.com/executablebooks/MyST-Parser/issues/1114">#1114</a>)</li> <li>Additional commits viewable in <a href="https://github.com/executablebooks/MyST-Parser/compare/v5.0.0...v5.1.0">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [sysinfo](https://github.com/GuillaumeGomez/sysinfo) from 0.38.4 to 0.39.2. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/GuillaumeGomez/sysinfo/blob/main/CHANGELOG.md">sysinfo's changelog</a>.</em></p> <blockquote> <h1>0.39.2</h1> <ul> <li>Windows: Greatly improve performance of <code>System::refresh_cpu_specifics</code> when CPU usage is not requested.</li> <li>iOS: Fix compilation error when <code>user</code> feature is enabled.</li> <li>Linux: Correctly set thread information for processes.</li> </ul> <h1>0.39.1</h1> <ul> <li>Linux: Fix wrong network numbers computation.</li> </ul> <h1>0.39.0</h1> <ul> <li>Update minimum supported rust version to <code>1.95</code>.</li> <li>Add new <code>NetworkData::operational_state</code> API.</li> <li>Add new <code>Process::cgroup_limits</code> API (only returning data on Linux).</li> <li>All supported systems other than Windows: Improve performance of <code>Networks::refresh*</code>.</li> <li>All supported systems other than Windows: Fix soundness issue when retrieving users.</li> <li>Linux: Take into account parent cgroup memory limits.</li> <li>Linux: Fix panic when retrieving process information on <code>ESXi</code>.</li> <li>FreeBSD: Use the name of dataset as <code>name</code> for zfs disks.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/GuillaumeGomez/sysinfo/commit/c43234a69dfc6f5c12e7d500e04e4ce5a5dc1b97"><code>c43234a</code></a> Update crate version to <code>0.39.2</code></li> <li><a href="https://github.com/GuillaumeGomez/sysinfo/commit/b71467e99dfea531667d83ba4b90e95581a7c50d"><code>b71467e</code></a> Update CHANGELOG for <code>0.39.2</code> version</li> <li><a href="https://github.com/GuillaumeGomez/sysinfo/commit/345915a6350514454527a1fb00484bd9e47bd7ba"><code>345915a</code></a> Improve code readability</li> <li><a href="https://github.com/GuillaumeGomez/sysinfo/commit/35f3c186da0629b6520e02e692fc06bfb14fdd1e"><code>35f3c18</code></a> Fix fmt and clippy</li> <li><a href="https://github.com/GuillaumeGomez/sysinfo/commit/4bf1fe4c078bc81c5acf370066a19266994414ca"><code>4bf1fe4</code></a> Windows: skip PDH setup when cpu_usage isn't requested (<a href="https://redirect.github.com/GuillaumeGomez/sysinfo/issues/1664">#1664</a>)</li> <li><a href="https://github.com/GuillaumeGomez/sysinfo/commit/2be72d77b095b884690a8d932d7cf2afdd638d70"><code>2be72d7</code></a> Stub <code>get_users</code> on iOS via <code>app_store::users</code></li> <li><a href="https://github.com/GuillaumeGomez/sysinfo/commit/c725c09d84ecf948906879eb5786fe1f22e6a642"><code>c725c09</code></a> linux: fix detect thread entries in ProcessesToUpdate::All mode</li> <li><a href="https://github.com/GuillaumeGomez/sysinfo/commit/6e883a32ffcc2b9fa8146cb96daf78d61b1ee05e"><code>6e883a3</code></a> Update CHANGELOG and crate version to <code>0.39.1</code></li> <li><a href="https://github.com/GuillaumeGomez/sysinfo/commit/a45d38af0ba8066ac0367ca078970314aa9fcf1e"><code>a45d38a</code></a> Merge pull request <a href="https://redirect.github.com/GuillaumeGomez/sysinfo/issues/1659">#1659</a> from isaidsari/fix/linux-network-buffer-truncation</li> <li><a href="https://github.com/GuillaumeGomez/sysinfo/commit/ee9cd3b611a3167c8fb00077e2e5dc201d07fbc6"><code>ee9cd3b</code></a> Merge pull request <a href="https://redirect.github.com/GuillaumeGomez/sysinfo/issues/1655">#1655</a> from GuillaumeGomez/update</li> <li>Additional commits viewable in <a href="https://github.com/GuillaumeGomez/sysinfo/compare/v0.38.4...v0.39.2">compare view</a></li> </ul> </details> <br /> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
## Which issue does this PR close? - Closes apache#22267 ## Rationale for this change `regexp_count` did not handle empty regular-expression patterns correctly. An empty pattern should be counted as valid matches instead of returning `0`. This also affects calls that use the `start` argument and certain flag combinations. ## What changes are included in this PR? - Fix `regexp_count` so empty-pattern matches are counted correctly. - Adjust `start` handling so character offsets are computed correctly. - Update unit tests and sqllogictest coverage for empty patterns, `start`, and flags. - Update expected results to match the corrected behavior. ## Are these changes tested? - Yes. Rust unit tests were updated. - Yes. Sqllogictest coverage was added/updated. - I also ran: - `cargo test -p datafusion-sqllogictest --test sqllogictests table_functions` - `cargo test -p datafusion-sqllogictest --test sqllogictests scalar` ## Are there any user-facing changes? - Yes. `regexp_count` now returns correct counts for empty patterns, so results may differ from the previous behavior.
…#21643) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes apache#21642. - Relevant PR to datafusion-testing: apache/datafusion-testing#20 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> It just makes more sense - if the user knows that they want a particular filter before the others, let them do that. Also, the existing comments indicate that this was the original intention. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> The combined predicate of multiple filter plan nodes now preserves execution order. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes, added unit tests. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> Yes, execution order of predicates might change.
## Which issue does this PR close? N/A ## Rationale for this change `coerce_int96_to_resolution` currently produces `Timestamp(unit, None)` for every INT96-derived column. Some downstream readers need the resulting Arrow type to carry a timezone, because the *absence* of a timezone is itself meaningful. The motivating case is Apache DataFusion Comet (a Spark accelerator) trying to enforce [SPARK-36182\: pre-Spark-4 Spark rejects reading a Parquet TimestampLTZ column as TimestampNTZ](https://issues.apache.org/jira/browse/SPARK-36182). Comet's schema adapter pattern-matches `Timestamp(_, Some(_)) -> Timestamp(_, None)` to detect this case, but for INT96 columns the post-coerce type is `Timestamp(unit, None)` — indistinguishable from a true TimestampNTZ source. The LTZ signal is destroyed at the wrong layer. Spark and other systems write INT96 as UTC-adjusted instants, so a caller can ask for the column to surface as `Timestamp(unit, Some(\"UTC\"))`, preserving the LTZ semantic at the Arrow level. ## What changes are included in this PR? - New `TableParquetOptions.global.coerce_int96_tz: Option<String>` config field (defaults to `None`). - `coerce_int96_to_resolution` gains a `timezone: Option<Arc<str>>` parameter and threads it into the constructed `Timestamp` type. - The new option is plumbed through `ParquetSource` -> `ParquetOpener` / `ParquetMorselizer` -> `DFParquetMetadata`. - `with_coerce_int96_tz` builder method on `DFParquetMetadata`. - Default behavior is unchanged when the option is unset. ## Are these changes tested? Yes, see apache/datafusion-comet#4357 ## Are there any user-facing changes? A new \`coerce_int96_tz\` config option. No change in behavior for the default value. --------- Co-authored-by: Oleks V <comphead@users.noreply.github.com>
…on-wasm-app (apache#22321) Bumps [qs](https://github.com/ljharb/qs) and [body-parser](https://github.com/expressjs/body-parser). These dependencies needed to be updated together. Updates `qs` from 6.14.0 to 6.14.2 <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/ljharb/qs/blob/main/CHANGELOG.md">qs's changelog</a>.</em></p> <blockquote> <h2><strong>6.14.2</strong></h2> <ul> <li>[Fix] <code>parse</code>: mark overflow objects for indexed notation exceeding <code>arrayLimit</code> (<a href="https://redirect.github.com/ljharb/qs/issues/546">#546</a>)</li> <li>[Fix] <code>arrayLimit</code> means max count, not max index, in <code>combine</code>/<code>merge</code>/<code>parseArrayValue</code></li> <li>[Fix] <code>parse</code>: throw on <code>arrayLimit</code> exceeded with indexed notation when <code>throwOnLimitExceeded</code> is true (<a href="https://redirect.github.com/ljharb/qs/issues/529">#529</a>)</li> <li>[Fix] <code>parse</code>: enforce <code>arrayLimit</code> on <code>comma</code>-parsed values</li> <li>[Fix] <code>parse</code>: fix error message to reflect arrayLimit as max index; remove extraneous comments (<a href="https://redirect.github.com/ljharb/qs/issues/545">#545</a>)</li> <li>[Robustness] avoid <code>.push</code>, use <code>void</code></li> <li>[readme] document that <code>addQueryPrefix</code> does not add <code>?</code> to empty output (<a href="https://redirect.github.com/ljharb/qs/issues/418">#418</a>)</li> <li>[readme] clarify <code>parseArrays</code> and <code>arrayLimit</code> documentation (<a href="https://redirect.github.com/ljharb/qs/issues/543">#543</a>)</li> <li>[readme] replace runkit CI badge with shields.io check-runs badge</li> <li>[meta] fix changelog typo (<code>arrayLength</code> → <code>arrayLimit</code>)</li> <li>[actions] fix rebase workflow permissions</li> </ul> <h2><strong>6.14.1</strong></h2> <ul> <li>[Fix] ensure <code>arrayLimit</code> applies to <code>[]</code> notation as well</li> <li>[Fix] <code>parse</code>: when a custom decoder returns <code>null</code> for a key, ignore that key</li> <li>[Refactor] <code>parse</code>: extract key segment splitting helper</li> <li>[meta] add threat model</li> <li>[actions] add workflow permissions</li> <li>[Tests] <code>stringify</code>: increase coverage</li> <li>[Dev Deps] update <code>eslint</code>, <code>@ljharb/eslint-config</code>, <code>npmignore</code>, <code>es-value-fixtures</code>, <code>for-each</code>, <code>object-inspect</code></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/ljharb/qs/commit/bdcf0c7f82387c18ac8fabfccd2f440645cef47b"><code>bdcf0c7</code></a> v6.14.2</li> <li><a href="https://github.com/ljharb/qs/commit/294db90c812ddbe7d7a35d5687c505fd21a2d6a2"><code>294db90</code></a> [readme] document that <code>addQueryPrefix</code> does not add <code>?</code> to empty output</li> <li><a href="https://github.com/ljharb/qs/commit/5c308e5516c270a78caa6f278465914090f91ec6"><code>5c308e5</code></a> [readme] clarify <code>parseArrays</code> and <code>arrayLimit</code> documentation</li> <li><a href="https://github.com/ljharb/qs/commit/6addf8cf738d529c54d91f6f3ffb6c1be91bbfdc"><code>6addf8c</code></a> [Fix] <code>parse</code>: mark overflow objects for indexed notation exceeding <code>arrayLimit</code></li> <li><a href="https://github.com/ljharb/qs/commit/cfc108f662326d6ab540f3545ef0b832baf83cdf"><code>cfc108f</code></a> [Fix] <code>arrayLimit</code> means max count, not max index, in <code>combine</code>/<code>merge</code>/`pars...</li> <li><a href="https://github.com/ljharb/qs/commit/febb64442a80e49200211fa38d3c96b58024ac77"><code>febb644</code></a> [Fix] <code>parse</code>: throw on <code>arrayLimit</code> exceeded with indexed notation when `thr...</li> <li><a href="https://github.com/ljharb/qs/commit/f6a7abff1f13d644db9b05fe4f2c98ada6bf8482"><code>f6a7abf</code></a> [Fix] <code>parse</code>: enforce <code>arrayLimit</code> on <code>comma</code>-parsed values</li> <li><a href="https://github.com/ljharb/qs/commit/fbc5206c25b4d1851cea683f02c10756c521d15a"><code>fbc5206</code></a> [Fix] <code>parse</code>: fix error message to reflect arrayLimit as max index; remove e...</li> <li><a href="https://github.com/ljharb/qs/commit/1b9a8b4e78c6aff4c22fa559107227f02fd0216a"><code>1b9a8b4</code></a> [actions] fix rebase workflow permissions</li> <li><a href="https://github.com/ljharb/qs/commit/2a35775614e0fb46ac8a3060201a32a7c23a7fda"><code>2a35775</code></a> [meta] fix changelog typo (<code>arrayLength</code> → <code>arrayLimit</code>)</li> <li>Additional commits viewable in <a href="https://github.com/ljharb/qs/compare/v6.14.0...v6.14.2">compare view</a></li> </ul> </details> <br /> Updates `body-parser` from 1.20.3 to 1.20.5 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/expressjs/body-parser/releases">body-parser's releases</a>.</em></p> <blockquote> <h2>v1.20.5</h2> <h2>What's Changed</h2> <p>The reason for this release is a fix to the extended urlencoded parser returning objects instead of arrays for large array inputs (> 100) on qs@6.14.2+. (<a href="https://redirect.github.com/expressjs/body-parser/pull/716">expressjs/body-parser#716</a>)</p> <ul> <li>refactor(json): simplify strict mode error string construction by <a href="https://github.com/jonchurch"><code>@jonchurch</code></a> in <a href="https://redirect.github.com/expressjs/body-parser/pull/692">expressjs/body-parser#692</a></li> <li>fix: correct off-by-one error in parameterCount by <a href="https://github.com/abhu85"><code>@abhu85</code></a> in <a href="https://redirect.github.com/expressjs/body-parser/pull/716">expressjs/body-parser#716</a></li> <li>deps(qs): bump qs to 6.15.1 by <a href="https://github.com/jonchurch"><code>@jonchurch</code></a> in <a href="https://redirect.github.com/expressjs/body-parser/pull/722">expressjs/body-parser#722</a></li> <li>Release: 1.20.5 by <a href="https://github.com/jonchurch"><code>@jonchurch</code></a> in <a href="https://redirect.github.com/expressjs/body-parser/pull/721">expressjs/body-parser#721</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/abhu85"><code>@abhu85</code></a> made their first contribution in <a href="https://redirect.github.com/expressjs/body-parser/pull/716">expressjs/body-parser#716</a></li> </ul> <p>Special thanks to triager <a href="https://github.com/krzysdz"><code>@krzysdz</code></a> for keeping this on our radar and effectively triaging the specific issue!</p> <p><strong>Full Changelog</strong>: <a href="https://github.com/expressjs/body-parser/compare/1.20.4...1.20.5">https://github.com/expressjs/body-parser/compare/1.20.4...1.20.5</a></p> <h2>1.20.4</h2> <h2>What's Changed</h2> <ul> <li>Remove redundant depth check by <a href="https://github.com/blakeembrey"><code>@blakeembrey</code></a> in <a href="https://redirect.github.com/expressjs/body-parser/pull/538">expressjs/body-parser#538</a></li> <li>ci: add support for Node.js v23 by <a href="https://github.com/Phillip9587"><code>@Phillip9587</code></a> in <a href="https://redirect.github.com/expressjs/body-parser/pull/553">expressjs/body-parser#553</a></li> <li>ci: restore CI for 1.x branch by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/expressjs/body-parser/pull/665">expressjs/body-parser#665</a></li> <li>deps: qs@^6.14.0 by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/expressjs/body-parser/pull/664">expressjs/body-parser#664</a></li> <li>deps: use tilde notation and update certain dependencies by <a href="https://github.com/Phillip9587"><code>@Phillip9587</code></a> in <a href="https://redirect.github.com/expressjs/body-parser/pull/668">expressjs/body-parser#668</a></li> <li>chore: remove SECURITY.md by <a href="https://github.com/Phillip9587"><code>@Phillip9587</code></a> in <a href="https://redirect.github.com/expressjs/body-parser/pull/669">expressjs/body-parser#669</a></li> <li>ci: add CodeQL (SAST) by <a href="https://github.com/Phillip9587"><code>@Phillip9587</code></a> in <a href="https://redirect.github.com/expressjs/body-parser/pull/670">expressjs/body-parser#670</a></li> <li>Release: 1.20.4 by <a href="https://github.com/UlisesGascon"><code>@UlisesGascon</code></a> in <a href="https://redirect.github.com/expressjs/body-parser/pull/672">expressjs/body-parser#672</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/expressjs/body-parser/compare/1.20.3...1.20.4">https://github.com/expressjs/body-parser/compare/1.20.3...1.20.4</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/expressjs/body-parser/blob/1.20.5/HISTORY.md">body-parser's changelog</a>.</em></p> <blockquote> <h1>1.20.5 / 2026-04-24</h1> <ul> <li>refactor(json): simplify strict mode error string construction</li> <li>fix: extended urlencoded parsing of arrays with >100 elements (<a href="https://redirect.github.com/expressjs/body-parser/issues/716">#716</a>)</li> <li>deps: qs@~6.15.1</li> </ul> <h1>1.20.4 / 2025-12-01</h1> <ul> <li>deps: qs@~6.14.0</li> <li>deps: use tilde notation for dependencies</li> <li>deps: http-errors@~2.0.1</li> <li>deps: raw-body@~2.5.3</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/expressjs/body-parser/commit/0defdbe7f95ad0d3bc007d3a7c59c8c0ab9e6575"><code>0defdbe</code></a> release(patch): 1.20.5</li> <li><a href="https://github.com/expressjs/body-parser/commit/cd0e7a000c53e7be7262d303e57a352b6a00db7f"><code>cd0e7a0</code></a> deps(qs): bump qs to 6.15.1</li> <li><a href="https://github.com/expressjs/body-parser/commit/6f24d7e8bcd9860b136920926ce86da1a7dd1d51"><code>6f24d7e</code></a> fix: correct off-by-one error in parameterCount (<a href="https://redirect.github.com/expressjs/body-parser/issues/716">#716</a>)</li> <li><a href="https://github.com/expressjs/body-parser/commit/b849bd533d8b4abf5576a3e301f28d9befa05ddd"><code>b849bd5</code></a> deps: qs@~6.14.1 (<a href="https://redirect.github.com/expressjs/body-parser/issues/690">#690</a>)</li> <li><a href="https://github.com/expressjs/body-parser/commit/2c55e2f712f320a8e8d0f9fcb1d06526d0e401c9"><code>2c55e2f</code></a> refactor(json): simplify strict mode error string construction (<a href="https://redirect.github.com/expressjs/body-parser/issues/692">#692</a>)</li> <li><a href="https://github.com/expressjs/body-parser/commit/7db202cac84a001e6566c2dc6516b44db98beff3"><code>7db202c</code></a> 1.20.4 (<a href="https://redirect.github.com/expressjs/body-parser/issues/672">#672</a>)</li> <li><a href="https://github.com/expressjs/body-parser/commit/d8f8adb898676dfdf997b4455e5f9b689b53e989"><code>d8f8adb</code></a> ci: add CodeQL (SAST) (<a href="https://redirect.github.com/expressjs/body-parser/issues/670">#670</a>)</li> <li><a href="https://github.com/expressjs/body-parser/commit/6d133c19b3e7c0bb8301959ca1dba283d23d23c3"><code>6d133c1</code></a> chore: remove SECURITY.md (<a href="https://redirect.github.com/expressjs/body-parser/issues/669">#669</a>)</li> <li><a href="https://github.com/expressjs/body-parser/commit/fcd15355041ada6f37288dd13858d50429016b66"><code>fcd1535</code></a> deps: use tilde notation and update certain dependencies (<a href="https://redirect.github.com/expressjs/body-parser/issues/668">#668</a>)</li> <li><a href="https://github.com/expressjs/body-parser/commit/ec5fa290d25d85e0049757e240249072331eaee6"><code>ec5fa29</code></a> deps: qs@~6.14.0 (<a href="https://redirect.github.com/expressjs/body-parser/issues/664">#664</a>)</li> <li>Additional commits viewable in <a href="https://github.com/expressjs/body-parser/compare/1.20.3...1.20.5">compare view</a></li> </ul> </details> <details> <summary>Maintainer changes</summary> <p>This version was pushed to npm by <a href="https://www.npmjs.com/~jonchurch">jonchurch</a>, a new releaser for body-parser since your current version.</p> </details> <br /> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
## Which issue does this PR close? Addresses the immediate problem identified in apache/datafusion-python#1551 and the first recommendation in apache#22367 ## Rationale for this change We expect Literal and Column expressions and they get `Transform::no` as expected. If we have other PhysicalExpr that similarly should not get through because they have no children then they should likewise get `Transform::no`. This was found when a FFI wrapped Column expression failed in simplification. ## What changes are included in this PR? Essentially a two line change to see if it is a leaf node (no children) instead of just checking that all of the children are literals. ## Are these changes tested? Unit test added. ## Are there any user-facing changes? No user facing changes. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…e#21895) ## Which issue does this PR close? Partially addresses apache#14509 — implements `array_filter` / `list_filter`. ## Rationale for this change `array_transform` (apache#21679) added the first `HigherOrderUDF`. `array_filter` is the natural companion: filter array elements with a boolean lambda, matching Spark `filter` / DuckDB `list_filter` semantics. ## What changes are included in this PR? - New `HigherOrderUDF` `ArrayFilter` (`array_filter` / `list_filter` alias) - Boolean lambda per element; `true` keeps, `false`/null drops (matches Spark semantics) - Handles `List`, `LargeList`, sliced arrays, null sublists - Scalar predicate short-circuit (`x -> true` / `x -> false`) - No-copy fast path when nothing is filtered (skips `arrow::compute::filter`) - Shared HOF helpers extracted from `array_transform` into a common module (`value_lambda_pair`, `coerce_single_list_arg`, `single_list_lambda_parameters`, `extract_list_values`) - Shared unit test helpers for higher-order function tests ## Are these changes tested? - Unit tests: basic filter, multiple sublists, sliced arrays, null sublists, all-filtered-out, nothing-filtered (fast path), scalar true/false predicates - SQL logic tests in `array_filter.slt`: filter variants, `array_filter` + `array_transform` combinations, error cases ## Are there any user-facing changes? Yes — `array_filter(array, lambda)` and alias `list_filter(array, lambda)` are now available as SQL functions.
## Which issue does this PR close? - Follow on to apache#21743 - Related to apache#22102 ## Rationale for this change As we have discovered, sometimes seemingly minor changes in SQL semantics have caused non trivial downstream pain, for example the recent changes to NULL semantics in `ANY/ALL`. See - apache#21743 (comment) ## What changes are included in this PR? Add a note to the the API guide making it clear that changes to SQL semantics are also API changes too and deserve extra attention. ## Are these changes tested? By CI ## Are there any user-facing changes? Clearer API docs
…20337)" (apache#22437) (apache#22445) - Backports apache#22437 from @alamb to the branch-54 line This PR cherry-picks the revert of `ExecutionPlan::apply_expressions()` (apache#20337) onto `branch-54` so that DataFusion 54.0 does not ship the new public API.
…apache#22443) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> Backport apache#22404 - Closes #. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
…ter stats-based file reorder (apache#22501) ## Which issue does this PR close? Cherry-pick of apache#22493 onto `branch-54`. ## Rationale for this change `branch-54` includes apache#21956 (`feat: globally reorder files and row groups by statistics for TopK queries`), which introduced a regression: for plain-column, multi-file scans where the on-disk file order does not match the declared sort order, `SortExec` was no longer eliminated even when stats-based reorder produced non-overlapping file groups whose declared ordering re-validated. apache#22493 restores the pre-apache#21956 sort-elimination behaviour by re-validating `output_ordering` after `rebuild_with_source` reorders files, and (per @adriangb's correctness follow-up) restoring the original hint-free `file_source` on the Inexact→Exact upgrade so leftover `reverse_row_groups` / `sort_order_for_reorder` hints don't mis-order row groups within a single file once the `SortExec` safety net is gone. ## What changes are included in this PR? Straight cherry-pick of merge commit `94c58d086`. Includes: - `FileScanConfig::try_pushdown_sort` Inexact arm: re-validate, upgrade to Exact (with file_source restore), guard with NULL safety + early-return - `rebuild_with_source`: `match (all_non_overlapping, is_exact)` decision table for keep_ordering - SLT updates restoring `SortExec` elimination expectations + Tests 5b/5c/8b for the NULL-safety and same-min row-group edge cases ## Are these changes tested? Cherry-picked cleanly (auto-merge in `sort_pushdown.rs`). `cargo build -p datafusion-datasource` — passes. `cargo test -p datafusion-sqllogictest --test sqllogictests -- sort_pushdown` — passes. ## Are there any user-facing changes? Same as apache#22493: plain-column wrong-order-files cases regain SortExec elimination when files happen to be non-overlapping by statistics. No new API. Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… container types (apache#21934) (apache#22446) - Backports apache#21934 from @bert-beyondloops to the branch-54 line This PR cherry-picks the fix for `ScalarValue::compact` to compact view buffers for all container types onto `branch-54`. Co-authored-by: Bert Vermeiren <103956021+bert-beyondloops@users.noreply.github.com> Co-authored-by: Bert Vermeiren <bert.vermeiren@datadobi.com> Co-authored-by: Dmitrii Blaginin <dmitrii@blaginin.me>
## Which issue does this PR close? - Refs apache#22557. - Companion PR to apache#22559, targeting `branch-54`. ## Rationale for this change DataFusion 54 changed `ExecutionPlan` downcasting to use the `Any` supertrait directly. That removes `ExecutionPlan::as_any`, which had also served as a customization point for wrapper nodes: wrappers could identify as themselves internally while exposing the wrapped plan type to normal downcast-based inspection. This PR adds an explicit `ExecutionPlan::downcast_delegate()` hook for wrapper nodes that want their public `ExecutionPlan` downcast identity to be delegated to another plan. The proposed behavior intentionally preserves the old `as_any` override semantics: when a node opts into downcast delegation, intermediate delegating wrappers are invisible to `dyn ExecutionPlan::is::<T>()` and `downcast_ref::<T>()`. ## What changes are included in this PR? - Adds `ExecutionPlan::downcast_delegate()` with a default implementation returning `None`. - Updates `dyn ExecutionPlan::is::<T>()` and `downcast_ref::<T>()` to delegate to `downcast_delegate()` when present, otherwise use the current concrete plan type. - Documents that `downcast_delegate()` is only for type introspection and is independent from `children()` / plan traversal. - Adds tests for direct and nested downcast-delegating wrappers, including that intermediate delegating wrappers remain invisible to normal downcast-based inspection. ## Are these changes tested? Yes. - `cargo test -p datafusion-physical-plan execution_plan_downcast` - `cargo test -p datafusion-physical-plan --lib` - `cargo fmt --all -- --check` - `git diff --check` ## Are there any user-facing changes? Yes. This adds a new public `ExecutionPlan` trait method with a default implementation, and it changes `ExecutionPlan` downcast helpers to honor wrappers that explicitly opt into delegating public downcast identity.
) (apache#22634) - Part of apache#21080 This PR: - Backports apache#22571 from @kumarUjjawal to the `branch-54` line Co-authored-by: Kumar Ujjawal <ujjawalpathak6@gmail.com>
…erUDF struct (apache#22593) (apache#22635) - Part of apache#21080 This PR: - Backports apache#22593 from @LiaCastaneda to the `branch-54` line ## Note on conflict resolution The cherry-pick had conflicts in two test areas: - `datafusion/functions-nested/src/array_any_match.rs` — only the test-module `use` line conflicted; `branch-54`'s test does not reference the renamed symbols, so the existing import was kept (the production `HigherOrderUDF` -> `HigherOrderUDFImpl` rename applied cleanly). - `datafusion/substrait/tests/cases/roundtrip_logical_plan.rs` — the upstream PR modified the `roundtrip_array_transform_higher_order_function` test and `ArrayTransform` helper, but that test was added to `main` after `branch-54` was cut and does not exist on `branch-54`. It is out of scope for this refactor backport, so the `branch-54` file was left unchanged. Co-authored-by: Lía Adriana <lia.castaneda@datadoghq.com>
…ryToJoin` (#… (apache#22693) This is PR packports apache#22316 from @neilconway to branch-54 Co-authored-by: Neil Conway <neil.conway@gmail.com>
…pache#22530) (apache#22690) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change This PR backports apache#22530 ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
…Pushdown (apache#22525) (apache#22631) - Part of apache#21080 This PR: - Backports apache#22525 from @kumarUjjawal to the `branch-54` line Co-authored-by: Kumar Ujjawal <ujjawalpathak6@gmail.com>
…flag (backport apache#22632) (apache#22648) ## Which issue does this PR close? - Backport of apache#22632 to `branch-54`. ## Rationale for this change Content-defined chunking (CDC) write options were added in apache#21110 and are slated for the 54.0.0 release. This backports the refactor in apache#22632 so the config/proto surface ships in its final form, before the release goes out. The CDC options previously worked as `use_content_defined_chunking: Option<CdcOptions>` with a `ConfigField` impl that accepted a bare `use_content_defined_chunking = true|false` and otherwise enabled CDC implicitly when any sub-field was set. This has a few problems: - **Naming diverges from parquet-rs.** `WriterProperties` exposes `content_defined_chunking()` / `set_content_defined_chunking(Option<CdcOptions>)` with no `use_` prefix. - **Implicit / order-dependent on the SQL side.** Format options in `COPY ... OPTIONS` / `CREATE EXTERNAL TABLE ... OPTIONS` are applied from a `HashMap` (non-deterministic order). With the old bare-boolean form, mixing `... = false` with a sub-field could resolve to enabled or disabled depending on iteration order. - **Extra machinery.** Supporting the bare boolean required hand-written `ConfigField` impls and a `#[expect(clippy::should_implement_trait)]` workaround, plus a zero-sentinel fallback in the proto mapping. Since CDC is unreleased, the config/proto surface can still be changed freely. ## What changes are included in this PR? - Rename the `ParquetOptions` field `use_content_defined_chunking` -> `content_defined_chunking` (matches parquet-rs). - Make `CdcOptions` a plain `config_namespace!` with an explicit `enabled: bool` field alongside the chunking parameters; the field is a bare `CdcOptions` (no longer `Option<CdcOptions>`). CDC is on iff `content_defined_chunking.enabled` is true. Setting a parameter no longer implicitly enables CDC, and the result is independent of key order. - Add `CdcOptions::enabled()` / `CdcOptions::disabled()` shorthand constructors. - Drop the `ConfigField` impls and the `should_implement_trait` workaround — all generated by the macro now. - Add an `enabled` field to the proto `CdcOptions` message so the proto <-> config mapping is a plain field copy in both directions. - Update unit tests, regenerate config docs + the `information_schema` snapshot, and add `parquet_cdc_config.slt` documenting the resolution behavior. ## Are these changes tested? Yes — `datafusion-common` config + writer unit tests, `datafusion-proto-common` proto round-trip tests, `datafusion/core` parquet integration tests, and sqllogictest (`parquet_cdc.slt` + new `parquet_cdc_config.slt`). Cherry-pick applied cleanly onto `branch-54`; affected crates build and the CDC unit tests pass. ## Are there any user-facing changes? Yes, but only to the unreleased CDC options: - Config key `datafusion.execution.parquet.use_content_defined_chunking` -> `datafusion.execution.parquet.content_defined_chunking.enabled` (plus `.min_chunk_size` / `.max_chunk_size` / `.norm_level`). - The bare-boolean form is removed; enable/disable via `content_defined_chunking.enabled = true|false`. No released API is affected. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…trip (backport apache#22104) (apache#22785) ## Which issue does this PR close? - Backport of apache#22104 to `branch-54` (for 54.1.0, tracked in apache#22547). This PR: - Backports apache#22104 to the `branch-54` line so the `null_aware` proto round-trip fix ships in 54.1.0, as requested in apache#22065 (comment) Clean cherry-pick; `datafusion-proto` builds and both round-trip regression tests pass on `branch-54`.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.