GH-3414: Add parseJson to VariantBuilder for JSON-to-Variant conversion by gaurav7261 · Pull Request #3415 · apache/parquet-java

gaurav7261 · 2026-03-07T06:50:27Z

Rationale for this change

Every consumer of parquet-variant currently has to independently implement JSON-to-Variant parsing. Apache Spark has one in its common/variant module
(source), our Kafka Connect S3 sink connector had to write one, and any other project (Flink, Trino, DuckDB-Java, etc.) would need to do the
same. Since VariantBuilder already provides all the low-level append*() primitives, parseJson() is the natural completion of that API — a canonical, reusable entry point for the most common use
case: converting a JSON string into a Variant.

What changes are included in this PR?

parquet-variant/pom.xml: Added jackson-core (compile) and parquet-jackson (runtime) dependencies, following the same pattern as parquet-hadoop.
VariantBuilder.java: Added two public static methods:
- parseJson(String json) — convenience method that creates a Jackson streaming parser internally.
- parseJson(JsonParser parser) — for callers who already have a positioned parser (e.g., reading from a stream).
- Internal helpers ported from Apache Spark's production VariantBuilder.buildJson:
  - buildJson() — recursive single-pass streaming parser handling OBJECT, ARRAY, STRING, NUMBER_INT, NUMBER_FLOAT, TRUE, FALSE, NULL.
  - appendSmallestLong() — selects the smallest integer type (BYTE/SHORT/INT/LONG) based on value range.
  - tryAppendDecimal() — decimal-first encoding for floating-point numbers; falls back to double only for scientific notation or values exceeding DECIMAL16 precision (38 digits).
TestVariantParseJson.java: 32 new tests covering all primitive types, objects (empty, simple, nested, null values, sorted keys, duplicate keys), arrays (empty, simple, nested, mixed types), and
edge cases (unicode, escaped strings, deeply nested documents, scientific notation, integer overflow to decimal, malformed JSON).

Are these changes tested?

Yes. 32 new tests in TestVariantParseJson.

Are there any user-facing changes?

Yes. Two new public static methods on VariantBuilder:

VariantBuilder.parseJson(String json) — returns a Variant
VariantBuilder.parseJson(JsonParser parser) — returns a Variant
These are additive API additions with no breaking changes to existing APIs.
Closes parseJson implementation in VariantBuilder #3414

gaurav7261 · 2026-03-07T06:54:46Z

@alamb @aihuaxu read https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/ and check the feasibility of having our S3 Sink connector write variant, found out that parseJson can be a better fit here, wdyt? is it making sense

…uctured fields Write Kafka Connect JSON fields (Debezium CDC, Confluent Protobuf Struct, custom messages, maps, arrays) as native Parquet VARIANT columns instead of plain STRING, improving storage efficiency and query performance. - Upgrade parquet-java to 1.17.0 for VARIANT logical type support - https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/ - Add config: parquet.variant.enabled, parquet.variant.connect.names, parquet.variant.field.names - Auto-detect recursive schemas (google.protobuf.Struct) as VARIANT - Stream JSON-to-Variant conversion ported from Apache Spark (PR also raised in parquet-java, once approved, code will be neat here in this repo: apache/parquet-java#3415) - Unwrap Protobuf Struct/Value/ListValue/map-as-array to clean JSON - Graceful fallback for non-JSON values (e.g. __debezium_unavailable_value) - Feature is fully opt-in (disabled by default), zero impact on existing connectors

gaurav7261 · 2026-03-09T16:36:41Z

@Fokko can you please review, is it looking good to you?

alamb · 2026-03-11T17:29:26Z

FWIW we have a similar method in Rust in case that is interesting

https://github.com/apache/arrow-rs/blob/d3c79006f2595e144d539f56b3054fe916ab184b/parquet-variant-compute/src/from_json.rs#L47

gaurav7261 · 2026-03-11T17:50:39Z

@julienledem thanks for the call, I have added notice, please review

gaurav7261 · 2026-03-16T16:14:38Z

@gszadovszky can you please review as well

wgtmac · 2026-03-17T09:24:08Z

I'm not familiar with this code yet but I think it is worth adding.

@emkornfield @gene-db @rdblue WDYT?

steveloughran

I think the json parsing should be in a class alongside VariantBuilder, rather than inside it, some VariantJsonParser class.

ensures that use of jackson classes are isolated; the core variant does not need jackson on the CP
if there is anything which can be done to improve single line JSON content performance, then it can be done there as it's life will span the whole file, rather than a row

Once it's split out, a new question surfaces: should it belong in parquet-jackson? Personally, I think it should

steveloughran · 2026-03-19T10:04:51Z

+This project includes code from Apache Spark with the following copyright
+notice:
+
+  Apache Spark


do cross-ASF projects need this credit? What I do think is good is ensure the original authors get credit in the final commit message

steveloughran · 2026-03-19T10:06:01Z

+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestVariantParseJson {


nested object parsing?

what about invalid json?

empty file

not a json file

incomplete

large json with many values

steveloughran · 2026-03-19T10:07:25Z

+   * @throws IOException if the JSON is malformed or an I/O error occurs
+   */
+  public static Variant parseJson(String json) throws IOException {
+    try (JsonParser parser = JSON_FACTORY.createParser(json)) {


can be quite slow

steveloughran · 2026-03-19T10:12:16Z

+   * @return the parsed Variant
+   * @throws IOException if the JSON is malformed or an I/O error occurs
+   */
+  public static Variant parseJson(String json) throws IOException {


I was to suggest making CharSequence for feeding in from other places (string fields within avro, ...) but it looks like jackson 2 doesn't support that itself.

steveloughran · 2026-03-19T10:21:29Z

 */
 public class VariantBuilder {

+  private static final JsonFactory JSON_FACTORY = new JsonFactory();


factory should be built with some constraints so that giving it a malicious json file should be rejected rather than trigger OOM problems, etc. From the javadocs.

JsonFactory f = JsonFactory.builder() .streamReadConstraints( StreamReadConstraints.builder() .maxNestingDepth(500) .maxStringLength(10_000_000) .maxDocumentLength(5_000_000) .build() ) .build();

gaurav7261 · 2026-03-19T21:46:09Z

@steveloughran can you please review again

gaurav7261 · 2026-04-02T09:53:50Z

@steveloughran can you please review

steveloughran

Commented. As I said before, I think this should go into parquet-jackson.

I see you've added jackson-core to the classpath. Is that because the parquet-jackson ones are shaded? If the class was in that module then it'd not be an issue.

of course, adding parquet-variant to the parquet-jackson module dependency list is its own complications, but I don't see any loops in the build.

Why not raise this PR on the dev list and bring up the "where should it go?" question.

steveloughran · 2026-04-02T13:39:47Z

      <version>${project.version}</version>
    </dependency>
+    <dependency>
+      <groupId>${jackson.groupId}</groupId>


comes with the parquet-jackson module; no need to reimport at a different scope

Hi @steveloughran , email is sent to dev@parquet.apache.org, can you please check there

Hi @steveloughran , saw your reply on email, so all looks good to you in above PR

steveloughran

I"m happy. Reviewed code and suggested some javadocs. Co-recursion is rare in production procedural code, so worth highlighting.

Test coverage good

steveloughran · 2026-04-13T13:14:52Z

+    }
+  }
+
+  private static void buildJsonObject(VariantBuilder builder, JsonParser parser) throws IOException {


nit: add javadoc explaining how this will co-recurse into buildJSON

steveloughran · 2026-04-13T13:15:12Z

+    builder.endObject();
+  }
+
+  private static void buildJsonArray(VariantBuilder builder, JsonParser parser) throws IOException {


same comment as above: mention co-recursion

done @steveloughran , please check

steveloughran · 2026-04-17T15:25:44Z

gaurav, you'll need to ask on the mail list for approval to run the workflows. Best to only do incremental changes on a PR

gaurav7261 · 2026-04-17T16:33:23Z

@steveloughran i don't have access to email archive, @alamb created thread for me, @alamb can you please help here?

gaurav7261 · 2026-04-23T04:59:20Z

@steveloughran can you please help here

gaurav7261 · 2026-04-24T04:35:36Z

@wgtmac i have fixed mvn spotless, please rerun

…onversion [apacheGH-3414] Extract JSON parsing into VariantJsonParser with StreamReadConstraints Move parseJson and helpers from VariantBuilder into dedicated VariantJsonParser class to isolate Jackson dependency. Add StreamReadConstraints to JsonFactory for safety against malicious input. Revert NOTICE (cross-ASF credit not needed). Add edge-case tests: empty input, non-JSON, incomplete array, large object. Ported from Apache Spark's VariantBuilder.parseJson.

gaurav7261 · 2026-04-24T14:16:09Z

@wgtmac java version issue in local, now fixed with java 17 spotless,please rerun again, thanks

gaurav7261 · 2026-04-30T08:22:50Z

@wgtmac can you please help run workflow

gaurav7261 · 2026-04-30T19:11:19Z

@emkornfield CI passed, can we merge now please

emkornfield · 2026-04-30T19:21:45Z

@gaurav7261 I think we need a committer to take a look. CC @wgtmac @Fokko @julienledem if you have time, otherwise I can take a look but I'm not super familiar with the java side of things.

wgtmac

+1 from my side

wgtmac · 2026-05-01T15:23:58Z

    </dependency>
+    <dependency>
+      <groupId>${jackson.groupId}</groupId>
+      <artifactId>jackson-core</artifactId>


Does this dependency change look good to you? @gszadovszky @Fokko

gaurav7261 · 2026-05-09T15:18:23Z

@wgtmac @Fokko can you please review and merge

wgtmac · 2026-05-12T07:42:11Z

Thanks @gaurav7261 for working on this and @steveloughran for the review!

gaurav7261 · 2026-05-12T08:06:16Z

thanks @wgtmac, @steveloughran , i see release 1.17.1 is released 5 hrs ago, just want to know by when next release will get release so that i can use same in flink parquet and confluent kafka s3 sink connector

wgtmac · 2026-05-12T08:18:01Z

1.17.1 is a patched release with bug fixes only. For this feature, it is targeted in the 1.18.0 which I don't have an ETA yet. Perhaps you may want to reply to the relevant discussion thread in the dev@parquet.apache.org.

gaurav7261 mentioned this pull request Mar 8, 2026

feat: add native Parquet VARIANT column support for JSON and semi structured fields confluentinc/kafka-connect-storage-cloud#917

Open

6 tasks

gaurav7261 force-pushed the gh-3414-variant-parse-json branch from 1274733 to 643ac64 Compare March 11, 2026 17:49

steveloughran reviewed Mar 19, 2026

View reviewed changes

gaurav7261 force-pushed the gh-3414-variant-parse-json branch from 643ac64 to 08c3672 Compare March 19, 2026 21:44

gaurav7261 requested a review from steveloughran March 19, 2026 21:46

steveloughran reviewed Apr 2, 2026

View reviewed changes

steveloughran approved these changes Apr 13, 2026

View reviewed changes

gaurav7261 force-pushed the gh-3414-variant-parse-json branch from 08c3672 to 9c3dbab Compare April 14, 2026 12:30

gaurav7261 force-pushed the gh-3414-variant-parse-json branch from 9c3dbab to 8459204 Compare April 24, 2026 04:34

gaurav7261 force-pushed the gh-3414-variant-parse-json branch from 8459204 to 814e6ef Compare April 24, 2026 14:15

wgtmac approved these changes May 1, 2026

View reviewed changes

wgtmac merged commit b9c11af into apache:master May 12, 2026
5 checks passed

Conversation

gaurav7261 commented Mar 7, 2026

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

gaurav7261 commented Mar 7, 2026

Uh oh!

gaurav7261 commented Mar 9, 2026

Uh oh!

alamb commented Mar 11, 2026

Uh oh!

gaurav7261 commented Mar 11, 2026

Uh oh!

gaurav7261 commented Mar 16, 2026

Uh oh!

wgtmac commented Mar 17, 2026

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaurav7261 commented Mar 19, 2026

Uh oh!

gaurav7261 commented Apr 2, 2026

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Apr 17, 2026

Uh oh!

gaurav7261 commented Apr 17, 2026

Uh oh!

gaurav7261 commented Apr 23, 2026

Uh oh!

gaurav7261 commented Apr 24, 2026

Uh oh!

gaurav7261 commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gaurav7261 commented Apr 30, 2026

Uh oh!

gaurav7261 commented Apr 30, 2026

Uh oh!

emkornfield commented Apr 30, 2026

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gaurav7261 commented May 9, 2026

Uh oh!

gaurav7261 commented Apr 24, 2026 •

edited

Loading

gaurav7261 commented May 12, 2026 •

edited

Loading