[DCP] Adds JSON-LD Support to the ingestion pipeline by gmechali · Pull Request #495 · datacommonsorg/import

gmechali · 2026-04-07T14:33:30Z

This PR adds JSON-LD support for the ingestion pipeline. The first stage of the pipeline is responsible for converting from MCF and TMCF to an MCFGraph object. we now also add the ability to convert from JSON-LD and CSV with a .jsonld template to the same MCFGraph object.

We also refactored the MCF parsing logic into a util/parser subdirectory.
Added some unit tests.
We verified the behavior by running the pipeline with following two MCF, JSONLD.

See the data in the tables under this Spanner DB

codacy-production · 2026-04-07T14:35:01Z

Not up to standards ⛔

🔴 Issues 8 medium · 7 minor

Alerts:
⚠ 15 issues (≤ 0 issues of at least minor severity)

Results:
15 new issues

Category Results

BestPractice 2 medium

CodeStyle 7 minor

Complexity 6 medium

View in Codacy

🟢 Metrics 73 complexity · 6 duplication

Metric Results

Complexity 73

Duplication 6

View in Codacy

_{TIP This summary will be updated as you push new changes. Give us feedback}

gemini-code-assist

Code Review

This pull request adds support for JSON-LD and JSON-LD templates to the ingestion pipeline and command-line tools. Key changes include the introduction of JsonLdParser and JsonLdTemplateParser, refactoring FileGroup to support multiple formats, and updating the Beam pipeline to handle JSON-LD inputs. Review feedback identifies critical concerns regarding memory usage with large files, resource leaks from unclosed streams, and the need to handle single JSON-LD objects. Suggestions also include improving logging and ensuring file path checks accommodate sharded files.

pipeline/util/src/main/java/org/datacommons/ingestion/util/PipelineUtils.java

tool/src/main/java/org/datacommons/tool/Processor.java

util/src/main/java/org/datacommons/util/parser/jsonld/JsonLdParser.java

pipeline/ingestion/src/main/java/org/datacommons/ingestion/pipeline/GraphIngestionPipeline.java

util/src/main/java/org/datacommons/util/parser/jsonld/JsonLdParser.java

util/src/main/java/org/datacommons/util/jsonld/JsonLdParser.java

gmechali · 2026-04-07T15:35:54Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces comprehensive support for JSON-LD and JSON-LD Template + CSV files within the ingestion pipeline and linting tool. Key changes include refactoring the FileGroup mechanism using a strategy pattern to handle different input file types, adding new parsers (JsonLdParser, JsonLdTemplateParser), and updating pipeline logic to process these new formats. New tests for JSON-LD parsing are also included. The review identified several issues, including resource leaks in PipelineUtils and Processor when handling file streams, a package declaration mismatch, and the presence of debug/error print statements that should be replaced with proper logging. Additionally, a redundant JsonLdOptions object was noted.

pipeline/util/src/main/java/org/datacommons/ingestion/util/PipelineUtils.java

tool/src/main/java/org/datacommons/tool/Processor.java

util/src/main/java/org/datacommons/util/parser/jsonld/JsonLdParser.java

util/src/main/java/org/datacommons/util/parser/jsonld/JsonLdTemplateParser.java

gmechali · 2026-04-07T15:51:35Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for JSON-LD and JSON-LD templates across the ingestion pipeline and command-line tools. It includes new parsers, a refactored file grouping logic using a strategy pattern, and updates to the Beam pipeline to process these formats. Feedback highlights a critical issue in the template parser where local IDs in property values are not correctly resolved with row-specific suffixes, which would break node relationships. Other improvements include addressing potential encoding inconsistencies in the processor, removing redundant property assignments, and hardening the namespace stripping logic to prevent invalid empty property names.

util/src/main/java/org/datacommons/util/parser/jsonld/JsonLdTemplateParser.java

tool/src/main/java/org/datacommons/tool/Processor.java

util/src/main/java/org/datacommons/util/parser/jsonld/JsonLdParser.java

gmechali · 2026-04-07T22:28:26Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for JSON-LD input files by implementing a new JsonLdParser and refactoring the file grouping logic into a strategy-based pattern. Key changes include updates to the ingestion pipeline to handle .jsonld extensions, the addition of JsonLdFileGroup and McfFileGroup subclasses, and new unit tests for JSON-LD parsing. Feedback focuses on fixing a potential ClassCastException in Processor.java when handling non-MCF groups, correcting package declarations that do not match the directory structure, and restoring the logic to automatically detect tab delimiters for .tsv files.

gemini-code-assist · 2026-04-07T22:32:30Z

tool/src/main/java/org/datacommons/tool/Processor.java

+    }
+
+    GraphSupplier parser;
+    McfFileGroup mcfGroup = (McfFileGroup) args.fileGroup;


This unconditional cast to McfFileGroup will cause a ClassCastException if the input includes a .jsonld file, as FileGroup.build will return a JsonLdFileGroup in that case. Since JsonLdFileGroup can also contain CSV files (as seen in JsonLdParserStrategy), this method should handle both types or verify the type before casting. When verifying if a file is an MCF file, ensure you use contains('.mcf') instead of endsWith('.mcf') to correctly handle sharded files. Additionally, if a .jsonld file is intended to be used as a template for the CSV, the logic to retrieve it and pass it to the parser is currently missing for JsonLdFileGroup.

References

To handle sharded files, check for MCF files using contains('.mcf') on the file path, not endsWith('.mcf').

We arent yet support .jsonld in the tool processor and unclear that we ever will.

gemini-code-assist · 2026-04-07T22:32:30Z

util/src/main/java/org/datacommons/util/parser/strategy/JsonLdParserStrategy.java

@@ -0,0 +1,30 @@
+package org.datacommons.util;


The package declaration org.datacommons.util does not match the directory structure org/datacommons/util/parser/strategy/. This violates standard Java project structure and can lead to issues with build tools and IDEs. Please update the package declaration to org.datacommons.util.parser.strategy and add the necessary imports in FileGroup.java.

Suggested change

package org.datacommons.util;

package org.datacommons.util.parser.strategy;

gemini-code-assist · 2026-04-07T22:32:31Z

util/src/main/java/org/datacommons/util/parser/strategy/JsonLdParserStrategy.java

+      }
+    }
+
+    char delim = overrideDelimiter == null ? ',' : overrideDelimiter;


This change introduces a regression where the default delimiter is always a comma, even if the input files are .tsv. The previous logic in FileGroup.build correctly detected .tsv files and set the default delimiter to a tab character. Note that contains() should be used instead of endsWith() to support sharded files.

Suggested change

char delim = overrideDelimiter == null ? ',' : overrideDelimiter;

boolean hasTsv = files.stream().anyMatch(f -> f.getName().toLowerCase().contains(".tsv"));

char delim = overrideDelimiter == null ? (hasTsv ? '\t' : ',') : overrideDelimiter;

References

To handle sharded files, use contains() for extension checks instead of endsWith().

util/src/main/java/org/datacommons/util/parser/strategy/McfParserStrategy.java

util/src/main/java/org/datacommons/util/parser/strategy/ParserStrategy.java

dwnoble

Nit on the input JSON-LD files: all property names and non-literal values should either be namespaced or have the values explicitly defined in the @context

in the linked example, name , value, and StatVarObservation are missing @context definitions, and the "schema" and "dcid" namespaces should be defined in the context. I also think we should use the datacommons.org definitions for observationDate, observationAbout, variableMeasured, and value because there are slight differences between the schema.org and datacommons.org definitions 😅

the way you define properties like "variableMeasured": "http://schema.org/variableMeasured", is totally fine (minus the nitpick about using datacommons.org instead of schema.org), but personally i like to just define the namespaces and use those everywhere in the @graph definition like this:

{
  "@context": {
    "schema": "https://schema.org/",
    "dcid": "https://datacommons.org/browser/"
  },
  "@graph": [
    {
      "@id": "dcid:TestNode",
      "@type": "schema:Thing",
      "schema:name": "Test Node Name"
    },
    {
      "@id": "dcid:TestObservation_2020",
      "@type": "dcid:StatVarObservation",
      "dcid:observationAbout": {
        "@id": "dcid:TestNode"
      },
      "dcid:variableMeasured": {
        "@id": "dcid:Count_Person"
      },
      "dcid:value": 1000,
      "dcid:observationDate": "2020"
    }
]

(you can see the differences between those properties by going to, for example, https://schema.org/observationAbout and https://datacommons.org/browser/observationAbout)

…move the namespace in JSON-LD, note the MCF import does not yet support the namespace. We should ensure these will be supported together.

gmechali · 2026-04-14T21:55:36Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for JSON-LD input files within the ingestion pipeline and the command-line tool. Key changes include refactoring the FileGroup logic into a strategy-based pattern, adding a new JsonLdParser, and updating the GraphIngestionPipeline and Processor to handle multiple input formats. Review feedback highlights a potential OutOfMemoryError when parsing large JSON-LD files in memory and an unsafe type cast in the Processor class that could lead to a ClassCastException. Additionally, the reviewer suggests using contains instead of endsWith for file path checks to support sharded files, expanding prefix stripping to include non-HTTPS schema URIs, and ensuring that @type and @language tags are preserved during JSON-LD value processing.

gemini-code-assist · 2026-04-14T21:57:45Z

pipeline/util/src/main/java/org/datacommons/ingestion/util/PipelineUtils.java

+                  public void processElement(
+                      @Element FileIO.ReadableFile file, OutputReceiver<McfGraph> receiver) {
+                    try (InputStream is = Channels.newInputStream(file.open())) {
+                      McfGraph graph = JsonLdParser.parse(is);


Parsing the entire JSON-LD file into memory using JsonLdParser.parse(is) can lead to OutOfMemoryError for large input files. Consider implementing a streaming parser or processing the file in chunks if possible.

gemini-code-assist · 2026-04-14T21:57:45Z

tool/src/main/java/org/datacommons/tool/Processor.java

+    }
+
+    GraphSupplier parser;
+    McfFileGroup mcfGroup = (McfFileGroup) args.fileGroup;


This cast to McfFileGroup is unsafe. If the input files are JSON-LD with associated CSVs, args.fileGroup will be an instance of JsonLdFileGroup, leading to a ClassCastException.

if (!(args.fileGroup instanceof McfFileGroup)) { throw new IllegalArgumentException("processTable requires an McfFileGroup for TMCF/CSV processing."); } McfFileGroup mcfGroup = (McfFileGroup) args.fileGroup;

pipeline/util/src/main/java/org/datacommons/ingestion/util/PipelineUtils.java

gemini-code-assist · 2026-04-14T21:57:45Z

util/src/main/java/org/datacommons/util/parser/jsonld/JsonLdParser.java

+  private static final String SCHEMA_PREFIX = "https://schema.org/";
+
+  private static String stripPrefix(String value) {
+    if (value == null) {
+      return null;
+    }
+    if (value.startsWith(DCID_PREFIX)) {
+      return value.substring(DCID_PREFIX.length());
+    }
+    if (value.startsWith(SCHEMA_PREFIX)) {
+      return value.substring(SCHEMA_PREFIX.length());
+    }
+    return value;
+  }


The SCHEMA_PREFIX should also account for http://schema.org/ as many datasets still use the non-HTTPS version of the URI.

private static final String SCHEMA_PREFIX_HTTPS = "https://schema.org/"; private static final String SCHEMA_PREFIX_HTTP = "http://schema.org/"; private static String stripPrefix(String value) { if (value == null) { return null; } if (value.startsWith(DCID_PREFIX)) { return value.substring(DCID_PREFIX.length()); } if (value.startsWith(SCHEMA_PREFIX_HTTPS)) { return value.substring(SCHEMA_PREFIX_HTTPS.length()); } if (value.startsWith(SCHEMA_PREFIX_HTTP)) { return value.substring(SCHEMA_PREFIX_HTTP.length()); } return value; }

gemini-code-assist · 2026-04-14T21:57:45Z

util/src/main/java/org/datacommons/util/parser/jsonld/JsonLdParser.java

+      Map<?, ?> map = (Map<?, ?>) item;
+      if (map.containsKey("@value")) {
+        Object val = map.get("@value");
+        addProperty(nodeBuilder, property, val.toString(), Mcf.ValueType.TEXT);


When processing @value nodes, the parser currently ignores @type (datatype) and @language tags. This may result in loss of information for typed literals or multi-language strings.

util/src/main/java/org/datacommons/util/parser/strategy/JsonLdParserStrategy.java

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Adds JSON-LD support - still needs much more testing and validiton

8348329

gemini-code-assist bot reviewed Apr 7, 2026

View reviewed changes

Clean up for JSON-LD support. Verified MCF and JSON-ld

ac8a9a3

gemini-code-assist bot reviewed Apr 7, 2026

View reviewed changes

More cleanup

d3a0d9f

gemini-code-assist bot reviewed Apr 7, 2026

View reviewed changes

gmechali added 2 commits April 7, 2026 16:14

Gemini comment cleanup

be5b333

Adds a todo

b742f4e

gmechali requested a review from dwnoble April 7, 2026 21:24

gmechali added 3 commits April 7, 2026 17:32

Remove the JSONLD + CSV combo for now.

b6b1234

Undo the JSONLD Tmpl + CSV code which wasnt working

ac9b136

More cleanup for PR

79d48a6

gemini-code-assist bot reviewed Apr 7, 2026

View reviewed changes

gmechali requested a review from vish-cs April 8, 2026 13:37

dwnoble requested changes Apr 10, 2026

View reviewed changes

gmechali and others added 4 commits April 14, 2026 17:13

Addresses comment from Dan regarding namespaced values. We current re…

53923bc

…move the namespace in JSON-LD, note the MCF import does not yet support the namespace. We should ensure these will be supported together.

Merge branch 'master' into jsonld

cf19a3c

Code cleanup

895e5d5

more cleanup, tested and verified with mcf

def651a

gmechali requested a review from dwnoble April 14, 2026 21:56

gemini-code-assist bot reviewed Apr 14, 2026

View reviewed changes

Apply suggestions from code review

5c3504d

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

	package org.datacommons.util;
	package org.datacommons.util.parser.strategy;

	char delim = overrideDelimiter == null ? ',' : overrideDelimiter;
	boolean hasTsv = files.stream().anyMatch(f -> f.getName().toLowerCase().contains(".tsv"));
	char delim = overrideDelimiter == null ? (hasTsv ? '\t' : ',') : overrideDelimiter;

Conversation

gmechali commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codacy-production bot commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Not up to standards ⛔

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gmechali commented Apr 7, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gmechali commented Apr 7, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gmechali commented Apr 7, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

gmechali Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dwnoble left a comment

Choose a reason for hiding this comment

Uh oh!

gmechali commented Apr 14, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gemini-code-assist bot Apr 14, 2026

gmechali commented Apr 7, 2026 •

edited

Loading

codacy-production bot commented Apr 7, 2026 •

edited

Loading