Add batch mode processing for entities#1911

Merged

vish-cs merged 2 commits intodatacommonsorg:masterfrom

Apr 15, 2026

Contributor

vish-cs commented Mar 11, 2026 •

edited

Loading

Update import helper to run import batch job for Schema/Place imports and add a parameter to trigger ingestion workflow (dataflow job)
Update logic to generate import input prefix paths using a fixed naming scheme as input{x}
Skip import processing if differ output shows no changes
Add a shell script to process Schema/Place imports (download GCS files)

This comment was marked as outdated.

Sign in to view

This comment was marked as outdated.

Sign in to view

vish-cs force-pushed the node branch 2 times, most recently from ab403ee to 0c89543 Compare

March 12, 2026 08:51

vish-cs force-pushed the node branch from 0c89543 to c05b9d5 Compare

March 30, 2026 12:37

vish-cs changed the title ~~Add script to generate provisional nodes~~ Add script to convert DC manifest to mcf

vish-cs closed this

vish-cs reopened this

vish-cs force-pushed the node branch from c05b9d5 to f95000a Compare

April 3, 2026 10:35

vish-cs changed the title ~~Add script to convert DC manifest to mcf~~ Add batch mode for schema

vish-cs force-pushed the node branch from f95000a to aa9ed43 Compare

April 6, 2026 16:50

vish-cs requested a review from ajaits

April 6, 2026 17:24

vish-cs force-pushed the node branch 3 times, most recently from 5f7d75d to 57d525a Compare

April 7, 2026 15:11

ajaits reviewed

View reviewed changes

import-automation/executor/app/executor/import_executor.py Outdated

ajaits reviewed

View reviewed changes

import-automation/executor/app/executor/import_executor.py

ajaits reviewed

View reviewed changes

import-automation/executor/app/executor/import_executor.py

ajaits reviewed

View reviewed changes

import-automation/executor/app/executor/import_executor.py

ajaits reviewed

View reviewed changes

import-automation/executor/app/executor/import_executor.py

ajaits reviewed

View reviewed changes

import-automation/executor/app/executor/import_executor.py Outdated

ajaits reviewed

View reviewed changes

import-automation/executor/app/executor/import_executor.py

ajaits reviewed

View reviewed changes

import-automation/executor/app/executor/import_executor.py Outdated

ajaits reviewed

View reviewed changes

import-automation/workflow/import-helper/main.py Outdated

ajaits reviewed

View reviewed changes

import-automation/workflow/import-automation-workflow.yaml Outdated

ajaits reviewed

View reviewed changes

scripts/entities/download.sh Outdated

ajaits reviewed

View reviewed changes

scripts/entities/download.sh Outdated

ajaits reviewed

View reviewed changes

scripts/entities/download.sh

ajaits reviewed

View reviewed changes

scripts/entities/manifest.json Outdated

ajaits reviewed

View reviewed changes

scripts/entities/manifest.json

ajaits reviewed

View reviewed changes

tools/import_differ/import_differ.py

Contributor Author

vish-cs commented Apr 9, 2026

Thanks for the detailed review!

ajaits approved these changes

View reviewed changes

vish-cs force-pushed the node branch 5 times, most recently from b3a51cf to 40f1d63 Compare

April 15, 2026 07:31

vish-cs added 2 commits

April 15, 2026 07:32


          Add batch mode processing for entities

205106f


          Fixes for review comments

3df012c

vish-cs force-pushed the node branch from 40f1d63 to 3df012c Compare

April 15, 2026 07:32

vish-cs changed the title ~~Add batch mode for schema~~ Add batch mode processing for entities

vish-cs merged commit fbdac9c into datacommonsorg:master

9 checks passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet