refactor: simplify template logic

[fbiville]

This commit moves all source reads and Neo4j writes to workers.
The template launcher node does not need any access to sources nor
Neo4j anymore.

It also builds from import-spec's ImportPipeline API, which does
most of the heavy lifting and offers a more convenient DAG view of
the import to run.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the template logic to improve performance and maintainability by offloading data processing tasks to worker nodes. It also updates core dependencies and cleans up the codebase by introducing modern Java features and improving naming conventions across the project.

Highlights

• Architecture Refactoring: Moved all source reads and Neo4j writes to workers, removing the need for the driver node to access sources or Neo4j directly.
• Dependency Update: Bumped import-spec to 1.0.0-rc13, which includes changes to property type representations.
• Code Cleanup and Modernization: Replaced OptionsParams with a record-based OverlayTokens, renamed various classes for clarity (e.g., Provider to SourceProvider), and simplified template logic.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[codecov]

Codecov Report

❌ Patch coverage is 63.05556% with 133 lines in your changes missing coverage. Please review.
✅ Project coverage is 55.95%. Comparing base (2d76a99) to head (d5058cc).

Files with missing lines	Patch %	Lines
.../v2/neo4j/templates/Neo4jImportPipelineRunner.java	80.00%	27 Missing ⚠️
...eleport/v2/neo4j/templates/GoogleCloudToNeo4j.java	0.00%	23 Missing ⚠️
...t/v2/neo4j/transforms/VerifyOrResetDatabaseFn.java	0.00%	15 Missing ⚠️
...ogle/cloud/teleport/v2/neo4j/utils/ModelUtils.java	62.85%	9 Missing and 4 partials ⚠️
...ud/teleport/v2/neo4j/database/CypherGenerator.java	71.42%	7 Missing and 3 partials ⚠️
...eport/v2/neo4j/providers/text/TextTargetToRow.java	0.00%	9 Missing ⚠️
.../teleport/v2/neo4j/templates/ValidationErrors.java	0.00%	9 Missing ⚠️
...ort/v2/neo4j/transforms/Neo4jBlockingUnwindFn.java	42.85%	7 Missing and 1 partial ⚠️
...o4j/providers/bigquery/BigQuerySourceProvider.java	0.00%	6 Missing ⚠️
...eleport/v2/neo4j/transforms/Neo4jInitSchemaFn.java	83.87%	4 Missing and 1 partial ⚠️
... and 3 more

❌ Your patch check has failed because the patch coverage (63.05%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3582      +/-   ##
============================================
+ Coverage     55.70%   55.95%   +0.25%     
- Complexity     6774     6786      +12     
============================================
  Files          1125     1124       -1     
  Lines         68437    68216     -221     
  Branches       7719     7703      -16     
============================================
+ Hits          38121    38171      +50     
+ Misses        27844    27577     -267     
+ Partials       2472     2468       -4     

Components	Coverage Δ
spanner-templates	87.67% <ø> (ø)
spanner-import-export	68.70% <ø> (-0.02%)	⬇️
spanner-live-forward-migration	89.29% <ø> (ø)
spanner-live-reverse-replication	83.46% <ø> (ø)
spanner-bulk-migration	92.50% <ø> (ø)
gcs-spanner-dv	88.55% <ø> (ø)

Files with missing lines	Coverage Δ
...oud/teleport/v2/neo4j/database/CypherPatterns.java	72.52% <100.00%> (-2.97%)	⬇️
.../teleport/v2/neo4j/model/helpers/StepSequence.java	100.00% <100.00%> (ø)
...port/v2/neo4j/providers/SourceProviderFactory.java	37.50% <ø> (ø)
...rt/v2/neo4j/providers/text/TextSourceProvider.java	70.00% <100.00%> (+10.00%)	⬆️
...eleport/v2/neo4j/telemetry/ReportedSourceType.java	45.00% <100.00%> (+2.89%)	⬆️
...t/v2/neo4j/transforms/Neo4jRowWriterTransform.java	84.12% <100.00%> (+13.29%)	⬆️
...loud/teleport/v2/neo4j/utils/DataCastingUtils.java	71.01% <100.00%> (ø)
...leport/v2/neo4j/model/helpers/TargetQuerySpec.java	93.75% <80.00%> (+30.11%)	⬆️
...ort/v2/neo4j/transforms/CastExpandTargetRowFn.java	0.00% <0.00%> (ø)
...oogle/cloud/teleport/v2/neo4j/utils/BeamUtils.java	43.39% <20.00%> (-2.06%)	⬇️
... and 10 more

... and 16 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[gemini-code-assist]

Code Review

This pull request performs a major refactoring of the Google Cloud to Neo4j Dataflow template to utilize a new pipeline orchestration model centered around ImportPipeline and its step-based architecture. Key changes include the introduction of Neo4jImportPipelineRunner to manage pipeline construction, the replacement of OptionsParams with OverlayTokens, and the migration of various transforms and utilities to work with the new step-based API. Review feedback suggests enhancing input validation in OverlayTokenParser to handle whitespace, ensuring null safety when resolving dependencies in the pipeline runner, and removing the Serializable interface from the SourceContext class as it is only used during pipeline construction.

[gemini-code-assist]

Code Review

This pull request refactors the Google Cloud to Neo4j Dataflow template to integrate the latest Neo4j Importer library, replacing legacy model objects with ImportPipeline steps and introducing Neo4jImportPipelineRunner to manage pipeline construction. Key changes include the transition from OptionsParams to OverlayTokens and the refactoring of source providers and write transforms to utilize the new abstraction layers. Feedback highlights a potential bug in OverlayTokenParser where JSON null values are converted to the literal string "null", and a serialization concern in SourceContext due to the inclusion of a PCollection field in a class marked Serializable.

[gemini-code-assist]

Code Review

This pull request refactors the Google Cloud to Neo4j Dataflow template to integrate with the latest version of the Neo4j import specification library. The changes replace custom pipeline orchestration and job specification parsing with the library's ImportPipeline model, introducing the Neo4jImportPipelineRunner to manage Beam pipeline construction. Key components, including action mappers, source providers, and Cypher generation logic, have been updated to use new abstractions like TargetStep and OverlayTokens. I have no feedback to provide as there were no review comments to assess.

[nord-erik]

Looks great to me

[damccorm]

This PR is quite large, and as a result it is hard to review. It also mixes dependency changes, non-functional changes, and functional changes, which will make it hard to pinpoint a problem if something goes wrong. Would it be possible to break it up?

I'm thinking, for example, that you could break out the following components:

1. import-spec to 1.0.0-rc15 (and associated changes)
2. Move reading to workers from template launcher (which is the most likely thing to break things and probably the larges set of changes. Its possible this could be broken up even further.
3. Other miscelaneous non-functional changes
4. Other functional changes

[fbiville]

Happy to proceed but, as you suspect, the PR for step 2 will be likely quite similar in size to that one.

[fbiville]

Moving this to draft, as I've started to split the work into smaller PRs.

[gemini-code-assist]

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

[fbiville]

@damccorm PR has now been rebased

It is left with 2 things:

• moving to ImportPipeline (which is a DAG-aware API of the simpler ImportSpecification object)
• moving all the work to workers from template launchers (which explains the removal of all these PreloadXxx classes)

They're interleaved so I decided to keep them together in a single PR, hope that's OK.

[fbiville]

/gemini review

[gemini-code-assist]

Code Review

This pull request refactors the Neo4j import pipeline by migrating from a target-based execution model to a step-based pipeline model using the new ImportPipeline API. It introduces the Neo4jImportPipelineRunner to orchestrate the import steps, replaces preload actions with a startup verification/reset step, and updates utility classes such as CypherGenerator, CypherPatterns, BeamUtils, DataCastingUtils, and ModelUtils to support the new step-based classes. The reviewer's feedback focuses on ensuring that mandatory parameters—such as source contexts, target steps, and dependencies—are validated for nullability at the entry point or during configuration loading to prevent downstream NullPointerExceptions and avoid redundant null checks.