[DCP] Auto create tables when DB is empty

[gmechali]

Modify the ingestion pipeline's logic for DB creation to be a bit more flexible. Before this change, it just checks whether the DB exists, and if not it creates it with the appropriate scheme and protodescriptor.

With this change, we retain the ability to create the DB and initialize it properly, but we also allow customers to provide an empty DB, and have it be initialized with the proper schema and proto descriptor.
This is important for DCP, where the DB should be created via Terraform at setup time in order to also connect the DCP service to the DB, however the schema is not necessary at that point, and so we can expect the ingestion pipeline to validate the schema and create it when necessary.

I've tested this on all the following scenarios:

• DB exists and is properly initialized
• DB exists but has no tables or proto descriptor
• DB exists but has no tables only
• DB does not exist.

[codacy-production]

Not up to standards ⛔

🔴 Issues 3 medium · 3 minor

Alerts:
⚠ 6 issues (≤ 0 issues of at least minor severity)

Results:
6 new issues

Category	Results
CodeStyle	3 minor
Complexity	3 medium

View in Codacy

🟢 Metrics 22 complexity · 0 duplication

Metric	Results
Complexity	22
Duplication	0

View in Codacy

_{TIP This summary will be updated as you push new changes. Give us feedback}

[gemini-code-assist]

Code Review

This pull request enhances the SpannerClient to handle existing but uninitialized databases by checking for the presence of core tables and applying DDL statements if the database is empty. It also introduces unit tests for these new initialization checks. Review feedback suggests using configured table names instead of hardcoded strings for consistency, optimizing I/O by deferring DDL reading until needed, removing redundant fully qualified class names, and adding a missing assertion in the test suite.

[gmechali]

/gemini review

[gemini-code-assist]

Code Review

This pull request refactors the Spanner database initialization process by replacing the createDatabase method with a more comprehensive validateOrInitializeDatabase method. This new logic verifies the existence of required tables and proto bundles, handling cases where the database might be partially initialized or empty. Additionally, the PR introduces unit tests for SpannerClient and updates the project's dependencies to support testing. Review feedback suggests improving the robustness of DDL statement detection by using more specific string matching and adhering to the principle of programming to an interface by using List instead of ArrayList in method signatures.

[vish-cs]

Thanks for making this change. This was one of the pending items since the previous logic was incompatible with the Terraform setup.
https://github.com/datacommonsorg/data/blob/master/import-automation/terraform/main.tf

FYI, Terraform deployment script creates additional tables in the DB for import automation.
https://github.com/datacommonsorg/data/blob/master/import-automation/workflow/spanner_schema.sql

[gmechali]

Thanks for making this change. This was one of the pending items since the previous logic was incompatible with the Terraform setup. https://github.com/datacommonsorg/data/blob/master/import-automation/terraform/main.tf

FYI, Terraform deployment script creates additional tables in the DB for import automation. https://github.com/datacommonsorg/data/blob/master/import-automation/workflow/spanner_schema.sql

Got it - so the DCP Terraform script now can also just keep its current logic of creating the DataBase, but letting the ingestion script initialize the tables.

We also could have initialized the tables in the Terraform script, but I think that holds some pretty significant risk. If a user ends up making a change to the Table Schema and reapplies terraform, it does not modify the table, it recreates it, and therefore could drop all the data in the tables.
That was why I did not want the DCP TF to create the Node/Edge/Obs tables. Maybe for ImportStatus it's a bit less critical...but im not sure.

I will put some thought into it in a few days when I get to the TF Import Status setup for DCP.