Python models fail with MANAGED_TABLE_FORMAT error when use_managed_iceberg=true

[supremeoverlord-ofdoom]

Describe the bug

Python dbt models fail to materialize as managed Iceberg tables on Databricks. When use_managed_iceberg is enabled and a Python model is configured with table_format='iceberg', the adapter generates a PySpark writer that calls df.write.format("parquet").saveAsTable(...). This fails with an INVALID_PARAMETER_VALUE.MANAGED_TABLE_FORMAT: Only Delta is supported for managed tables. error.

The equivalent SQL model with table_format='iceberg' succeeds. The equivalent Python model without table_format='iceberg' (default Delta) also succeeds. Only the Python + managed-Iceberg combination fails.

Command run: dbt run on a Python model with table_format="iceberg" and file_format="iceberg" config; use_managed_iceberg: true set in dbt_project.yml.

Steps To Reproduce

1. Set the behavior flag in dbt_project.yml:

   flags:
     use_managed_iceberg: true

2. Create a Python model configured for managed Iceberg (this fails):

   def model(dbt, session):
       dbt.config(
           materialized="table",
           table_format="iceberg",
           submission_method="all_purpose_cluster",
           cluster_id="1993-123451-wimaesen"
       )
       source_df = dbt.source("nyctaxi", "trips")
       return source_df

3. Run dbt run --select <model_name> against a Unity Catalog + all-purpose cluster (DBR 18.2)

Expected behavior

A Python model with table_format='iceberg' and use_managed_iceberg enabled should create a managed Iceberg table, matching the behavior of an equivalent SQL model using table_format='iceberg' and use_managed_iceberg enabled.

{{ config(
    materialized='table',
    table_format='iceberg'
) }}

select * from {{ source('nyctaxi', 'trips') }}

08:07:52  1 of 1 START sql table model my_catalog.my_schema.test_dbt_model_iceberg_sql  [RUN]
08:08:02  1 of 1 OK created sql table model my_catalog.my_schema.test_dbt_model_iceberg_sql [OK in 9.99s]
08:08:02  
08:08:02  Finished running 1 table model in 0 hours 0 minutes and 17.09 seconds (17.09s).
08:08:02  
08:08:02  Completed successfully
08:08:02  
08:08:02  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=1

Screenshots and log output

dbt log excerpt (failing run):

07:53:03 1 of 1 START python table model my_schema.test_dbt_model_iceberg_python .... [RUN]
07:53:03 MATERIALIZING TABLE
07:53:08 Databricks adapter: Job submitted successfully with run_id=2416520466857
07:54:22 Databricks adapter: [Python Model Debug] Result state: FAILED
07:54:22 Databricks adapter: [Python Model Debug] Error trace found:
AnalysisException  Traceback (most recent call last)
File , line 87
     80 writer = (
     81     df.write
     82         .mode("overwrite")
     83         .option("overwriteSchema", "true")
     84         .format("parquet")
     85 )
---> 87 writer.saveAsTable("`my_catalog`.`my_schema`.`test_dbt_model_iceberg_python`")
07:54:23 1 of 1 ERROR creating python table model my_schema.test_dbt_model_iceberg_python [ERROR in 79.41s]

Error message:

Runtime Error in model test_dbt_model_iceberg_python
  Python model failed (run_id: 2416520466857, task_id: 961340160003348)
  [RequestId=e56b984c-6b87-4717-9fb2-08059b28b025 ErrorClass=INVALID_PARAMETER_VALUE.MANAGED_TABLE_FORMAT]
  Only Delta is supported for managed tables. Provided datasource format is PARQUET.
  
  JVM stacktrace:
  com.databricks.sql.managedcatalog.UnityCatalogServiceException
    at com.databricks.sql.managedcatalog.client.ManagedCatalogClientImpl.createTable(ManagedCatalogClientImpl.scala:1718)
    at org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.run(createDataSourceTables.scala:264)
    at org.apache.spark.sql.classic.DataFrameWriter.saveAsTable(DataFrameWriter.scala:633)

System information

The output of dbt --version:

Core:
  - installed: 1.11.12
  - latest:    1.11.12 - Up to date!

Plugins:
  - databricks: 1.12.2 - Up to date!
  - spark:      1.10.3 - Up to date!

The operating system you're using:
macOS Tahoe 26.4.1; dbt Core run locally via CLI against Databricks all-purpose cluster (DBR 18.2

The output of python --version:
Python 3.11.2

Additional context

Python model control case to delta (succeeds) - identical model without table_format and file_format (materializes as Delta):

def model(dbt, session):
    dbt.config(
        materialized="table",
        submission_method="all_purpose_cluster",
        cluster_id="1993-123451-wimaesen"
    )
    source_df = dbt.source("nyctaxi", "trips")
    return source_df

Control python delta model (passing) log:

08:02:11  1 of 1 START python table model my_catalog.my_schema.test_dbt_model_delta_python  [RUN]
08:02:16  Databricks adapter: Execution context 38707997752d401babd0be4647877d01 reached RUNNING state
08:02:27  1 of 1 OK created python table model my_catalog.my_schema.test_dbt_model_delta_python  [OK in 16.70s]
08:02:27  
08:02:27  Finished running 1 table model in 0 hours 0 minutes and 29.94 seconds (29.94s).
08:02:27  
08:02:27  Completed successfully
08:02:27  
08:02:27  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=1

[Divya-Kovvuru-0802]

@sd-db could you please review and let me know if this looks good