You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This has a downside that now it's much more difficult to determine which table corresponds to which model by just looking at the database with a SQL client. However, the table names have a predictable length so there are no longer any surprises with identfiers exceeding the max length at the physical layer.
540
540
541
+
#### Virtual Data Environment modes
542
+
543
+
By default, Virtual Data Environments (VDE) are applied across both development and production environments. This allows SQLMesh to reuse physical tables when appropriate, even when promoting from development to production.
544
+
545
+
However, users may sometimes prefer their production environment to be non-virtual. The non-exhaustive list of reasons may include:
546
+
547
+
- Integration with third-party tools and platforms, such as data catalogs, may not work well with the virtual view layer that SQLMesh imposes by default
548
+
- A desire to rely on time travel features provided by cloud data warehouses such as BigQuery, Snowflake, and Databricks
549
+
550
+
To mitigate this, SQLMesh offers an alternative 'dev-only' mode for using VDE. It can be enabled in the project configuration like so:
551
+
552
+
=== "YAML"
553
+
554
+
```yaml linenums="1"
555
+
virtual_environment_mode: dev_only
556
+
```
557
+
558
+
=== "Python"
559
+
560
+
```python linenums="1"
561
+
from sqlmesh.core.config import Config
562
+
563
+
config = Config(
564
+
virtual_environment_mode="dev_only",
565
+
)
566
+
```
567
+
568
+
As the name suggests, 'dev-only' mode means that VDE is applied only in development environments, while in production, model tables and views are updated directly, bypassing the virtual layer. This also means that physical tables in production will be created using the original, unversioned model names. Users will still benefit from VDE and data reuse across development environments.
569
+
570
+
Please note that enabling this mode means that all data inserted in development environments is used only for [preview](../concepts/plans.md#data-preview-for-forward-only-changes) and will **not** be reused in production.
571
+
572
+
573
+
!!! warning
574
+
Switching the mode for an existing project will result in a complete rebuild of all models in the project. Refer to the [Table Migration Guide](./table_migration.md) to migrate existing tables without rebuilding them from scratch.
575
+
576
+
541
577
#### Environment view catalogs
542
578
543
579
By default, SQLMesh creates an environment view in the same [catalog](../concepts/glossary.md#catalog) as the physical table the view points to. The physical table's catalog is determined by either the catalog specified in the model name or the default catalog defined in the connection.
Copy file name to clipboardExpand all lines: docs/reference/configuration.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -46,6 +46,7 @@ Configuration options for how SQLMesh manages environment creation and promotion
46
46
|`environment_suffix_target`| Whether SQLMesh views should append their environment name to the `schema`, `table` or `catalog` - [additional details](../guides/configuration.md#view-schema-override). (Default: `schema`) | string | N |
47
47
|`gateway_managed_virtual_layer`| Whether SQLMesh views of the virtual layer will be created by the default gateway or model specified gateways - [additional details](../guides/multi_engine.md#gateway-managed-virtual-layer). (Default: False) | boolean | N |
48
48
|`environment_catalog_mapping`| A mapping from regular expressions to catalog names. The catalog name is used to determine the target catalog for a given environment. | dict[string, string]| N |
49
+
|`virtual_environment_mode`| Determines the Virtual Data Environment (VDE) mode. If set to `full`, VDE is used in both production and development environments. The `dev_only` option enables VDE only in development environments, while in production, no virtual layer is used and models are materialized directly using their original names (i.e., no versioned physical tables). (Default: `full`) | string | N |
0 commit comments