Skip to content

Commit 68d0215

Browse files
committed
add docs
1 parent a3819c6 commit 68d0215

3 files changed

Lines changed: 39 additions & 2 deletions

File tree

docs/guides/configuration.md

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -538,6 +538,42 @@ sqlmesh_md5__d3b07384d113edec49eaa6238ad5ff00__dev
538538

539539
This has a downside that now it's much more difficult to determine which table corresponds to which model by just looking at the database with a SQL client. However, the table names have a predictable length so there are no longer any surprises with identfiers exceeding the max length at the physical layer.
540540

541+
#### Virtual Data Environment modes
542+
543+
By default, Virtual Data Environments (VDE) are applied across both development and production environments. This allows SQLMesh to reuse physical tables when appropriate, even when promoting from development to production.
544+
545+
However, users may sometimes prefer their production environment to be non-virtual. The non-exhaustive list of reasons may include:
546+
547+
- Integration with third-party tools and platforms, such as data catalogs, may not work well with the virtual view layer that SQLMesh imposes by default
548+
- A desire to rely on time travel features provided by cloud data warehouses such as BigQuery, Snowflake, and Databricks
549+
550+
To mitigate this, SQLMesh offers an alternative 'dev-only' mode for using VDE. It can be enabled in the project configuration like so:
551+
552+
=== "YAML"
553+
554+
```yaml linenums="1"
555+
virtual_environment_mode: dev_only
556+
```
557+
558+
=== "Python"
559+
560+
```python linenums="1"
561+
from sqlmesh.core.config import Config
562+
563+
config = Config(
564+
virtual_environment_mode="dev_only",
565+
)
566+
```
567+
568+
As the name suggests, 'dev-only' mode means that VDE is applied only in development environments, while in production, model tables and views are updated directly, bypassing the virtual layer. This also means that physical tables in production will be created using the original, unversioned model names. Users will still benefit from VDE and data reuse across development environments.
569+
570+
Please note that enabling this mode means that all data inserted in development environments is used only for [preview](../concepts/plans.md#data-preview-for-forward-only-changes) and will **not** be reused in production.
571+
572+
573+
!!! warning
574+
Switching the mode for an existing project will result in a complete rebuild of all models in the project. Refer to the [Table Migration Guide](./table_migration.md) to migrate existing tables without rebuilding them from scratch.
575+
576+
541577
#### Environment view catalogs
542578

543579
By default, SQLMesh creates an environment view in the same [catalog](../concepts/glossary.md#catalog) as the physical table the view points to. The physical table's catalog is determined by either the catalog specified in the model name or the default catalog defined in the connection.

docs/reference/configuration.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ Configuration options for how SQLMesh manages environment creation and promotion
4646
| `environment_suffix_target` | Whether SQLMesh views should append their environment name to the `schema`, `table` or `catalog` - [additional details](../guides/configuration.md#view-schema-override). (Default: `schema`) | string | N |
4747
| `gateway_managed_virtual_layer` | Whether SQLMesh views of the virtual layer will be created by the default gateway or model specified gateways - [additional details](../guides/multi_engine.md#gateway-managed-virtual-layer). (Default: False) | boolean | N |
4848
| `environment_catalog_mapping` | A mapping from regular expressions to catalog names. The catalog name is used to determine the target catalog for a given environment. | dict[string, string] | N |
49+
| `virtual_environment_mode` | Determines the Virtual Data Environment (VDE) mode. If set to `full`, VDE is used in both production and development environments. The `dev_only` option enables VDE only in development environments, while in production, no virtual layer is used and models are materialized directly using their original names (i.e., no versioned physical tables). (Default: `full`) | string | N |
4950

5051
### Models
5152

sqlmesh/core/config/common.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,8 +52,8 @@ def __repr__(self) -> str:
5252
class VirtualEnvironmentMode(str, Enum):
5353
"""Mode for virtual environment behavior.
5454
55-
FULL: Use full virtual environment functionality with hashed table names and virtual layer updates.
56-
DEV_ONLY: Bypass virtual environments in production, using simple table names without hashes.
55+
FULL: Use full virtual environment functionality with versioned table names and virtual layer updates.
56+
DEV_ONLY: Bypass virtual environments in production, using original unversioned model names.
5757
"""
5858

5959
FULL = "full"

0 commit comments

Comments
 (0)