Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 30 additions & 16 deletions PYDANTIC_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,8 @@ from overture.schema.system.string import (
NoWhitespaceString,
StrippedString,
)
from overture.schema.core.types import (
ConfidenceScore,
LanguageTag,
)
from overture.schema.core.types import ConfidenceScore
from overture.schema.system.string import LanguageTag

# Numeric primitives (use these instead of int/float)
from overture.schema.system.primitive import (
Expand Down Expand Up @@ -182,7 +180,7 @@ from typing import Literal
from overture.schema.core import OvertureFeature
from overture.schema.core.models import Stacked
from overture.schema.core.names import Named
from overture.schema.system.primitives import float64
from overture.schema.system.primitive import float64

class Building(OvertureFeature[Literal["buildings"], Literal["building"]], Named, Stacked):
# Gets fields from Feature: id, theme, type, geometry, etc.
Expand Down Expand Up @@ -506,18 +504,34 @@ class Building(OvertureFeature):

Add documentation to describe what the enum and its values mean. In Python, you do this with **docstrings** - text enclosed in triple quotes `"""` that describes what something does:

TODO: DocumentedEnum
Use `DocumentedEnum` from `overture.schema.system.doc` when enum members need their own descriptions for code generation and documentation tooling. Each member takes a `(value, description)` tuple:

```python
class VehicleType(str, Enum):
from overture.schema.system.doc import DocumentedEnum

class VehicleType(str, DocumentedEnum):
"""Types of vehicles for transportation."""

CAR = "car" # Standard passenger vehicle
TRUCK = "truck" # Commercial freight vehicle
BICYCLE = "bicycle" # Human-powered two-wheeler
MOTORCYCLE = "motorcycle" # Motorized two-wheeler
CAR = ("car", "Standard passenger vehicle")
TRUCK = ("truck", "Commercial freight vehicle")
BICYCLE = ("bicycle", "Human-powered two-wheeler")
MOTORCYCLE = ("motorcycle", "Motorized two-wheeler")
```

Members without descriptions use the plain value form -- documentation is optional per-member:

```python
class ConnectionState(str, DocumentedEnum):
CONNECTED = "connected"
DISCONNECTED = "disconnected"
QUIESCING = (
"quiescing",
"Gracefully shutting down, rejecting new requests but completing existing ones",
)
```

Use `DocumentedEnum` over plain `str, Enum` when the enum members' semantics aren't obvious from their names and downstream tools (code generators, documentation renderers) need access to member-level descriptions. Use plain `str, Enum` for self-explanatory values.

#### Why str, Enum?

Inheriting from `str, Enum` makes enum values work as both enums and strings, which is useful for JSON serialization and compatibility.
Expand Down Expand Up @@ -560,7 +574,7 @@ class DivisionArea(OvertureFeature[Literal["divisions"], Literal["division_area"
] = None
```

**Available relationship types (see [Relationship](packages/overture-schema-core/src/overture/schema/core/ref.py)):**
**Available relationship types (see [Relationship](packages/overture-schema-system/src/overture/schema/system/ref/ref.py)):**

- **`BELONGS_TO`**: The referencing feature belongs to the referenced feature (division area belongs to division)
- **`CONNECTS_TO`**: The referencing feature connects to the referenced feature (segment connects to connector)
Expand Down Expand Up @@ -627,7 +641,7 @@ class Building(OvertureFeature[Literal["buildings"], Literal["building"]]):

#### Best Practices

**1. Always Use Reference Annotations**
##### Always Use Reference Annotations

Include `Reference` annotations for semantic clarity and documentation:

Expand All @@ -643,7 +657,7 @@ division_id: Annotated[
division_id: Id
```

**2. Choose the Right Pattern**
##### Choose the Right Pattern

- **Simple relationships** → Direct references (foreign keys)
- **Relationships with metadata** → Separate association features
Expand Down Expand Up @@ -938,7 +952,7 @@ Organize code by scope and avoid circular imports:

**Cross-theme shared**: `overture-schema-core` package

- Used by multiple themes (e.g., `LanguageTag`, `CountryCode`, `OvertureFeature`)
- Used by multiple themes (e.g., `OvertureFeature`, `Names`, `Sources`, `Scope`)

**Theme-level shared**: Theme package root (e.g., `overture-schema-transportation-theme/src/overture/schema/transportation/`)

Expand Down Expand Up @@ -1110,7 +1124,7 @@ JSON Schema containers become **mixin classes** in Pydantic that you inherit fro
```python models.py
from typing import Annotated
from pydantic import BaseModel, Field
from overture.schema.model_constraints import no_extra
from overture.schema.system.model_constraint import no_extra_fields
from overture.schema.system.primitive import int8, float64

@no_extra_fields
Expand Down
8 changes: 5 additions & 3 deletions README.pydantic.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,9 +88,11 @@ This workspace contains the following packages:

- **`overture-schema`** - Main entrypoint package that aggregates all types for
convenient usage
- **`overture-schema-core`** - Base classes, geometry models, and common structures
shared across all themes
- **`overture-schema-system`** - Foundational system of primitive types and constraints
- **`overture-schema-core`** - Overture-specific models shared across themes: base
feature class, scoping framework, names, sources, and cartographic hints
- **`overture-schema-system`** - Portable primitive types, constraints, and a
GeoJSON-aware base model for building Pydantic schemas that serialize to
JSON, Parquet, and Spark

### Theme Packages

Expand Down
187 changes: 59 additions & 128 deletions packages/overture-schema-core/README.md
Original file line number Diff line number Diff line change
@@ -1,166 +1,97 @@
# Overture Schema Core

Core Pydantic models and base classes for Overture Maps schemas, providing foundational types, geometry handling, and a comprehensive scoping system for conditional rule application.
Shared models and conventions for building Overture Maps feature types. Defines the base feature class all themes extend, a scoping framework for expressing conditional values (this speed limit applies *here*, *then*, to *these vehicles*), and common structures for names, sources, and cartographic hints.

## Installation

```bash
pip install overture-schema-core
```

## Key Components
## OvertureFeature

- **Base Classes**: Extensible base models for Overture Maps features
- **Geometry Types**: WKB geometry type hints and utilities
- **Common Structures**: Shared models used across all themes
- **Primitive Data Types**: Validated primitive types with multi-target serialization support
- **Scoping System**: Flexible conditional rule application framework

## Enhanced Primitive Types

The enhanced primitive types system provides validated primitive types with automatic
constraint checking and multi-target serialization support. This enables consistent type
definitions that can generate appropriate representations for different targets (Spark,
Parquet, etc.).

### Available Types

Built-in Python primitive types (`str`, `int`, `float`, `bool`, `list`, etc.) are
automatically mapped.

We also provide the following additional types:

#### Integer Types

- **`uint8`**: 8-bit unsigned integer (0-255)
- **`uint16`**: 16-bit unsigned integer (0-65535)
- **`uint32`**: 32-bit unsigned integer (0-4294967295)
- **`int8`**: 8-bit signed integer (-128 to 127)
- **`int32`**: 32-bit signed integer (-2³¹ to 2³¹-1)
- **`int64`**: 64-bit signed integer (-2⁶³ to 2⁶³-1)

#### Floating Point Types

- **`float32`**: 32-bit floating point number
- **`float64`**: 64-bit floating point number

### Basic Usage
Every Overture feature type inherits from `OvertureFeature`, which extends `system.Feature` with the fields present on all Overture data: `id`, `theme`, `type`, `version`, `geometry`, and `sources`.

```python
from pydantic import BaseModel, Field
from overture.schema.core.primitives import (
uint8, uint32, float32
)

class Building(BaseModel):
"""Building feature with specific primitive data types."""

height: float32 | None = Field(
None,
description="Height of building in meters"
)

num_floors: uint8 | None = Field(
None,
description="Number of floors in building"
)

area: uint32 | None = Field(
None,
description="Floor area in square meters"
)
from typing import Literal
from overture.schema.core import OvertureFeature

class Park(OvertureFeature[Literal["places"], Literal["park"]]):
area_hectares: float | None = None
```

### Automatic Validation
## Scoping

Enhanced primitive types automatically validate constraints:
Many Overture values only apply under specific conditions -- a speed limit that holds during rush hour, along a sub-segment, in the forward direction. The `@scoped` decorator adds conditional fields to any Pydantic model:

```python
# Valid values
building = Building(height=45.5, num_floors=12, area=2500)
from pydantic import BaseModel
from overture.schema.core.scoping import Scope, scoped
from overture.schema.system.primitive import float32

# Invalid values raise ValidationError
Building(num_floors=256) # Error: 256 > UInt8 maximum (255)
Building(num_floors=-1) # Error: -1 < UInt8 minimum (0)
@scoped(Scope.GEOMETRIC_RANGE, Scope.TEMPORAL)
class SpeedLimit(BaseModel):
max_speed: float32
```

### Type Safety
This produces a model with `between` (geometric range) and `when.during` (temporal) fields, both optional. The full set of scopes and the fields they inject:

The enhanced primitive types provide strong type safety guarantees at both static and
runtime levels:
| Scope | Field |
|----------------------------|-------------------|
| `Scope.GEOMETRIC_POSITION` | `at` |
| `Scope.GEOMETRIC_RANGE` | `between` |
| `Scope.HEADING` | `when.heading` |
| `Scope.TEMPORAL` | `when.during` |
| `Scope.TRAVEL_MODE` | `when.mode` |
| `Scope.PURPOSE_OF_USE` | `when.using` |
| `Scope.RECOGNIZED_STATUS` | `when.recognized` |
| `Scope.SIDE` | `side` |
| `Scope.VEHICLE` | `when.vehicle` |

**Static Type Checking**: mypy can distinguish between different primitive types,
*preventing common errors:
Scopes are optional by default. Make them mandatory via `required`:

```python
from overture.schema.core.primitives import uint8, uint32
@scoped(Scope.TEMPORAL, required=(Scope.GEOMETRIC_POSITION, Scope.HEADING))
class TrafficSignal(BaseModel):
signal_type: str
```

def process_floor_count(floors: uint8) -> str:
return f"Building has {floors} floors"
## Names

def process_area(area: uint32) -> str:
return f"Area: {area} sq meters"
Multilingual naming with support for common names, name rules (official, alternate, short variants), and scoping by geometric range, side, or political perspective. Mix `Named` into a feature type to give it a `names` field:

# Type checker prevents mixing incompatible types
floors: uint8 = 12
area: uint32 = 2500
```python
from typing import Literal
from overture.schema.core import OvertureFeature
from overture.schema.core.names import Named

process_floor_count(area) # mypy error: Expected UInt8, got UInt32
process_area(floors) # mypy error: Expected UInt32, got UInt8
class Lake(OvertureFeature[Literal["base"], Literal["water"]], Named):
pass # inherits names: Names | None from Named
```

Name rules support geometric range and side scoping for cases like a street whose name changes partway along or differs on each side. `NameRule` variants: `common`, `official`, `alternate`, `short`.

### Examples
## Sources

#### Temporal Speed Limit

```yaml
speed_limits:
- between: [0, 1]
max_speed: {value: 30, unit: km/h}
when:
during: "Mo-Fr 07:00-09:00,17:00-19:00" # Rush hours only
```
Source attribution tracking. Each `SourceItem` identifies which dataset a feature or property came from, with optional license, record ID, update time, and confidence score. Source items support geometric range scoping for per-segment attribution.

#### Vehicle-Specific Access Restriction

```yaml
access_restrictions:
- between: [0.2, 0.8]
access_type: denied
when:
vehicle:
- dimension: weight
comparison: greater_than
value: 7.5
unit: t
```

#### Multi-Dimensional Scoping

```yaml
access_restrictions:
- between: [0, 1]
access_type: denied
when:
mode: [bus]
during: "Mo-Fr 15:00-18:00"
heading: forward
using: [to_deliver]
```python
from overture.schema.core.sources import SourceItem

sources = [
SourceItem(property="", dataset="OpenStreetMap"),
SourceItem(property="/geometry", dataset="Microsoft ML Buildings"),
# first 30% of the segment's geometry came from a different source
SourceItem(property="/geometry", dataset="County GIS", between=[0, 0.3]),
]
```

### Design Principles

1. **Composability**: Mix-in design allows combining only needed scoping dimensions
2. **Reusability**: Base scope classes work across all rule types and themes
3. **Extensibility**: Easy to add new scoping dimensions or modify existing ones
4. **Type Safety**: Full Pydantic validation for all scoping conditions
5. **Linear Reference Integration**: Seamless integration with geometric positioning
## Cartography

### Rule Complexity Patterns
Rendering hints for map-making: `prominence` (1--100 significance scale), `min_zoom`/`max_zoom` (tile zoom bounds), and `sort_key` (draw order). Mix `CartographicallyHinted` into a model to add a `cartography` field.

- **Simple Rules** (flags, dimensions): Geometric scoping only
- **Complex Rules** (speed limits, access): Geometric + conditional scoping
- **Transition Rules**: Full scoping including directional constraints
## Also Included

This scoping system provides the foundation for precise, flexible rule specification across all Overture Maps transportation features.
- **Types** -- domain-specific aliases built on system primitives: `ConfidenceScore` (0.0--1.0), `Level` (z-order), `FeatureVersion`.
- **Units** -- measurement enumerations: `SpeedUnit`, `LengthUnit`, `WeightUnit`.
- **Discovery** -- entry-point-based model registry. Theme packages register models via `overture.models` entry points; `discover_models()` resolves them at runtime.
Loading
Loading