|
| 1 | +--- |
| 2 | +sidebar_position: 4 |
| 3 | +--- |
| 4 | + |
| 5 | +# Web3 Adapter |
| 6 | + |
| 7 | +The Web3 Adapter is the bridge between a platform's local database and eVault. It enables "write once, sync everywhere": when data changes locally, the adapter converts it to the global schema and stores it in the owner's eVault; when awareness protocol packets arrive from other platforms, the adapter converts them back to the local schema and applies them locally. |
| 8 | + |
| 9 | +Web3 Adapter is only needed when you have a database which you want to |
| 10 | +automatically sync with eVaults, if your application is stateless or the application only writes to the eVault directly then you don't need a Web3 Adapter. |
| 11 | + |
| 12 | +## Overview |
| 13 | + |
| 14 | +Platforms keep their own schemas and databases. To participate in W3DS, they need a component that: |
| 15 | + |
| 16 | +- **Outbound**: Detects local changes, maps local data to the global ontology, resolves the owner's and/or target eVault (via the user's [eName](/docs/W3DS%20Basics/W3ID) / Registry), and writes to the eVault (GraphQL). |
| 17 | +- **Inbound**: Receives awareness protocol packets at `POST /api/webhook`, maps global data to the local schema, and creates or updates local entities while maintaining global-ID-to-local-ID mappings. See [Awareness Protocol](/docs/W3DS%20Protocol/Awareness-Protocol) for more details. |
| 18 | + |
| 19 | +Please Note: That neither inbound or outbound are immediate, or have any transactional guarantees. |
| 20 | + |
| 21 | +The Web3 Adapter implements this bridge. Understanding its core ideas helps you design better ontologies, mappings, and consistency strategies. |
| 22 | + |
| 23 | +Please note that the web3 adapter may not come pre-assembled with hooks for all databases. |
| 24 | + |
| 25 | +### Key Features |
| 26 | + |
| 27 | +- **Bidirectional mapping**: Local schema ↔ global ontology via JSON mapping configs. |
| 28 | +- **ID mapping**: Stores pairs of (localId, globalId) so the same entity is recognized across sync and webhooks. When using our implementation of the Web3 Adapter, you don't need to worry about ID Mapping yourself, the adapter already handles it. |
| 29 | +- **eVault client**: Resolves [eNames](/docs/W3DS%20Basics/W3ID) via the Registry, obtains platform tokens, and calls eVault GraphQL (store/update) with retries and health checks. |
| 30 | +- **Change handling**: `handleChange` is the main entry for outbound sync; webhook handlers use `fromGlobal` for inbound. |
| 31 | + |
| 32 | +## Theory: Core Ideas |
| 33 | + |
| 34 | +### 1. Universal ontology as the common language |
| 35 | + |
| 36 | +All platforms agree on a small set of global schemas (e.g. User, Post, Group) identified by W3IDs. Each platform maps its local tables to these schemas. The ontology is the contract: if you store data in that shape in an eVault, other platforms can interpret it via their own mappings. Better ontology design (versioning, optional fields, extensibility) leads to easier evolution and fewer breaking changes. |
| 37 | + |
| 38 | +### 2. Per-entity owner eVault (W3ID) |
| 39 | + |
| 40 | +Every piece of data has an "owner" — the [eName](/docs/W3DS%20Basics/W3ID) of the eVault where it lives. The adapter uses `ownerEnamePath` in the mapping to determine the owner from the local entity (e.g. a direct field `ename` or a nested path like `user(createdBy.ename)`). Data is always written to that owner's eVault. This keeps ownership explicit and supports ACLs and multi-tenant resolution. |
| 41 | + |
| 42 | +### 3. Bidirectional mapping and ID mapping |
| 43 | + |
| 44 | +- **Field mapping**: `localToUniversalMap` defines how each local field maps to a global field (including relations and special functions like `__date`, `__calc`). The same map is used in both directions: `toGlobal` for outbound, `fromGlobal` for inbound. |
| 45 | +- **ID mapping**: A separate store (e.g. SQLite `MappingDatabase`) holds `(globalId, localId)`. When syncing out, after a successful `storeMetaEnvelope` the adapter stores the new global ID against the local ID. When a webhook arrives, the adapter looks up the global ID to decide whether to create or update the local entity and then stores or updates the mapping. Without this, the same logical entity could be duplicated or never linked across platforms. When consuming our TypeScript implementation of the Web3 Adapter, this is already taken care of. |
| 46 | + |
| 47 | +### 4. Change detection on the platform side |
| 48 | + |
| 49 | +The adapter does not poll the database. The platform must detect changes (e.g. via ORM hooks, DB triggers, or application events) and call `handleChange({ data, tableName, participants })`. So the core idea is: the platform owns the trigger; the adapter owns the translation and eVault write. Better detection (e.g. transactional outbox, CDC) can improve consistency and avoid missed or duplicate syncs. |
| 50 | + |
| 51 | +### 5. Webhooks (Awareness Protocol) propagate to other platforms |
| 52 | + |
| 53 | +After data is stored or updated in an eVault, the [Awareness Protocol](/docs/W3DS%20Protocol/Awareness-Protocol) delivers webhooks to all other registered platforms. Those platforms use the same adapter's `fromGlobal` and ID mapping to apply the change locally. So the full loop is: Platform A → adapter → eVault → Awareness Protocol → Platform B's webhook → adapter → Platform B's DB. |
| 54 | + |
| 55 | +### Design Limitations |
| 56 | + |
| 57 | +Current implementation has the following known limitations, which we aim to fix with subsequent versions: |
| 58 | + |
| 59 | +- **Ontology design**: Clear schema versioning, optional vs required fields, and conventions for references (e.g. eNames vs local IDs in payloads). |
| 60 | +- **Mapping expressiveness**: Richer `ownerEnamePath` (e.g. fallbacks), relation resolution, and handling of arrays and nested structures. |
| 61 | +- **Conflict handling**: The current implementation is last-write-wins via eVault updates; you could add version fields, conflict resolution, or merge strategies. |
| 62 | +- **Eventual consistency**: The system does not guarantee ordering or at-least-once delivery of webhooks; you could add idempotency keys, retries, or acknowledgments. |
| 63 | + |
| 64 | +## Architecture |
| 65 | + |
| 66 | +```mermaid |
| 67 | +graph TB |
| 68 | + subgraph Platform["Platform"] |
| 69 | + App[Application] |
| 70 | + DB[(Local DB)] |
| 71 | + WebhookHandler[Webhook Handler<br/>POST /api/webhook] |
| 72 | + end |
| 73 | +
|
| 74 | + subgraph Web3Adapter["Web3 Adapter"] |
| 75 | + Adapter[Web3Adapter<br/>handleChange / fromGlobal] |
| 76 | + Mapper[Mapper<br/>toGlobal / fromGlobal] |
| 77 | + MappingDB[(MappingDatabase<br/>localId ↔ globalId)] |
| 78 | + EVaultClient[EVaultClient] |
| 79 | + end |
| 80 | +
|
| 81 | + subgraph External["External"] |
| 82 | + Registry[Registry<br/>resolve eName] |
| 83 | + EVault[eVault Core<br/>GraphQL] |
| 84 | + end |
| 85 | +
|
| 86 | + App -->|Entity change| Adapter |
| 87 | + Adapter --> Mapper |
| 88 | + Adapter --> MappingDB |
| 89 | + Adapter --> EVaultClient |
| 90 | + EVaultClient -->|GET /resolve| Registry |
| 91 | + EVaultClient -->|storeMetaEnvelope<br/>updateMetaEnvelopeById| EVault |
| 92 | + EVault -->|Awareness Protocol| WebhookHandler |
| 93 | + WebhookHandler --> Adapter |
| 94 | + Adapter --> DB |
| 95 | +``` |
| 96 | + |
| 97 | +## Implementation |
| 98 | + |
| 99 | +### Components |
| 100 | + |
| 101 | +- **Web3Adapter** (`infrastructure/web3-adapter`): Main class. Loads mapping configs from JSON files, owns `MappingDatabase` and `EVaultClient`, and exposes `handleChange` and `fromGlobal`. Config includes `schemasPath`, `dbPath`, `registryUrl`, `platform` (name used for platform token). |
| 102 | +- **EVaultClient** (`src/evault/evault.ts`): Resolves eName to GraphQL endpoint via Registry `GET /resolve?w3id=`, obtains platform token via `POST /platforms/certification`, caches clients per eName, and performs health checks (`HEAD /whois`). Uses retries and timeouts for store/update/fetch. |
| 103 | +- **Mapper** (`src/mapper/mapper.ts`): `toGlobal({ data, mapping, mappingStore })` and `fromGlobal(...)` using `IMapping` and `MappingDatabase` for resolving relation IDs. |
| 104 | +- **MappingDatabase** (`src/db/mapping.db.ts`): SQLite store for `(local_id, global_id)`. Methods: `storeMapping`, `getLocalId(globalId)`, `getGlobalId(localId)`. |
| 105 | + |
| 106 | +### Flow: Outbound (local change → eVault) |
| 107 | + |
| 108 | +1. Platform detects a change and calls `adapter.handleChange({ data, tableName, participants })`. |
| 109 | +2. Adapter loads the mapping for `tableName`; if missing or `readOnly`, returns. |
| 110 | +3. If a global ID already exists for this local ID, adapter calls `toGlobal`, then `evaultClient.updateMetaEnvelopeById(existingGlobalId, { ... })` (fire-and-forget). Optionally uses `lockedIds` to avoid re-entrant sync from webhooks. |
| 111 | +4. If no global ID yet, adapter calls `toGlobal` to get owner eName and global-shaped payload. If no owner, returns. Otherwise calls `evaultClient.storeMetaEnvelope({ id: null, w3id, data, schemaId })`, then `mappingDb.storeMapping({ localId, globalId })`. If `participants` includes other eNames, adapter may call `storeReference(ownerEvault/globalId, otherEvault)` for each. |
| 112 | +5. [Awareness Protocol](/docs/W3DS%20Protocol/Awareness-Protocol) later delivers webhooks to other platforms; the originating platform is excluded. |
| 113 | + |
| 114 | +### EVaultClient details |
| 115 | + |
| 116 | +- **Resolution**: `GET /resolve?w3id=@...` → response `{ uri }` → GraphQL endpoint is `uri + "/graphql"`. |
| 117 | +- **Caching**: One GraphQL client per eName; if health check fails (e.g. `HEAD /whois`), client is evicted and re-resolved next time. |
| 118 | +- **Retries**: Configurable for store/update/fetch; typically no retry on 4xx. |
| 119 | + |
| 120 | +### Mapping configuration (IMapping) |
| 121 | + |
| 122 | +Mapping configs define how local fields map to the global ontology. For the full syntax (direct fields, relations, arrays, `__date`, `__calc`, owner path), see the **Web3 Adapter Mapping Rules** in the repository at `infrastructure/web3-adapter/MAPPING_RULES.md`. |
| 123 | + |
| 124 | +Each mapping is a JSON file with: |
| 125 | + |
| 126 | +- **tableName**: Local table (or entity) name. |
| 127 | +- **schemaId**: Global ontology UUID. |
| 128 | +- **ownerEnamePath**: Path to the owner eName in the local entity (e.g. `"ename"` or `"users(createdBy.ename)"`). Supports fallbacks with `||`. |
| 129 | +- **localToUniversalMap**: Object mapping local field names to global field names or expressions (e.g. `"createdAt": "__date(createdAt)"`, relation syntax `"tableName(path),globalAlias"`). |
| 130 | +- **readOnly** (optional): If true, `handleChange` does not sync this table to the eVault. |
| 131 | + |
| 132 | +### Receiving data (inbound) |
| 133 | + |
| 134 | +When a platform receives an awareness protocol packet at `POST /api/webhook`: |
| 135 | + |
| 136 | +1. Parse body: `id`, `w3id`, `schemaId`, `data`. |
| 137 | +2. Find the mapping whose `schemaId` matches (or map by schema UUID to table name). |
| 138 | +3. Call `adapter.fromGlobal({ data: body.data, mapping })` to get local-shaped data. |
| 139 | +4. Call `mappingDb.getLocalId(body.id)`; if found, update the existing local entity; otherwise create and then `mappingDb.storeMapping({ localId: newEntity.id, globalId: body.id })`. |
| 140 | +5. Return 200. |
| 141 | + |
| 142 | +See the [Webhook Controller Guide](/docs/Post%20Platform%20Guide/webhook-controller) for a full implementation example and the [Awareness Protocol](/docs/W3DS%20Protocol/Awareness-Protocol) for the packet format and delivery mechanism. |
| 143 | + |
| 144 | +## Sequence: Platform → Adapter → eVault |
| 145 | + |
| 146 | +```mermaid |
| 147 | +sequenceDiagram |
| 148 | + participant App as Platform App |
| 149 | + participant Adapter as Web3 Adapter |
| 150 | + participant MappingDB as MappingDatabase |
| 151 | + participant EVaultClient as EVaultClient |
| 152 | + participant Registry as Registry |
| 153 | + participant EVault as eVault Core |
| 154 | +
|
| 155 | + App->>Adapter: handleChange(data, tableName) |
| 156 | + Adapter->>MappingDB: getGlobalId(localId) |
| 157 | + alt Existing global ID |
| 158 | + Adapter->>Adapter: toGlobal(data, mapping) |
| 159 | + Adapter->>EVaultClient: updateMetaEnvelopeById(globalId, ...) |
| 160 | + EVaultClient->>Registry: resolve eName |
| 161 | + EVaultClient->>EVault: GraphQL updateMetaEnvelopeById |
| 162 | + else New entity |
| 163 | + Adapter->>Adapter: toGlobal(data, mapping) |
| 164 | + Adapter->>EVaultClient: storeMetaEnvelope(w3id, data, schemaId) |
| 165 | + EVaultClient->>Registry: resolve eName |
| 166 | + EVaultClient->>EVault: GraphQL storeMetaEnvelope |
| 167 | + EVault-->>EVaultClient: metaEnvelope.id |
| 168 | + EVaultClient-->>Adapter: globalId |
| 169 | + Adapter->>MappingDB: storeMapping(localId, globalId) |
| 170 | + end |
| 171 | +``` |
| 172 | + |
| 173 | +## Limitations & Planned Extensions |
| 174 | + |
| 175 | +- **Ontology versioning**: Today `schemaId` is a single UUID, there are plans for adding a `schemaVersion` key to allow for schema versioning. |
| 176 | +- **Conflict resolution**: Add version or timestamp fields and resolve conflicts in the adapter or in a separate service. |
| 177 | +- **Idempotency**: Use idempotency keys in payloads or in the mapping DB to make webhook handling and outbound sync idempotent. |
| 178 | +- **Transactional outbox**: Have the platform write changes to an outbox table and have a worker call `handleChange` so that sync is tied to the same transaction as the local write. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?v=4&size=40){kind=avatar}
0 commit comments