Skip to content

Commit 8fa0278

Browse files
nathanielcnathanielc
authored
docs: add pipeline docs (#691)
1 parent a5ed3a7 commit 8fa0278

2 files changed

Lines changed: 276 additions & 50 deletions

File tree

‎pipeline/DESIGN.md‎

Lines changed: 45 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,6 @@ graph LR;
1515
streams;
1616
time_conclusions;
1717
conclusion_events;
18-
model_schemas;
1918
event_states;
2019
stream_tips;
2120
stream_states;
@@ -24,7 +23,6 @@ graph LR;
2423
streams --> conclusion_events;
2524
time_conclusions --> conclusion_events;
2625
conclusion_events --> event_states;
27-
model_schemas --> event_states;
2826
event_states --> stream_tips;
2927
stream_tips --> stream_states;
3028
```
@@ -83,17 +81,17 @@ The conclusion_events table contains a row for each event in a stream and repres
8381

8482
#### Schema
8583

86-
| Column | Type | Description |
87-
| ------ | ---- | ----------- |
88-
| index | u64 | Order of this event. Index is always greater than the index of any previous event in the stream |
89-
| stream_cid | bytes | Cid of the stream |
90-
| stream_type | u8 | Type of the stream, see [stream type values](#stream-types) |
91-
| controller | string | Controller of the stream |
92-
| dimensions | map(string,bytes) | Set of key values dimension pairs of the stream |
93-
| event_cid | bytes | Cid of the event |
94-
| event_type | u8 | Type of the event, see [event type values](#event-types) |
95-
| data | bytes | The event payload, content is stream type specific |
96-
| previous | list(bytes) | Ordered list of CID previous to this event. Meaning of the order is stream type dependent |
84+
| Column | Type | Description |
85+
| ------ | ---- | ----------- |
86+
| conclusion_event_order | u64 | Order of this event. Value is always greater than any previous event in the same stream |
87+
| stream_cid | bytes | Cid of the stream |
88+
| stream_type | u8 | Type of the stream, see [stream type values](#stream-types) |
89+
| controller | string | Controller of the stream |
90+
| dimensions | map(string,bytes) | Set of key values dimension pairs of the stream |
91+
| event_cid | bytes | Cid of the event |
92+
| event_type | u8 | Type of the event, see [event type values](#event-types) |
93+
| data | bytes | The event payload, content is stream type specific |
94+
| previous | list(bytes) | Ordered list of CID previous to this event. Meaning of the order is stream type dependent |
9795

9896

9997
#### Transformation
@@ -108,13 +106,6 @@ Conclusions include:
108106

109107
This table joins the raw_events, time_conclusions, and streams tables in order to make the conclusions about the raw events.
110108

111-
### model_schemas
112-
113-
The model_schemas table contains a row for each model known to the node and contains the complete resolved schema of the model.
114-
115-
TBD how this table is populated and its schema.
116-
This table may be able to be combined with the streams table. Should we?
117-
118109
### event_states
119110

120111
The event_states table contains a row for each event in a stream and the state of the document at that point in the stream.
@@ -126,16 +117,19 @@ The event_states table contains a row for each event in a stream and the state o
126117

127118
#### Schema
128119

129-
| Column | Type | Description |
130-
| ------ | ---- | ----------- |
131-
| index | u64 | Order of this event. Index is always greater than the index of any previous event in the stream |
132-
| stream_cid | bytes | Cid of the stream |
133-
| stream_type | u8 | Type of the stream, see [stream type values](#stream-types) |
134-
| controller | string | Controller of the stream |
135-
| dimensions | map(string,bytes) | Set of key values dimension pairs of the stream |
136-
| event_cid | bytes | Cid of the event |
137-
| event_type | u8 | Type of the event, see [event type values](#event-types) |
138-
| data | bytes | The event payload, content is stream type specific |
120+
| Column | Type | Description |
121+
| ------ | ---- | ----------- |
122+
| conclusion_event_order | u64 | Order of this event from the conclusion_events table. |
123+
| event_state_order | u64 | Order of this event state. Value is always greater than any previous event in the same stream and any dependent streams (i.e. model streams) |
124+
| stream_cid | bytes | Cid of the stream |
125+
| stream_type | u8 | Type of the stream, see [stream type values](#stream-types) |
126+
| controller | string | Controller of the stream |
127+
| dimensions | map(string,bytes) | Set of key values dimension pairs of the stream |
128+
| event_cid | bytes | Cid of the event |
129+
| event_type | u8 | Type of the event, see [event type values](#event-types) |
130+
| event_height | i32 | Number of events between this event and the init event of the stream. |
131+
| data | bytes | The event payload, content is stream type specific |
132+
| validation_errors | list(string) | List of validation errors, will always be an empty list (not null) when the stream state is valid. |
139133

140134
#### Transformation
141135

@@ -153,16 +147,15 @@ The tip represents the most recent event in each branch of the stream, where _re
153147

154148
#### Schema
155149

156-
| Column | Type | Description |
157-
| ------ | ---- | ----------- |
158-
| index | u64 | Order of this event. Index is always greater than the index of any previous event in the stream |
159-
| stream_cid | bytes | Cid of the stream |
160-
| stream_type | u8 | Type of the stream, see [stream type values](#stream-types) |
161-
| controller | string | Controller of the stream |
162-
| dimensions | map(string,bytes) | Set of key values dimension pairs of the stream |
163-
| event_cid | bytes | Cid of the event |
164-
| event_type | u8 | Type of the event, see [event type values](#event-types) |
165-
| data | bytes | The event payload, content is stream type specific |
150+
| Column | Type | Description |
151+
| ------ | ---- | ----------- |
152+
| event_state_order | u64 | Order of this event from the event_states table. |
153+
| stream_tip_order | u64 | Order of this stream tips. Value is always greater than any previous stream tips set. |
154+
| stream_cid | bytes | Cid of the stream |
155+
| stream_type | u8 | Type of the stream, see [stream type values](#stream-types) |
156+
| controller | string | Controller of the stream |
157+
| dimensions | map(string,bytes) | Set of key values dimension pairs of the stream |
158+
| tips | list(bytes) | Complete list of all tips for the stream. |
166159

167160
#### Transformation
168161

@@ -179,16 +172,18 @@ The stream_states table contains a row for each stream representing the canonica
179172

180173
#### Schema
181174

182-
| Column | Type | Description |
183-
| ------ | ---- | ----------- |
184-
| index | u64 | Order of this event. Index is always greater than the index of any previous event in the stream |
185-
| stream_cid | bytes | Cid of the stream |
186-
| stream_type | u8 | Type of the stream, see [stream type values](#stream-types) |
187-
| controller | string | Controller of the stream |
188-
| dimensions | map(string,bytes) | Set of key values dimension pairs of the stream |
189-
| event_cid | bytes | Cid of the event |
190-
| event_type | u8 | Type of the event, see [event type values](#event-types) |
191-
| data | bytes | The event payload, content is stream type specific |
175+
| Column | Type | Description |
176+
| ------ | ---- | ----------- |
177+
| stream_tip_order | u64 | Order of this event from the stream_tips table. |
178+
| stream_state_order | u64 | Order of this stream state. Value is always greater than any previous stream state. |
179+
| stream_cid | bytes | Cid of the stream |
180+
| stream_type | u8 | Type of the stream, see [stream type values](#stream-types) |
181+
| controller | string | Controller of the stream |
182+
| dimensions | map(string,bytes) | Set of key values dimension pairs of the stream |
183+
| event_cid | bytes | Cid of the event |
184+
| event_type | u8 | Type of the event, see [event type values](#event-types) |
185+
| event_height | i32 | Number of events between this event and the init event of the stream. |
186+
| data | bytes | The event payload, content is stream type specific |
192187

193188
#### Transformation
194189

0 commit comments

Comments
 (0)