Commit 5b46b9d
committed
feat: refactor vector storage configuration and implement LanceDB
- Removed the Qdrant vector configuration file (`qdrant.ts`) and its associated logic.
- Introduced a new LanceDB vector configuration file (`lance.ts`) to support vector storage and similarity search using LanceDB.
- Implemented comprehensive settings for LanceDB, including memory management, embedding generation, and advanced filtering capabilities.
- Added functions for generating embeddings, validating filters, and performing storage operations with detailed logging.
- Created new JSON files for crawler statistics and session pool state to track request metrics and session management.
- Added a request queue JSON file to manage requests with unique identifiers and metadata.
chore: update storage and request queue structures
- Created `SDK_CRAWLER_STATISTICS_0.json` to log crawler performance metrics.
- Created `SDK_SESSION_POOL_STATE.json` to manage session states and usability.
- Created a request queue JSON file to handle requests with relevant metadata and error tracking.1 parent 6091646 commit 5b46b9d
13 files changed
Lines changed: 498 additions & 736 deletions
File tree
- .github/agents
- src/mastra
- agents
- config
- vector
- public/storage
- key_value_stores/default
- request_queues/default
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
| 69 | + | |
69 | 70 | | |
70 | 71 | | |
71 | 72 | | |
| |||
127 | 128 | | |
128 | 129 | | |
129 | 130 | | |
130 | | - | |
| 131 | + | |
131 | 132 | | |
132 | 133 | | |
133 | 134 | | |
| |||
141 | 142 | | |
142 | 143 | | |
143 | 144 | | |
144 | | - | |
| 145 | + | |
145 | 146 | | |
146 | 147 | | |
147 | 148 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
49 | | - | |
| 49 | + | |
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
58 | | - | |
59 | 58 | | |
60 | 59 | | |
61 | 60 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
| 8 | + | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
45 | 51 | | |
46 | 52 | | |
47 | 53 | | |
| |||
70 | 76 | | |
71 | 77 | | |
72 | 78 | | |
73 | | - | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
74 | 86 | | |
75 | 87 | | |
76 | 88 | | |
| |||
104 | 116 | | |
105 | 117 | | |
106 | 118 | | |
107 | | - | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
108 | 126 | | |
109 | 127 | | |
110 | 128 | | |
| |||
140 | 158 | | |
141 | 159 | | |
142 | 160 | | |
143 | | - | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
144 | 168 | | |
145 | 169 | | |
146 | 170 | | |
| |||
160 | 184 | | |
161 | 185 | | |
162 | 186 | | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | 187 | | |
170 | 188 | | |
171 | 189 | | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
176 | 190 | | |
177 | 191 | | |
178 | 192 | | |
179 | 193 | | |
180 | 194 | | |
181 | 195 | | |
182 | 196 | | |
183 | | - | |
184 | | - | |
| 197 | + | |
| 198 | + | |
185 | 199 | | |
186 | 200 | | |
187 | 201 | | |
188 | | - | |
| 202 | + | |
189 | 203 | | |
190 | 204 | | |
191 | 205 | | |
192 | 206 | | |
193 | 207 | | |
194 | 208 | | |
195 | 209 | | |
196 | | - | |
| 210 | + | |
197 | 211 | | |
198 | 212 | | |
199 | 213 | | |
| |||
208 | 222 | | |
209 | 223 | | |
210 | 224 | | |
211 | | - | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
212 | 232 | | |
0 commit comments