Skip to content

Commit 0c03c66

Browse files
danlu1danlu1
committed
simplify the diagram
1 parent 0313974 commit 0c03c66

1 file changed

Lines changed: 75 additions & 140 deletions

File tree

‎docs/explanations/storage_location_architecture.md‎

Lines changed: 75 additions & 140 deletions
Original file line numberDiff line numberDiff line change
@@ -92,11 +92,12 @@ classDiagram
9292
}
9393
9494
class StorageLocationType {
95-
<<enumeration>>
95+
<<dataclass frozen>>
9696
SYNAPSE_S3
9797
EXTERNAL_S3
9898
EXTERNAL_GOOGLE_CLOUD
9999
EXTERNAL_SFTP
100+
EXTERNAL_HTTPS
100101
EXTERNAL_OBJECT_STORE
101102
PROXY
102103
}
@@ -107,19 +108,24 @@ classDiagram
107108
GOOGLE_CLOUD_STORAGE
108109
SFTP
109110
HTTPS
111+
PROXYLOCAL
110112
NONE
111113
}
112114
113115
class StorageLocationConfigurable {
114116
<<mixin>>
115-
+set_storage_location(storage_location_id) ProjectSetting
116-
+get_project_setting(setting_type) ProjectSetting
117-
+delete_project_setting(setting_id)
118117
+get_sts_storage_token(permission, output_format) dict
119118
+index_files_for_migration(dest_storage_location_id, db_path) MigrationResult
120119
+migrate_indexed_files(db_path) MigrationResult
121120
}
122121
122+
class ProjectSettingsMixin {
123+
<<mixin>>
124+
+set_storage_location(storage_location_id) ProjectSetting
125+
+get_project_setting(setting_type) ProjectSetting
126+
+delete_project_setting(setting_id)
127+
}
128+
123129
class Project {
124130
+str id
125131
+str name
@@ -132,31 +138,25 @@ classDiagram
132138
+str parent_id
133139
}
134140
135-
class UploadDestinationListSetting {
136-
<<enumeration>>
137-
concreteType
138-
id
139-
projectId
140-
settingsType
141-
etag
142-
locations
143-
}
144-
145141
class ProjectSetting {
146-
<<enumeration>>
147-
concreteType
148-
id
149-
projectId
150-
settingsType
151-
etag
152-
142+
<<dataclass>>
143+
+str id
144+
+str project_id
145+
+str settings_type
146+
+List~int~ locations
147+
+str concrete_type
148+
+str etag
149+
+store() ProjectSetting
150+
+get() ProjectSetting
151+
+delete()
153152
}
153+
154154
StorageLocation --> StorageLocationType : storage_type
155155
StorageLocation --> UploadType : upload_type
156-
StorageLocationConfigurable <|-- Project : implements
157-
StorageLocationConfigurable <|-- Folder : implements
158-
StorageLocationConfigurable ..> ProjectSetting : returns
159-
StorageLocationConfigurable ..> UploadDestinationListSetting : uses
156+
StorageLocationConfigurable <|-- ProjectSettingsMixin : extends
157+
ProjectSettingsMixin <|-- Project : implements
158+
ProjectSettingsMixin <|-- Folder : implements
159+
ProjectSettingsMixin ..> ProjectSetting : returns
160160
161161
```
162162

@@ -170,9 +170,9 @@ classDiagram
170170
| [synapseclient.models.StorageLocation] | The model representing a storage location setting in Synapse |
171171
| [synapseclient.models.StorageLocationType] | Enumeration defining the supported storage backend types |
172172
| [synapseclient.models.UploadType] | Enumeration defining the upload protocol for each storage type |
173-
| [synapseclient.models.mixins.StorageLocationConfigurable] | Mixin providing storage management methods to entities |
174-
| [synapseclient.models.mixins.UploadDestinationListSetting] | Dataclass defining the upload destination list setting containing storage location IDs |
175-
| [synapseclient.models.mixins.ProjectSetting] | Dataclass defining the base project setting structure |
173+
| [synapseclient.models.mixins.StorageLocationConfigurable] | Mixin providing STS token and file migration methods |
174+
| [synapseclient.models.mixins.ProjectSettingsMixin] | Mixin extending `StorageLocationConfigurable` with storage location and project settings management |
175+
| [synapseclient.models.ProjectSetting] | Dataclass representing a project's upload destination configuration, backed by `UploadDestinationListSetting` in the REST API |
176176

177177
---
178178

@@ -305,24 +305,29 @@ flowchart TB
305305

306306
## Entity Inheritance Hierarchy
307307

308-
Projects and Folders inherit storage configuration capabilities through the
309-
`StorageLocation` mixin. This pattern allows consistent storage
310-
management across container entities.
308+
Projects and Folders inherit storage configuration capabilities through two
309+
cooperating mixins: `StorageLocationConfigurable` (STS tokens and file migration)
310+
and `ProjectSettingsMixin` (storage location and project settings management).
311+
This pattern allows consistent storage management across container entities.
311312

312313
```mermaid
313314
classDiagram
314315
direction TB
315316
316-
class StorageLocation {
317+
class StorageLocationConfigurable {
317318
<<mixin>>
318-
+set_storage_location()
319-
+get_project_setting()
320-
+delete_project_setting()
321319
+get_sts_storage_token()
322320
+index_files_for_migration()
323321
+migrate_indexed_files()
324322
}
325323
324+
class ProjectSettingsMixin {
325+
<<mixin>>
326+
+set_storage_location()
327+
+get_project_setting()
328+
+delete_project_setting()
329+
}
330+
326331
class Project {
327332
+str id
328333
+str name
@@ -337,13 +342,14 @@ classDiagram
337342
+str etag
338343
}
339344
340-
StorageLocation <|-- Project
341-
StorageLocation <|-- Folder
345+
StorageLocationConfigurable <|-- ProjectSettingsMixin
346+
ProjectSettingsMixin <|-- Project
347+
ProjectSettingsMixin <|-- Folder
342348
```
343349

344350
The mixin pattern allows `Project` and `Folder` to share storage location
345-
functionality without code duplication. Both classes inherit the same
346-
methods from `StorageLocation`.
351+
functionality without code duplication. Both classes inherit all methods
352+
from `ProjectSettingsMixin`, which itself extends `StorageLocationConfigurable`.
347353

348354
---
349355

@@ -416,8 +422,6 @@ When a Synapse client is constructed (`Synapse.__init__`), it creates an in-memo
416422

417423
- `self._sts_token_store = sts_transfer.StsTokenStore()` (see `synapseclient/client.py`)
418424

419-
The store caches STS tokens per entity and permission so repeated access to the same storage location can reuse credentials without a round-trip to the REST API.
420-
421425
```mermaid
422426
sequenceDiagram
423427
participant User
@@ -682,116 +686,47 @@ File migration is a two-phase process that first indexes all candidate files and
682686
```mermaid
683687
sequenceDiagram
684688
participant User
685-
participant Entity as Project/Folder
686689
participant IndexFn as index_files_for_migration
687690
participant DB as SQLite Database
688691
participant MigrateFn as migrate_indexed_files
689692
participant Synapse as Synapse REST API
690693
691-
Note over User,Synapse: === Phase 1: Index Files ===
692-
User->>Entity: index_files_for_migration_async
693-
activate Entity
694-
695-
Entity->>IndexFn: index_files_for_migration_async(dest_id, source_ids, file_version_strategy, include_table_files)
696-
activate IndexFn
697-
698-
IndexFn->>Synapse: Verify user owns destination storage location
699-
Synapse-->>IndexFn: OK / error
700-
701-
IndexFn->>DB: Create/open DB + ensure schema
702-
IndexFn->>DB: Store migration settings (root_id, dest_id, source_ids, file_version_strategy, include_table_files)
703-
704-
alt Entity is Project/Folder (container)
705-
IndexFn->>Synapse: get_children(parent, include_types)
706-
Synapse-->>IndexFn: Child references (folders/files/tables)
707-
708-
loop For each child (bounded concurrency)
709-
IndexFn->>Synapse: get_async(child, downloadFile=false)
710-
Synapse-->>IndexFn: Child entity
711-
IndexFn->>IndexFn: _index_entity_async(child)
712-
end
713-
714-
IndexFn->>DB: Mark container as indexed (PROJECT/FOLDER)
715-
716-
else Entity is File
717-
alt file_version_strategy = new / latest / all
718-
IndexFn->>Synapse: Get file handle metadata (and versions if needed)
719-
Synapse-->>IndexFn: File handle(s)
720-
IndexFn->>DB: Insert/append FILE migration rows (INDEXED and ALREADY_MIGRATED)
721-
else file_version_strategy = skip
722-
Note over IndexFn: Skip file entities
723-
end
724-
725-
else Entity is Table (include_table_files=true)
726-
IndexFn->>Synapse: get_columns(table_id)
727-
Synapse-->>IndexFn: Column list
728-
IndexFn->>Synapse: Query rows for FILEHANDLEID columns (+ rowId,rowVersion)
729-
Synapse-->>IndexFn: Row results (fileHandleId values)
730-
loop For each row + file-handle cell (bounded concurrency)
731-
IndexFn->>Synapse: get_file_handle_for_download(fileHandleId, objectType=TableEntity)
732-
Synapse-->>IndexFn: File handle
733-
IndexFn->>DB: Insert TABLE_ATTACHED_FILE migration row (or ALREADY_MIGRATED)
734-
end
694+
Note over User,Synapse: Phase 1: Index Files
695+
User->>IndexFn: index_files_for_migration_async(dest_id, source_ids, ...)
696+
IndexFn->>Synapse: Verify ownership of destination storage location
697+
IndexFn->>DB: Initialize DB and store migration settings
698+
699+
alt Project/Folder
700+
IndexFn->>Synapse: get_children() → recurse into each child
701+
IndexFn->>DB: Insert FILE / TABLE_ATTACHED_FILE rows per entity
702+
else File
703+
IndexFn->>Synapse: Get file handle(s) per version strategy
704+
IndexFn->>DB: Insert FILE migration rows
705+
else Table (include_table_files=true)
706+
IndexFn->>Synapse: Query FILEHANDLEID columns + fetch handles
707+
IndexFn->>DB: Insert TABLE_ATTACHED_FILE rows
735708
end
736709
737-
opt continue_on_error=true
738-
Note over IndexFn,DB: Indexing errors are recorded in DB instead of aborting
739-
end
740-
741-
IndexFn-->>Entity: MigrationResult (db_path)
742-
deactivate IndexFn
743-
744-
Entity-->>User: MigrationResult
745-
deactivate Entity
710+
IndexFn-->>User: MigrationResult (db_path)
746711
747-
Note over User,Synapse: === Phase 2: Migrate Files ===
748-
User->>Entity: migrate_indexed_files / migrate_indexed_files_async (db_path)
749-
activate Entity
712+
Note over User,Synapse: Phase 2: Migrate Files
713+
User->>MigrateFn: migrate_indexed_files_async(db_path)
714+
MigrateFn->>User: Confirm migration (skipped if force=True)
750715
751-
Entity->>MigrateFn: Start migration
752-
activate MigrateFn
753-
754-
MigrateFn->>DB: Open DB, ensure schema, load settings
755-
MigrateFn->>User: Confirm migration (unless force=True)
756-
Note over MigrateFn,DB: If not confirmed, abort and return
757-
758-
loop While there are indexed items
759-
MigrateFn->>DB: Query next batch (respecting pending/completed handles & concurrency)
760-
761-
loop For each item in batch
762-
MigrateFn->>MigrateFn: Skip if key or file handle already pending
763-
764-
MigrateFn->>DB: Check if destination file handle already exists
765-
alt Existing copy found
766-
Note over MigrateFn,DB: Reuse existing to_file_handle_id
767-
else No existing copy
768-
MigrateFn->>Synapse: Copy file to new storage (bounded concurrency)
769-
Synapse-->>MigrateFn: New to_file_handle_id
770-
end
771-
772-
alt Item is FILE (entity)
773-
alt file_version_strategy = new (version is None)
774-
MigrateFn->>Synapse: Create new file version with new file handle
775-
else specific version
776-
MigrateFn->>Synapse: Update existing version's file handle
777-
end
778-
else Item is TABLE_ATTACHED_FILE
779-
alt create_table_snapshots=True
780-
MigrateFn->>Synapse: Create table snapshot
781-
end
782-
MigrateFn->>Synapse: Update table cell via transactional table update (PartialRowSet/TableUpdateTransaction)
783-
end
784-
785-
MigrateFn->>DB: Update row status to MIGRATED/ERRORED
716+
loop Batches of indexed items
717+
MigrateFn->>DB: Check for existing destination file handle
718+
alt Not already copied
719+
MigrateFn->>Synapse: Copy file to new storage location
786720
end
787-
721+
alt FILE entity
722+
MigrateFn->>Synapse: Create new version or update existing file handle
723+
else TABLE_ATTACHED_FILE
724+
MigrateFn->>Synapse: Snapshot table (if enabled) + update cell via PartialRowSet
725+
end
726+
MigrateFn->>DB: Mark row MIGRATED / ERRORED
788727
end
789728
790-
MigrateFn-->>Entity: MigrationResult (migrated counts)
791-
deactivate MigrateFn
792-
793-
Entity-->>User: MigrationResult
794-
deactivate Entity
729+
MigrateFn-->>User: MigrationResult (counts)
795730
```
796731

797732
<br>
@@ -816,5 +751,5 @@ sequenceDiagram
816751
|----------|-------------|
817752
| [Storage Location Tutorial](../tutorials/python/storage_location.md) | Step-by-step guide to using storage locations |
818753
| [StorageLocation API Reference][synapseclient.models.StorageLocation] | Complete API documentation |
819-
| [StorageLocation Mixin][synapseclient.models.mixins.StorageLocation] | Mixin methods for Projects and Folders |
754+
| [ProjectSettingsMixin][synapseclient.models.mixins.ProjectSettingsMixin] | Mixin methods for Projects and Folders |
820755
| [Custom Storage Locations (Synapse Docs)](https://help.synapse.org/docs/Custom-Storage-Locations.2048327803.html) | Official Synapse documentation |

0 commit comments

Comments
 (0)