|
1 | | -# storage_location_mixin |
| 1 | +# Storage Location |
2 | 2 |
|
3 | | -The `storage_location_mixin` module provides two mixins for managing storage locations |
4 | | -on entities (Projects and Folders): |
5 | | - |
6 | | -- **`StorageLocationConfigurable`** — base mixin providing STS credentials and file migration. |
7 | | -- **`ProjectSettingsMixin`** — extends `StorageLocationConfigurable` with project-level storage settings. |
8 | | - |
9 | | -For architecture diagrams and design documentation, see |
10 | | -[Storage Location Architecture](../../../explanations/storage_location_architecture.md). |
11 | | - |
12 | | -## Methods Overview |
13 | | - |
14 | | -### `StorageLocationConfigurable` |
15 | | - |
16 | | -| Method | Description | |
17 | | -|--------|-------------| |
18 | | -| `get_sts_storage_token` | Get STS credentials for direct S3 access | |
19 | | -| `index_files_for_migration` | Index files for migration to a new storage location | |
20 | | -| `migrate_indexed_files` | Migrate previously indexed files | |
21 | | - |
22 | | -### `ProjectSettingsMixin` (extends `StorageLocationConfigurable`) |
23 | | - |
24 | | -| Method | Description | |
25 | | -|--------|-------------| |
26 | | -| `set_storage_location` | Set the upload storage location for this entity (destructive replace) | |
27 | | -| `get_project_setting` | Get project settings (upload, external_sync, etc.) | |
28 | | -| `delete_project_setting` | Delete a project setting | |
29 | | - |
30 | | -All methods have async equivalents with an `_async` suffix (e.g. `get_sts_storage_token_async`). |
31 | | - |
32 | | -## Migration Workflow |
33 | | - |
34 | | -Migration is a two-step process: |
35 | | - |
36 | | -1. **Index** — call `index_files_for_migration` to scan the entity and record files to migrate in a local SQLite database. |
37 | | -2. **Migrate** — call `migrate_indexed_files` with the database path to perform the actual copy. |
38 | | - |
39 | | -### `index_files_for_migration` parameters |
40 | | - |
41 | | -| Parameter | Default | Description | |
42 | | -|-----------|---------|-------------| |
43 | | -| `dest_storage_location_id` | required | Destination storage location ID | |
44 | | -| `db_path` | `None` | Path for the SQLite tracking database; a temp path is used if omitted | |
45 | | -| `source_storage_location_ids` | `None` | Restrict to files in these source locations; `None` means all locations | |
46 | | -| `file_version_strategy` | `"new"` | `"new"` / `"all"` / `"latest"` / `"skip"` | |
47 | | -| `include_table_files` | `False` | Whether to include files attached to tables | |
48 | | -| `continue_on_error` | `False` | Record errors and continue rather than raising | |
49 | | - |
50 | | -Returns a `MigrationResult` — access `result.db_path` to pass to the next step. |
51 | | - |
52 | | -### `migrate_indexed_files` parameters |
53 | | - |
54 | | -| Parameter | Default | Description | |
55 | | -|-----------|---------|-------------| |
56 | | -| `db_path` | required | Path returned by `index_files_for_migration` | |
57 | | -| `create_table_snapshots` | `True` | Create table snapshots before migrating table files | |
58 | | -| `continue_on_error` | `False` | Record errors and continue rather than raising | |
59 | | -| `force` | `False` | Skip the interactive confirmation prompt (required for non-interactive/CI use) | |
60 | | - |
61 | | -Returns a `MigrationResult`, or `None` if migration was aborted (user declined the prompt, |
62 | | -or the session is non-interactive and `force=False`). |
63 | | - |
64 | | -## Usage Examples |
65 | | - |
66 | | - |
67 | | -### Set a storage location |
68 | | - |
69 | | -```python |
70 | | -from synapseclient.models import Folder |
71 | | - |
72 | | -# Replace all storage locations on a folder |
73 | | -folder = Folder(id="syn123").get() |
74 | | -folder.set_storage_location(storage_location_id=12345) |
75 | | - |
76 | | -# Add a storage location without removing existing ones |
77 | | -setting = folder.get_project_setting(setting_type="upload") |
78 | | -if setting: |
79 | | - setting.locations.append(67890) |
80 | | - setting.store() |
81 | | -``` |
82 | | - |
83 | | -### Get STS credentials |
84 | | -Note: Entity must have an STS-enabled storage location |
85 | | -```python |
86 | | -credentials = folder.get_sts_storage_token( |
87 | | - permission="read_write", |
88 | | - output_format="boto", |
89 | | -) |
90 | | -``` |
91 | | -### Migrate files to a new storage location |
92 | | - |
93 | | -```python |
94 | | -import asyncio |
95 | | -from synapseclient import Synapse |
96 | | -from synapseclient.models import Project |
97 | | - |
98 | | -syn = Synapse() |
99 | | -syn.login() |
100 | | - |
101 | | -async def main(): |
102 | | - project = await Project(id="syn123").get_async() |
103 | | - |
104 | | - # Step 1: index |
105 | | - index_result = await project.index_files_for_migration_async( |
106 | | - dest_storage_location_id=12345, |
107 | | - ) |
108 | | - print(f"Database path: {index_result.db_path}") |
109 | | - |
110 | | - # Step 2: migrate (force=True for non-interactive scripts) |
111 | | - result = await project.migrate_indexed_files_async( |
112 | | - db_path=index_result.db_path, |
113 | | - force=True, |
114 | | - ) |
115 | | - print(result.counts_by_status) |
116 | | - |
117 | | -asyncio.run(main()) |
118 | | -``` |
119 | 3 | ::: synapseclient.models.mixins.StorageLocationConfigurable |
120 | 4 |
|
121 | 5 | --- |
|
0 commit comments