Merge branch 'develop' into SYNPY-1802

Author: andrewelamb

docs/CLAUDE.md

@@ -33,6 +33,7 @@ Reference markdown files use `::: synapseclient.ClassName` syntax to trigger aut
 - `filters: ["!^_", "!to_synapse_request", "!fill_from_dict"]` — private members, `to_synapse_request()`, and `fill_from_dict()` are excluded from docs
 - `inherited_members: true` — shows mixin methods on inheriting classes
 - Member lists are explicit — each reference page specifies which methods to document
+- When adding a new public method to a model class, add it to the `members:` list in the corresponding reference pages (`docs/reference/experimental/sync/` and `docs/reference/experimental/async/`). Without this, mkdocstrings won't generate an anchor and cross-references like `[synapseclient.models.ClassName.method]` will break.
 
 ### Anchor links for cross-referencing
 Pattern: `[](){ #reference-anchor }` in reference pages. Tutorials link to reference via `[API Reference][project-reference-sync]`. Explicit type hints use: `[syn.login][synapseclient.Synapse.login]`.

docs/explanations/manifest_csv.md

@@ -0,0 +1,119 @@
+# Manifest CSV
+
+The manifest is a CSV file with file locations and metadata used to bulk upload and download files in Synapse. It is the standard manifest format used by `Project.sync_from_synapse`, `Project.sync_to_synapse`, `Folder.sync_from_synapse`, `Folder.sync_to_synapse`, the Synapse UI download cart, and the `synapse get-download-list` CLI command.
+
+!!! note
+    This CSV manifest replaces the legacy TSV manifest produced by `synapseutils.syncFromSynapse`. The `syncFromSynapse` and `syncToSynapse` utility functions are deprecated and will be removed in v5.0.0. Use `Project.sync_from_synapse` / `Folder.sync_from_synapse` and `Project.sync_to_synapse` / `Folder.sync_to_synapse` instead. See the [legacy TSV manifest documentation](manifest_tsv.md) for details on the old format.
+
+## Manifest file format
+
+The format of the manifest file is a comma-separated value (CSV) file with one row per file and columns describing the file. The minimum required columns for uploading are **path** and **parentId**, where `path` is the local file path and `parentId` is the Synapse ID of the project or folder where the file is uploaded to. Values that contain commas are automatically quoted (e.g., `"hello, world"`).
+
+### Required fields for upload
+
+| Field    | Meaning                    | Example                 |
+|----------|----------------------------|-------------------------|
+| path     | local file path or URL     | /path/to/local/file.txt |
+| parentId | Synapse ID of parent       | syn1235                 |
+
+!!! note
+    The legacy TSV manifest used a column named `parent`. The CSV manifest uses `parentId` instead, which is consistent with the Synapse REST API field name. If you are migrating an existing TSV manifest to CSV, rename the `parent` column to `parentId`.
+
+### Standard fields
+
+These columns are recognized by `sync_to_synapse` and have specific meaning. Any of these columns may be present in the manifest but only `path` and `parentId` are required for upload.
+Each of these are individual examples and is what you would find in a row in each of these columns. To clarify, "syn1235;/path/to_local/file.txt" below states that you would like both "syn1235" and "/path/to_local/file.txt" added as items used to generate a file. You can also specify one item by specifying "syn1234"
+
+| Field               | Meaning                                    | Example                                      |
+|---------------------|--------------------------------------------|----------------------------------------------|
+| path                | local file path or URL                     | /path/to/local/file.txt                      |
+| parentId            | Synapse ID of parent container             | syn1235                                      |
+| ID                  | Synapse entity ID                          | syn2345                                      |
+| name                | name of file in Synapse                    | Example_file                                 |
+| synapseStore        | whether to upload the file                 | True                                         |
+| contentType         | content type of file to overwrite defaults | text/html                                    |
+| forceVersion        | whether to update version                  | False                                        |
+| activityName        | name of activity in provenance             | Ran normalization                            |
+| activityDescription | text description of what was done          | Ran algorithm xyz with parameters...         |
+| used                | list of items used to generate file        | syn1235;/path/to_local/file.txt              |
+| executed            | list of items executed                     | https://github.org/;/path/to_local/code.py   |
+
+### Metadata fields (ignored during upload)
+
+These columns are present in manifests generated by the Synapse UI download cart and `synapse get-download-list` CLI. They are ignored by `sync_to_synapse` and are **not** treated as annotations.
+
+| Field             | Meaning                       |
+|-------------------|-------------------------------|
+| error             | any error in downloading file |
+| versionNumber     | version of the file           |
+| dataFileSizeBytes | size of the file in bytes     |
+| createdBy         | user who created the file     |
+| createdOn         | date the file was created     |
+| modifiedBy        | user who last modified        |
+| modifiedOn        | date last modified            |
+| synapseURL        | URL to the file in Synapse    |
+| dataFileMD5Hex    | MD5 hash of the file          |
+
+### Annotations
+
+Any columns that are not in the standard or metadata fields described above will be interpreted as annotations of the file.
+
+Adding annotations to each row:
+
+| path | parentId | annot1 | annot2 | annot3 | annot4 | annot5 |
+| --- | --- | --- | --- | --- | --- | --- |
+| /path/file1.txt | syn1243 | bar | 3.1415 | "aaaa, bbbb" | "[14,27,30]" | "Annotation, with a comma" |
+| /path/file2.txt | syn12433 | baz | 2.71 | value_1 | "[1,2,3]" | test 123 |
+| /path/file3.txt | syn12455 | zzz | 3.52 | value_3 | "[42,56,77]" | a single annotation |
+
+#### Multiple values of annotations per key
+
+Using multiple values for a single annotation should be used sparingly as it makes it more
+difficult for you to manage the data. However, it is supported.
+
+**Annotations can be comma `,` separated lists surrounded by brackets `[]`.**
+
+Because the manifest is a CSV file, multi-value annotations that contain commas are automatically quoted. For example, `[a,b,c]` will appear in the CSV as `"[a,b,c]"`.
+
+This is an annotation with 3 values:
+
+| path            | parentId | annot1       |
+|-----------------|----------|--------------|
+| /path/file1.txt | syn1243  | "[a,b,c]"    |
+
+
+
+### Dates in the manifest file
+
+Dates within the manifest file will always be written as [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) format in UTC without milliseconds. For example: `2023-12-20T16:55:08Z`.
+
+Dates can be written in other formats specified in ISO 8601 and they will be recognized. However, `sync_from_synapse` will always write dates in the UTC format specified above. For example, you may want to specify a datetime at a specific timezone like `2023-12-20 23:55:08-07:00` and this will be recognized as a valid datetime.
+
+## Manifest sources
+
+The CSV manifest format is shared across multiple tools:
+
+| Source                                                   | Filename                        |
+|----------------------------------------------------------|---------------------------------|
+| `Project.sync_from_synapse` / `Folder.sync_from_synapse` | manifest.csv                    |
+| Synapse UI download cart                                 | manifest.csv                    |
+| CLI `synapse get-download-list`                          | `manifest_<timestamp>.csv`      |
+
+A manifest generated by any of these sources can be used as input to `sync_to_synapse`, provided the `path` column is present with valid local file paths. Manifests from the Synapse UI do not include a `path` column by default, so users must add it before uploading.
+
+### Example manifest file
+
+| path            | parentId | ID      | name      | annot1 | annot2 | collection_date           | used                     | executed                     |
+|-----------------|----------|---------|-----------|--------|--------|---------------------------|--------------------------|------------------------------|
+| /path/file1.txt | syn1243  | syn5001 | file1.txt | bar    | 3.1415 | 2023-12-04T07:00:00Z      | syn124;/path/file2.txt   | https://github.org/foo/bar   |
+| /path/file2.txt | syn12433 | syn5002 | file2.txt | baz    | 2.71   | 2001-01-01T08:00:00Z      |                          | https://github.org/foo/baz   |
+| /path/file3.txt | syn12455 | syn5003 | file3.txt | zzz    | 3.52   | 2023-12-04T07:00:00Z      |                          | https://github.org/foo/zzz   |
+
+## References
+
+- [Project.sync_from_synapse][synapseclient.models.Project.sync_from_synapse]
+- [Project.sync_to_synapse][synapseclient.models.Project.sync_to_synapse]
+- [Folder.sync_from_synapse][synapseclient.models.Folder.sync_from_synapse]
+- [Folder.sync_to_synapse][synapseclient.models.Folder.sync_to_synapse]
+- [Manifest TSV (legacy)](manifest_tsv.md)
+- [Managing custom metadata at scale](https://help.synapse.org/docs/Managing-Custom-Metadata-at-Scale.2004254976.html#ManagingCustomMetadataatScale-BatchUploadFileswithAnnotations)

docs/explanations/manifest_tsv.md

@@ -1,6 +1,9 @@
-# Manifest
+# Manifest TSV (Legacy)
 The manifest is a tsv file with file locations and metadata to be pushed to Synapse. The purpose is to allow bulk actions through a TSV without the need to manually execute commands for every requested action.
 
+!!! warning "Deprecated"
+    This TSV manifest format is produced by [synapseutils.syncFromSynapse][] and consumed by [synapseutils.syncToSynapse][], both of which are deprecated and will be removed in v5.0.0. Use `Project.sync_from_synapse` / `Folder.sync_from_synapse` and `Project.sync_to_synapse` / `Folder.sync_to_synapse` instead, which use the [CSV manifest format](manifest_csv.md).
+
 ## Manifest file format
 
 The format of the manifest file is a tab delimited file with one row per file to upload and columns describing the file. The minimum required columns are **path** and **parent** where path is the local file path and parent is the Synapse Id of the project or folder where the file is uploaded to.
@@ -20,6 +23,9 @@ Any additional columns will be added as annotations.
 | path   | local file path or URL | /path/to/local/file.txt |
 | parent | synapse id             | syn1235                 |
 
+!!! note "Column renamed in CSV format"
+    The CSV manifest format uses `parentId` instead of `parent`. If you are migrating to the new [CSV manifest format](manifest_csv.md), rename the `parent` column to `parentId`.
+
 ### Common fields:
 
 | Field        | Meaning                   | Example      |

docs/reference/experimental/async/folder.md

@@ -16,6 +16,7 @@ at your own risk.
         - copy_async
         - walk_async
         - sync_from_synapse_async
+        - sync_to_synapse_async
         - flatten_file_list
         - map_directory_to_all_contained_files
         - get_permissions_async

docs/reference/experimental/async/project.md

@@ -15,6 +15,7 @@ at your own risk.
         - delete_async
         - walk_async
         - sync_from_synapse_async
+        - sync_to_synapse_async
         - flatten_file_list
         - map_directory_to_all_contained_files
         - get_permissions_async

docs/reference/experimental/sync/folder.md

@@ -27,6 +27,7 @@ at your own risk.
         - copy
         - walk
         - sync_from_synapse
+        - sync_to_synapse
         - flatten_file_list
         - map_directory_to_all_contained_files
         - get_permissions

docs/reference/experimental/sync/project.md

@@ -26,6 +26,7 @@ at your own risk.
         - delete
         - walk
         - sync_from_synapse
+        - sync_to_synapse
         - flatten_file_list
         - map_directory_to_all_contained_files
         - get_permissions

docs/tutorials/configuration.md

@@ -2,32 +2,107 @@
 
 The Synapse Python client can be configured either programmatically or by using a configuration file.
 
-**The default configuration file does not need to be modified for most use-cases**.
+!!! note "Default Configuration"
+    The default configuration file does not need to be modified for most use-cases
 
+When installing the Synapse Python client, the `.synapseConfig` file is added to your home directory if it doesn't exist already. This file stores configuration options including your Synapse auth token, cache location, multi-threading settings, and storage credentials.
 
-When installing the Synapse Python client, the `.synapseConfig` is added to your home directory. This configuration file is used to store a number of configuration options, including your Synapse authtoken, cache, and multi-threading settings.
-
-A full example `.synapseConfig` can be found in the [github repository](https://github.com/Sage-Bionetworks/synapsePythonClient/blob/develop/synapseclient/.synapseConfig).
+A full annotated example `.synapseConfig` can be found in the [GitHub repository](https://github.com/Sage-Bionetworks/synapsePythonClient/blob/develop/synapseclient/.synapseConfig).
 
 ## `.synapseConfig` sections
 
-### `[authentication]`
+### `[default]` and `[profile <name>]`
+
+Holds Synapse login credentials. `[default]` is used when no profile is specified; named profiles use `[profile <name>]` syntax. See the [authentication](./authentication.md) document for full details including how to create tokens, select profiles, and use environment variables.
+
+### `[sftp://hostname]`
+
+Credentials for files stored on SFTP servers. Use one section per server; the section name is the full SFTP URL.
+
+| Key | Description |
+| --- | --- |
+| `username` | Username for the SFTP server. |
+| `password` | Password for the SFTP server. |
+
+```ini
+[sftp://some.sftp.url.com]
+username = sftpuser
+password = sftppassword
+```
 
-See details on this section in the [authentication](./authentication.md) document.
+### `[https://s3.amazonaws.com/bucket_name]`
+
+Credentials for files stored in AWS S3 or S3-compatible storage that Synapse does not manage access for. Use one section per bucket; the section name is the full endpoint URL including the bucket name.
+
+| Key | Description |
+| --- | --- |
+| `profile_name` | Name of an AWS CLI profile from `~/.aws/credentials`. If omitted, the `default` AWS profile is used. |
+
+```ini
+[https://s3.amazonaws.com/bucket_name]
+profile_name = local_credential_profile_name
+```
+
+For more information on AWS credentials files, see the [AWS CLI documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html).
 
 ### `[cache]`
 
-Your downloaded files are cached to avoid repeat downloads of the same file. Change 'location' to use a different folder on your computer as the cache location
+Downloaded files are cached to avoid repeat downloads of the same file.
+
+| Key | Description |
+| --- | --- |
+| `location` | Path to the cache directory. Supports `~` and environment variables. Default: `~/.synapseCache`. |
+
+```ini
+[cache]
+location = ~/.synapseCache
+```
+
+### `[debug]`
+
+When this section is present (no keys required), the client prints debug-level log output. Equivalent to passing `debug=True` to the `Synapse()` constructor.
+
+```ini
+[debug]
+```
 
 ### `[endpoints]`
 
-Configuring these will cause the Python client to use these as Synapse service endpoints instead of the default prod endpoints.
+Override the default Synapse production service endpoints. Useful for testing against staging or development environments.
+
+| Key | Description |
+| --- | --- |
+| `repoEndpoint` | Synapse repository REST API endpoint. |
+| `authEndpoint` | Synapse authentication service endpoint. |
+| `fileHandleEndpoint` | Synapse file service endpoint. |
+| `portalEndpoint` | Synapse web portal URL. |
+
+Note: The following are the default endpoints.
+
+```ini
+[endpoints]
+repoEndpoint = https://repo-prod.prod.sagebase.org/repo/v1
+authEndpoint = https://auth-prod.prod.sagebase.org/auth/v1
+fileHandleEndpoint = https://file-prod.prod.sagebase.org/file/v1
+portalEndpoint = https://www.synapse.org/
+```
 
 ### `[transfer]`
 
-Settings to configure how Synapse uploads/downloads data.
+Settings to configure how Synapse uploads and downloads data.
+
+| Key | Description |
+| --- | --- |
+| `max_threads` | Number of concurrent threads/connections for file transfers. Applies to AWS S3 transfers (uploads and downloads). Default: `min(cpu_count + 4, 128)`. Maximum: `128`. Minimum: `1`. |
+| `use_boto_sts` | If `true`, use AWS STS (Security Token Service) to obtain temporary credentials for S3 transfers instead of using stored AWS credentials directly. Valid values: `true` or `false` (case-insensitive). Default: `false`. |
+
+```ini
+[transfer]
+max_threads = 16
+use_boto_sts = false
+```
 
-You may also set the `max_threads` programmatically via:
+You may also set `max_threads` programmatically:
 
 ```python
 import synapseclient