Merge branch 'develop' into synpy-1764-trivy-scanning

Author: BryanFauble

CLAUDE.md

@@ -11,8 +11,9 @@ Synapse Python Client — official Python SDK and CLI for Synapse (synapse.org),
 - Models: stdlib dataclasses (NOT Pydantic)
 - Tests: pytest 8.2, pytest-asyncio, pytest-socket, pytest-xdist
 - Docs: MkDocs with Material theme, mkdocstrings
-- Linting: ruff, black (line-length 88), isort (profile=black), bandit, flake8
+- Linting: ruff, black (line-length 88), isort (profile=black), bandit
 - CI: GitHub Actions → SonarCloud, PyPI deploy on release
+- Docker: `Dockerfile` at repo root, published to `ghcr.io/sage-bionetworks/synapsepythonclient`
 
 ## Commands
 
@@ -94,8 +95,8 @@ Data flow: User → `operations/` factory → model async methods → `api/` ser
 ## Constraints
 
 - Do not use Pydantic for models — the codebase uses stdlib dataclasses with custom serialization. Mixing would break the `@async_to_sync` decorator and `fill_from_dict()` pattern.
-- Do not write synchronous test files — write async tests only. The `@async_to_sync` decorator is validated by a dedicated smoke test. Duplicate sync tests were removed to cut CI cost.
-- Unit tests must not make network calls — `pytest-socket` blocks all sockets. Use `pytest-mock` for HTTP mocking.
+- For new tests, prefer async test modules. Existing synchronous unit tests under `tests/unit/` are retained and maintained; the `@async_to_sync` decorator is covered by a dedicated smoke test, so avoid adding duplicate sync/async test coverage.
+- On non-Windows platforms, unit tests must not make external network calls — `pytest-socket` blocks internet-facing sockets while allowing Unix domain sockets. Socket blocking is skipped on Windows. Use `pytest-mock` for HTTP mocking.
 - `develop` is the default/main branch, not `main` or `master`. PRs target `develop`.
 - Legacy classes in root `synapseclient/` (entity.py, table.py, etc.) are kept for backwards compatibility. New features go in `models/` using the dataclass pattern.
 - Avoid adding new methods to `client.py` (9600+ lines) — prefer the `api/` + `models/` layered pattern.
@@ -108,3 +109,9 @@ Data flow: User → `operations/` factory → model async methods → `api/` ser
 - Unit test client fixture: session-scoped, `skip_checks=True`, `cache_client=False`
 - Integration tests use `--reruns 3` for flaky retries and `-n 8 --dist loadscope` for parallelism
 - Integration fixtures create per-worker Synapse projects; use `schedule_for_cleanup()` for teardown
+- Auth env vars: `SYNAPSE_AUTH_TOKEN` (bearer token), `SYNAPSE_PROFILE` (config file profile, default: `"default"`), `SYNAPSE_TOKEN_AWS_SSM_PARAMETER_NAME` (AWS SSM path)
+- CI runs integration tests only on Python 3.10 and 3.14 (oldest + newest) to limit Synapse server load
+
+## Maintenance
+
+Each CLAUDE.md file has a `<!-- Last reviewed: YYYY-MM -->` header. Update this when the file is reviewed or modified. If a code change invalidates guidance in a CLAUDE.md file, update the guidance in the same PR.

docs/CLAUDE.md

@@ -8,6 +8,9 @@ User-facing documentation for the Synapse Python Client. Built with MkDocs + Mat
 
 MkDocs with Material theme, mkdocstrings (Google-style docstrings), termynal (CLI animations), markdown-include (file embedding).
 
+### Python style
+- Use built-in generics (`list`, `dict`, `tuple`, `set`) instead of `typing.List`, `typing.Dict`, etc. (Python 3.9+)
+
 ## Conventions
 
 ### Content types (Diataxis framework)
@@ -50,4 +53,9 @@ Defined in `mkdocs.yml` nav section. 5 main sections: Home, Tutorials, How-To Gu
 - Do not edit tutorial code inline in markdown — edit the `.py` script file in `tutorial_scripts/` and update line ranges if needed.
 - Reference docs auto-generate from source docstrings — to change method documentation, edit the docstring in the Python source, not the markdown.
 - `mkdocs.yml` is at the repo root, not in `docs/` — it configures the entire doc build.
-- Docs deploy via `mkdocs gh-deploy --force` targeting the `master` branch (not `develop`).
+- Docs deploy to Read the Docs (configured via `.readthedocs.yaml` at repo root).
+- Local build output goes to `docs_site/` (via `site_dir` in `mkdocs.yml`) — gitignored.
+- Cross-referencing uses the `autorefs` plugin: `[display text][synapseclient.ClassName.method]` auto-resolves to mkdocstrings anchors.
+
+### news.md
+Release notes live in `docs/news.md`. Each release gets a heading with the version number and date, followed by bullet points describing changes. Group entries by category (Features, Bug Fixes, etc.). Reference Jira ticket numbers (SYNPY-XXXX) in each entry.

synapseclient/api/CLAUDE.md

@@ -16,33 +16,68 @@ async def verb_resource(
 ) -> Dict[str, Any]:
 ```
 - All functions are `async def`
+- `synapse_client` is **always** `Optional["Synapse"] = None` — never make it required. Callers omit it to use the cached singleton returned by `Synapse.get_client()`.
 - `synapse_client` is always the last parameter, keyword-only (after `*`)
 - Use `Synapse.get_client(synapse_client=synapse_client)` to get the client instance
 - Use `TYPE_CHECKING` guard for `Synapse` import — avoids circular dependencies between `api/` and `client.py`
+- Construct a `query_params` dictionary for non-null optional args, and pass it to the `params` arg of the REST call. See `entity_services.py` for the pattern.
+
+### Docstring conventions
+Module-level — every file opens with boilerplate linking to the Synapse REST controller:
+```python
+"""This module is responsible for exposing the services defined at:
+<https://rest-docs.synapse.org/rest/#org.sagebionetworks.repo.web.controller.XController>
+"""
+```
+Function-level (Google style):
+```python
+"""
+One-line summary.
+
+<https://rest-docs.synapse.org/rest/POST/endpoint.html>
+
+Arguments:
+    param: Description.
+    synapse_client: If not passed in and caching was not disabled by
+        `Synapse.allow_client_caching(False)` this will use the last created
+        instance from the Synapse class constructor.
+
+Returns:
+    Description of return value.
+"""
+```
+- The `synapse_client` argument description is boilerplate — always copy it verbatim, not paraphrased.
+- The REST endpoint URL uses `<link>` format (angled brackets), not markdown `[text](url)`.
+- Parameter descriptions in `Arguments:` must be copied verbatim from the Synapse REST API docs for that endpoint — do not paraphrase or infer.
 
 ### REST call pattern
 ```python
 client = Synapse.get_client(synapse_client=synapse_client)
 return await client.rest_post_async(uri="/endpoint", body=json.dumps(request))
 ```
-Available methods: `rest_get_async`, `rest_post_async`, `rest_put_async`, `rest_delete_async`. Pass `endpoint=client.fileHandleEndpoint` for file handle operations; omit for the default repository endpoint. Use `json.dumps()` for request bodies — not raw dicts.
+Available methods: `rest_get_async`, `rest_post_async`, `rest_put_async`, `rest_delete_async`. Pass `endpoint=client.fileHandleEndpoint` for file handle operations; omit for the default repository endpoint. Use `json.dumps()` for request bodies — not raw dicts. Always assign the response to a named `response` variable before returning or extracting attributes from it.
 
 ### Return values
 - Most functions return raw `Dict[str, Any]` — transformation happens in the model layer via `fill_from_dict()`
 - Some return typed dataclass instances (e.g., `EntityHeader` from `entity_services.py`) when the data is only used internally
 - Delete operations return `None`
 
 ### Pagination
-Use helpers from `api_client.py`:
+Use async pagination helpers when the API endpoint returns a list of results. For single-object responses, a simple `return` is sufficient.
+
+Helpers from `api_client.py`:
 - `rest_get_paginated_async()` — for GET endpoints with limit/offset. Expects `results` or `children` key in response.
 - `rest_post_paginated_async()` — for POST endpoints with `nextPageToken`. Expects `page` array in response.
-Both are async generators yielding individual items.
+Both are async generators yielding individual items. Reference `entity_services.py`, `table_services.py`, or `evaluation_services.py` for pagination patterns.
 
 ### Entity factory (`entity_factory.py`)
 Polymorphic entity deserialization via concrete type dispatch. Maps Java class names from `core/constants/concrete_types.py` to model classes. When adding a new entity type, register the type mapping here.
 
+### When to add a new service file vs. update an existing one
+Add a new file when the Synapse REST controller is different (each file maps to one controller). Update an existing file when adding endpoints under the same controller.
+
 ### Adding a new service file
 1. Create `synapseclient/api/new_service.py`
 2. Add all public functions to `api/__init__.py` imports and `__all__` — every public function must be re-exported
 3. Use `json.dumps()` for request bodies (not dict)
-4. Reference `entity_services.py` for CRUD pattern, `table_services.py` for pagination pattern
+4. Reference `entity_services.py` for CRUD pattern, `table_services.py` or `evaluation_services.py` for pagination pattern

synapseclient/core/CLAUDE.md

@@ -32,7 +32,7 @@ Provider chain tries in order: login args → config file → env var (`SYNAPSE_
 - Download validates MD5 post-transfer, raises `SynapseMd5MismatchError` on mismatch
 - Progress via `tqdm`; multi-threaded uploads suppress per-file messages via `cumulative_transfer_progress`
 
-### concrete_types.py
+### concrete_types.py (`core/constants/concrete_types.py`)
 Maps Java class names from Synapse REST API for polymorphic deserialization. When adding a new entity type, add its concrete type string here AND in `api/entity_factory.py` type map AND in `models/mixins/asynchronous_job.py` ASYNC_JOB_URIS if it's an async job type.
 
 ### Key reusable utilities (`utils.py`)

synapseclient/core/download/CLAUDE.md

@@ -7,7 +7,7 @@ File download from Synapse storage with MD5 validation, collision handling, and
 ## Conventions
 
 ### Primary download path
-`download_async.py` is the primary async download implementation. `download_functions.py` contains shared helpers and the sync download wrapper.
+`download_async.py` is the primary async download implementation. `download_functions.py` contains shared helpers and the sync download wrapper. The default part size of 8 MiB was empirically optimized for Synapse download throughput — do not change it without benchmarking.
 
 ### MD5 validation
 Post-transfer MD5 validation is mandatory. Raises `SynapseMd5MismatchError` on mismatch — the download is retried automatically (60 retries spanning ~30 minutes).

synapseclient/extensions/curator/CLAUDE.md

@@ -10,28 +10,14 @@ Optional dependencies (gated by `[curator]` extras): pandas, pandarallel, networ
 
 ## Conventions
 
-### schema_generation.py (5984 lines)
-Largest file in the codebase. Contains `DataModelParser`, `DataModelComponent`, `DataModelRelationships` classes. Uses networkx (DiGraph, MultiDiGraph) for node/edge relationships and cycle detection (via multiprocessing). Many deprecated validation rule enums marked for removal (SYNPY-1724, SYNPY-1692). Active development area — multiple recent PRs modifying conditionals, display names, and grouping.
-
-### schema_registry.py
-Query engine for the schema registry table. Default table ID: `syn69735275` (configurable via parameter). Builds SQL WHERE clauses from filter kwargs — supports exact match and LIKE pattern match. `return_latest_only=True` returns newest version URI only.
+### schema_generation.py
+Largest file in the codebase. Uses networkx (DiGraph, MultiDiGraph) for node/edge relationships and cycle detection (via multiprocessing). Many deprecated validation rule enums marked for removal (SYNPY-1724, SYNPY-1692). Active development area.
 
 ### schema_management.py
-Thin wrappers around `JSONSchema` OOP model:
-- `register_jsonschema()` / `register_jsonschema_async()` — loads schema from file, calls `.store_async()`
-- `bind_jsonschema()` / `bind_jsonschema_async()` — binds schema to entity
-- `fix_schema_name()` — replaces dashes/underscores with periods for Synapse compliance
-
-Uses `wrap_async_to_sync()` for sync versions (not class decorator).
-
-### file_based_metadata_task.py
-Creates EntityView from JSON Schema bound to folder/project. `create_json_schema_entity_view()` auto-reorders columns (createdBy→name→id to front). `create_or_update_wiki_with_entity_view()` embeds EntityView query in Wiki page.
-
-### record_based_metadata_task.py
-Extracts schema properties → DataFrame → RecordSet → CurationTask + Grid. Supports URI-based schemas via `JSONSchema.from_uri()`.
+Uses `wrap_async_to_sync()` for sync versions (not class decorator). `fix_schema_name()` replaces dashes/underscores with periods for Synapse compliance.
 
 ### utils.py
-`project_id_from_entity_id()` — traverses folder hierarchy up to project (max 1000 iterations). Uses legacy sync `get()` API in a loop — known tech debt.
+`project_id_from_entity_id()` — traverses folder hierarchy up to project (max 1000 iterations). Uses `operations.get` in a loop — known tech debt.
 
 ## Constraints
 

synapseclient/extensions/curator/file_based_metadata_task.py

@@ -24,7 +24,7 @@
 from synapseclient.operations import FileOptions, get
 
 TYPE_DICT = {
-    "string": ColumnType.STRING,
+    "string": ColumnType.MEDIUMTEXT,
     "number": ColumnType.DOUBLE,
     "integer": ColumnType.INTEGER,
     "boolean": ColumnType.BOOLEAN,
@@ -199,48 +199,67 @@ def _create_columns_from_json_schema(json_schema: dict[str, Any]) -> list[Column
         raise ValueError(
             "The 'properties' field in the JSON Schema must be a dictionary."
         )
-    columns = []
-    for name, prop_schema in properties.items():
-        column_type = _get_column_type_from_js_property(prop_schema)
-        maximum_size = None
-        if column_type == "STRING":
-            maximum_size = 100
-        if column_type in LIST_TYPE_DICT.values():
-            maximum_size = 5
-
-        column = Column(
-            name=name,
-            column_type=column_type,
-            maximum_size=maximum_size,
-            default_value=None,
-        )
-        columns.append(column)
+    columns = [
+        _create_synapse_column_from_js_property(prop_schema, name)
+        for name, prop_schema in properties.items()
+    ]
     return columns
 
 
+def _create_synapse_column_from_js_property(
+    js_property: dict[str, Any], name: str
+) -> Column:
+    """
+    Creates a Synapse Column based on a JSON Schema property.
+
+    Args:
+        js_property: A JSON Schema property in dict form.
+        name: The name of the column.
+
+    Returns:
+        A Synapse Column based on the JSON Schema property.
+    """
+    column_type = _get_column_type_from_js_property(js_property)
+    return Column(name=name, column_type=column_type)
+
+
 def _get_column_type_from_js_property(js_property: dict[str, Any]) -> ColumnType:
     """
     Gets the Synapse column type from a JSON Schema property.
     The JSON Schema should be valid but that should not be assumed.
-    If the type can not be determined ColumnType.STRING will be returned.
+    If the type can not be determined ColumnType.MEDIUMTEXT will be returned.
 
     Args:
         js_property: A JSON Schema property in dict form.
 
     Returns:
         A Synapse ColumnType based on the JSON Schema type
     """
-    # Enums are always strings in Synapse tables
+    # Enums are set as MediumText columns
     if "enum" in js_property:
-        return ColumnType.STRING
+        return ColumnType.MEDIUMTEXT
     if "type" in js_property:
-        if js_property["type"] == "array":
+        js_type = js_property["type"]
+        # Synapse columns cannot be more than one type
+        # If the JSONSchema type is a list of types, check if it's a nullable single type
+        if isinstance(js_type, list):
+            types = [t for t in js_type if t != "null"]
+            if len(types) == 1:
+                js_type = types[0]
+            # If there are multiple non-null types, we cannot determine a single column type, so default to MediumText
+            else:
+                return ColumnType.MEDIUMTEXT
+        if js_type == "array":
             return _get_list_column_type_from_js_property(js_property)
-        return TYPE_DICT.get(js_property["type"], ColumnType.STRING)
+        # If there is only one JSONSChema type, return the corresponding Synapse column type,
+        #  defaulting to MediumText if there is no match
+        return TYPE_DICT.get(js_type, ColumnType.MEDIUMTEXT)
     # A oneOf list usually indicates that the type could be one or more different things
+    # Curator extension does not create the types of JSON Schemas where this is the case
+    #  but if it is present we will attempt to determine the type based on the items in the oneOf list.
     if "oneOf" in js_property and isinstance(js_property["oneOf"], list):
         return _get_column_type_from_js_one_of_list(js_property["oneOf"])
-    return ColumnType.STRING
+    return ColumnType.MEDIUMTEXT
 
 
 def _get_column_type_from_js_one_of_list(js_one_of_list: list[Any]) -> ColumnType:
@@ -258,15 +277,15 @@ def _get_column_type_from_js_one_of_list(js_one_of_list: list[Any]) -> ColumnTyp
     items = [item for item in js_one_of_list if isinstance(item, dict)]
     # Enums are always strings in Synapse tables
     if [item for item in items if "enum" in item]:
-        return ColumnType.STRING
+        return ColumnType.MEDIUMTEXT
     # For Synapse ColumnType we can ignore null types in JSON Schemas
     type_items = [item for item in items if "type" in item if item["type"] != "null"]
     if len(type_items) == 1:
         type_item = type_items[0]
         if type_item["type"] == "array":
             return _get_list_column_type_from_js_property(type_item)
-        return TYPE_DICT.get(type_item["type"], ColumnType.STRING)
-    return ColumnType.STRING
+        return TYPE_DICT.get(type_item["type"], ColumnType.MEDIUMTEXT)
+    return ColumnType.MEDIUMTEXT
 
 
 def _get_list_column_type_from_js_property(js_property: dict[str, Any]) -> ColumnType: