Skip to content

Commit 21a6b0f

Browse files
Added job type override info to upload_data.md (#88)
1 parent 919128f commit 21a6b0f

1 file changed

Lines changed: 19 additions & 0 deletions

File tree

docs/source/acquire_upload/upload_data.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,25 @@ In general, most users should interact with the transfer service by requesting d
88

99
For example, this [upload script](https://github.com/AllenNeuralDynamics/aind-data-transfer-service/blob/d1f84020862c3de340020b6cb45bef0fd5105515/docs/examples/aind_data_schema_v2.py) demonstrates how to setup the upload parameters for a standard ecephys data asset using the `"default"` job_type. You can view [all available job_type options](https://aind-data-transfer-service.corp.alleninstitute.org/job_params). Please reach out to the Data & Infrastructure team in Scientific Computing to develop custom job types for your data assets.
1010

11+
Job types define convenient defaults, but you are not locked into them — **any parameter set by the job type can be overridden in your upload script**. For example, to pin the metadata mapper to a specific version rather than using the job type's default:
12+
13+
```python
14+
from aind_data_transfer_service.models.core import Task, UploadJobConfigsV2
15+
16+
gather_preliminary_metadata = Task(
17+
image_version="v1.1.0", # overrides the job_type default
18+
job_settings={"metadata_dir": "/path/to/your/data"},
19+
)
20+
21+
upload_job_configs = UploadJobConfigsV2(
22+
job_type="vr_foraging_fiber", # all other defaults still come from the job_type
23+
...
24+
tasks={"gather_preliminary_metadata": gather_preliminary_metadata},
25+
)
26+
```
27+
28+
Available mapper versions are listed [here](https://github.com/AllenNeuralDynamics/aind-metadata-mapper/pkgs/container/aind-metadata-mapper). For a complete reference of all parameters you can control, see [this example script](https://github.com/AllenNeuralDynamics/aind-data-transfer-service/blob/dev/docs/examples/aind_data_schema_v2.py).
29+
1130
## GatherMetadataJob
1231

1332
The [GatherMetadataJob](https://github.com/AllenNeuralDynamics/aind-metadata-mapper/tree/release-v1.0.0#usage) is the primary tool used to assemble and validate metadata during upload of data assets. The job handles construction of the `data_description`, `subject`, and `procedures` as well as merging and validating `instrument` and `acquisition` metadata. It also runs a full validation step on all available metadata files to ensure cross-compatibility.

0 commit comments

Comments
 (0)