flattening IA (#42)

* flattening IA

* docs: fix some bad xrefs and adding missing extensions

---------

Co-authored-by: Dan Birman <danbirman@gmail.com>

Author: dyf

docs/source/acquire_upload.md

@@ -1,4 +1,4 @@
-# During acquisition
+# Acquire data
 
 During data acquisition you are responsible for running version-controlled acquisition software and ensuring your data files for each modality are organized according to standardized conventions.
 
@@ -8,7 +8,7 @@ Metadata generated during acquisition captures **what data** should appear in th
 
 ### Data organization conventions
 
-Raw data assets are required to be organized according to our [data organization conventions](../philosophy/data_organization.md).
+Raw data assets are required to be organized according to our [data organization conventions](../policies_practices/data_organization.md).
 
 #### Per-modality file standards
 
@@ -24,7 +24,7 @@ Rigs are responsible for generating the [acquisition.json](https://aind-data-sch
 
 If you can't generate your aind-data-schema formatted metadata on your rig, you can use what we call the “extractor/mapper” pattern. We refer to the code on the rig that extracts metadata from data files as the extractor. We prefer for you to maintain this code in [aind-metadata-extractor](https://github.com/AllenNeuralDynamics/aind-metadata-extractor/) but you can also maintain it yourself. The code that takes the extractor output and transforms it to aind-data-schema is called the mapper. Scientific computing will help develop the mapper as well as maintain it, you are responsible for your extractor. The key to the extractor/mapper pattern is the data contract that defines the extractor output. The data contract must be a pydantic model or JSON schema file and must live in the [aind_metadata_extractor.models](https://github.com/AllenNeuralDynamics/aind-metadata-extractor/tree/main/src/aind_metadata_extractor/models) module.
 
-On your rig you should output files that match the name of the corresponding mapper that will be run. So if your mapper is called fip you should write a `fip.json` file that validates against the fip extractor schema. The [GatherMetadataJob](upload.md#GatherMetadataJob) will automatically run your mapper. 
+On your rig you should output files that match the name of the corresponding mapper that will be run. So if your mapper is called fip you should write a `fip.json` file that validates against the fip extractor schema. The [GatherMetadataJob](upload_data.md#gathermetadatajob) will automatically run your mapper. 
 
 #### Multiple independent rigs
 

docs/source/acquire_upload/on_rig.md …cs/source/acquire_upload/acquire_data.mddocs/source/acquire_upload/on_rig.md renamed to docs/source/acquire_upload/acquire_data.md docs/source/acquire_upload/on_rig.md renamed to docs/source/acquire_upload/acquire_data.md

@@ -0,0 +1,27 @@
+---
+orphan: true
+---
+
+# Acquire, upload & process
+
+## I want to...
+
+[Prepare my metadata](prepare_before_acquisition.md) before acquisition.
+
+[Calibrate and test](calibration.md) an instrument.
+
+[Acquire data](acquire_data.md) on an instrument.
+
+[Upload data](upload_data.md) to the Cloud.
+
+[Process data](process_data.md) using our per-modality standard pipelines.
+
+```{toctree}
+:hidden:
+
+prepare_before_acquisition
+acquire_data
+calibration
+upload_data
+process_data
+```

docs/source/acquire_upload/index.md

@@ -309,7 +309,7 @@ The `instrument_id` for AIND should be the SIPE ID for an instrument. If an inst
 
 #### Multiple instruments
 
-Multiple `instrument.json` files can be provided when multiple separate instruments are used simultaneously to acquire a data asset. The combined instrument metadata stored with the associated data asset will have an `instrument_id` that is the combined names of the individual instruments, joined with the `'_'` character. See [metadata merging rules](upload.md#metadata-merging-rules) for information about how metadata files are merged during data upload.
+Multiple `instrument.json` files can be provided when multiple separate instruments are used simultaneously to acquire a data asset. The combined instrument metadata stored with the associated data asset will have an `instrument_id` that is the combined names of the individual instruments, joined with the `'_'` character. See [metadata merging rules](upload_data.md#merge-rules) for information about how metadata files are merged during data upload.
 
 #### Upload options
 
@@ -337,7 +337,7 @@ Users have two options for providing instrument metadata files:
 
    The data transfer service will then pull the instrument metadata from the database during upload. 
 
-Note that it is possible to combine these methods. For example, a user could pass the instrument JSON for the behavior instrument in the data directory (named something like `instrument_behavior.json`) and also specify a physiology rig by instrument ID in the `gather_preliminary_metadata` job type settings. The two instrument files would be merged by the data transfer service. See [metadata merging rules](upload.md#metadata-merging-rules).
+Note that it is possible to combine these methods. For example, a user could pass the instrument JSON for the behavior instrument in the data directory (named something like `instrument_behavior.json`) and also specify a physiology rig by instrument ID in the `gather_preliminary_metadata` job type settings. The two instrument files would be merged by the data transfer service. See [metadata merging rules](upload_data.md#merge-rules).
 
 Also note that we require all devices in the database to have a unique `instrument_id`. 
 

docs/source/acquire_upload/prepare_before_acquisition.md

@@ -0,0 +1,41 @@
+# Process data
+
+Scientific computing is currently re-organizing pipelines to be per-modality, rather than per-project.
+
+Pipeline development requirements are documented in [Pipeline development](../policies_practices/pipeline_development.md).
+
+## Per-modality physiology pipelines
+
+| Modality | Modalities | Pipeline repository |
+|---|---|---|
+| Barcoded anatomy resolved by sequencing | `barseq` |  |
+| Brightfield microscopy | `brightfield` |  |
+| Confocal microscopy | `confocal` |  |
+| Extracellular electrophysiology | `ecephys` | [aind-ephys-pipeline](https://github.com/AllenNeuralDynamics/aind-ephys-pipeline) |
+| Electron microscopy | `EM` |  |
+| Electromyography | `EMG` |  |
+| Fiber photometry | `fib` | [aind-fiber-photometry-harp-pipeline](https://github.com/AllenNeuralDynamics/aind-fiber-photometry-harp-pipeline) |
+| Fluorescence micro-optical sectioning tomography | `fMOST` |  |
+| Intracellular electrophysiology | `icephys` |  |
+| Intrinsic signal imaging | `ISI` | [isi_segmentation](https://github.com/AllenNeuralDynamics/isi_segmentation) |
+| Multiplexed analysis of projections by sequencing | `MAPseq` |  |
+| Multiplexed error-robust fluorescence in situ hybridization | `merfish` |  |
+| Magnetic resonance imaging | `MRI` |  |
+| Planar optical physiology | `pophys` | [aind-pophys-pipeline](https://github.com/AllenNeuralDynamics/aind-pophys-pipeline) |
+| Single cell RNA sequencing | `scRNAseq` |  |
+| Random access projection microscopy | `slap2` |  |
+| Selective plane illumination microscopy | `SPIM` |  |
+| Serial two-photon tomography | `STPT` |  |
+
+## Behavior pipelines
+
+| Behavior | Modalities | Pipeline repository |
+|---|---|---|
+| Patch foraging behavior | `behavior`, `behavior-videos` | [aind-vr-foraging-pipeline](https://github.com/AllenNeuralDynamics/aind-vr-foraging-pipeline) |
+| Camstim/Sync Behavior | `behavior`, `behavior-videos` | [aind-behavior-camstim-pipeline](https://github.com/AllenNeuralDynamics/aind-behavior-camstim-pipeline) |
+
+## Per-project pipelines
+
+| Project name | Modalities | Pipeline repository |
+|---|---|---|
+| Dynamic foraging | `behavior`, `behavior-videos`, `fib` | [aind-dynamic-foraging-pipeline](https://github.com/AllenNeuralDynamics/aind-dynamic-foraging-pipeline) |

docs/source/acquire_upload/process_data.md

@@ -1,4 +1,4 @@
-# Upload
+# Upload data
 
 Uploading data is done by using the [aind-data-transfer-service](http://aind-data-transfer-service/) ([docs](https://aind-data-transfer-service.readthedocs.io/en/latest/index.html)) which handles running containerized tasks for data copying, compression, metadata gathering, and final upload to S3 and Code Ocean.
 

docs/source/acquire_upload/processing.md

@@ -1,22 +1,4 @@
-# Scientific Computing at AIND
-
-## Core principles
-
-All of the teams in Scientific Computing adhere to a set of core principles about our software.
-
-### Code Review 
-
-At least one other software developer needs to approve a pull request in order for it to be merged. Please be courteous when providing feedback. The team lead can resolve any conflicts. 
-
-### Style
-
-We use `black`, `flake8`, and `interrogate` to enforce [PEP 8](https://peps.python.org/pep-0008/) standards with [docstrings](https://peps.python.org/pep-0257/) in [NUMPY](https://numpydoc.readthedocs.io/en/latest/format.html) format.
-
-### Versioning
-
-We use [semver](https://semver.org/) major.minor.patch versions, these are automatically incremented when you use the `aind-library-template`. Note that for major versions you need to put the exact string "BREAKING CHANGE" in the commit *comment* (not the title).
-
-You should set patch version floor `>=1.0.0` and major version ceiling `<2` for each internal dependency that you use. This is good practice for all dependencies.
+# Scientific Computing Teams
 
 ## Data Infrastructure
 
@@ -32,7 +14,7 @@ FastAPI service to run data compression and transfer jobs on the HPC
 
 **aind-metadata-service**
 
-REST service to retrieve metadata from AIND databases 
+REST service to retrieve metadata from AIND databases
 
 [link](http://aind-metadata-service/) | [readthedoc](http://aind-metadata-service/docs) | [repo](https://github.com/AllenNeuralDynamics/aind-metadata-service)
 
@@ -44,11 +26,10 @@ Library to interface with AIND databases
 
 **aind-data-asset-indexer**
 
-Index jobs for AIND metadata in AWS DocumentDB and S3 
+Index jobs for AIND metadata in AWS DocumentDB and S3
 
 [readthedoc](https://aind-data-asset-indexer.readthedocs.io/en/latest/) | [repo](https://github.com/AllenNeuralDynamics/aind-data-asset-indexer)
 
-
 ## Data & Outreach
 
 The Data & Outreach team maintains the data schema and associated downstream tools and is responsible for coordinating workshops and other outreach events.
@@ -61,7 +42,7 @@ Metadata schema for neuroscience
 
 **aind-metadata-mapper**
 
-Repository to help gather and map metadata from different sources 
+Repository to help gather and map metadata from different sources
 
 [readthedoc](https://aind-metadata-mapper.readthedocs.io/en/latest/) | [repo](https://github.com/AllenNeuralDynamics/aind-metadata-mapper)
 
@@ -75,15 +56,8 @@ Data book for the SWDB course
 
 ## Physiology & Behavior
 
-The Physiology & Behavior team maintains the pipelines that process each modality of data asset that we acquire in AIND. Details can be found on the [processing](../acquire_upload/processing.md)
+The Physiology & Behavior team maintains the pipelines that process each modality of data asset that we acquire in AIND. Details can be found in [Process data](../acquire_upload/process_data.md).
 
 ## Computer Vision
 
 [todo]
-
-## Resources for SWEs
-
-For research software engineers, [Good Research Code](https://goodresearch.dev/) is a good primer.
-
-[Data structure fundamentals](https://www.crackingthecodinginterview.com/)
-