docs: QC

Author: dbirman

docs/source/explore_analyze/quality_control.md

@@ -1 +1,11 @@
-# Quality control
+# Quality control
+
+All data assets generated at AIND should undergo automated (and sometimes manual) quality control before being used in analysis. To make this as efficient as possible we provide a standardized metadata schema for tracking quality control metrics as well as a convenient portal for reviewing QC metadata.
+
+## Preparing QC metadata
+
+Please see the documentation on [QualityControl](https://aind-data-schema.readthedocs.io/en/latest/quality_control.html) for a comprehensive overview of QC.
+
+## QC Portal
+
+Please see the [QC Portal](https://github.com/AllenNeuralDynamics/aind-qc-portal?tab=readme-ov-file) documentation for more information.

docs/source/philosophy/data_organization.md

@@ -66,23 +66,25 @@ A few points:
 
 Primary data assets are organized as follows: 
 
-- <asset name, as above>  
-  - data_description.json (administrative information, funders, licenses, projects, etc) 
-  - subject.json (species, sex, DOB, unique identifier, genotype, etc) 
-  - procedures.json (subject surgeries, tissue preparation, water restriction, training protocols, etc) 
-  - instrument.json (static hardware components) 
-  - acquisition.json (device settings that change acquisition-to-acquisition) 
-  - <modality-1>  
-    - <list of data files>  
-  - <modality-2>  
-    - <list of data files> 
-  - <modality-n> 
-    - <list of data files> 
-  - derivatives (processed data generated during acquisition) 
-    - <label> (e.g. MIP) 
-      - <list of files> 
-  - logs (log files generated by the instrument or rig) 
-    - <list of files> 
+```
+📦<asset name, as above>
+ ┣ 📜data_description.json (administrative information, funders, licenses, projects, etc)
+ ┣ 📜subject.json (species, sex, DOB, unique identifier, genotype, etc)
+ ┣ 📜procedures.json (subject surgeries, tissue preparation, water restriction, training protocols, etc)
+ ┣ 📜instrument.json (static hardware components)
+ ┣ 📜acquisition.json (device settings that change acquisition-to-acquisition)
+ ┣ 📂<modality-1>
+ ┃ ┗ 📜<list of data files>
+ ┣ 📂<modality-2>
+ ┃ ┗ 📜<list of data files>
+ ┣ 📂<modality-n>
+ ┃ ┗ 📜<list of data files>
+ ┣ 📂derivatives (processed data generated during acquisition)
+ ┃ ┗ 📂<label> (e.g. MIP)
+ ┃ ┃ ┗ 📜<list of files>
+ ┗ 📂logs (log files generated by the instrument or rig)
+ ┃ ┗ 📜<list of files>
+```
 
 Modality terms come from controlled vocabularies in aind-data-schema-models.
 
@@ -120,4 +122,88 @@ Example for exaSPIM data:
  ┗ 📂derivatives
  ┃ ┗ 📂MIP
  ┃ ┃ ┗ 📜<list of e.g. tiff files>
-```
+```
+
+## Derived data conventions
+
+Anything computed in a single run should be logically grouped in a folder. The folder should be named: 
+
+<primary-asset-name>_<process-label>_<process-date>_<process-time> 
+
+Examples: 
+
+- ANM457202_2022-07-11_22-11-32_processed_2022-08-11_22-11-32 
+- 595262_2022-02-21_15-18-07_processed_2022-08-11_22-11-32 
+
+Processed outputs are usually the result of a multi-stage pipeline handling a single data modality. Utilize a modality-specific <process-label>. Other common process labels include: 
+
+- “curation” - tags assigned to input data (e.g. merge/split/noise calls for ephys units) 
+- ... 
+
+Overlong names are difficult to read, so do not daisy-chain. The goal is to keep names as simple as possible while being readable, not to encode all metadata or the entire provenance chain. If various stages of processing are being performed manually over extended periods of time, anchor each derived asset on the primary data asset. 
+
+Folder organization is as follows: 
+
+```
+📦<asset name, as above>
+ ┣ 📜data_description.json
+ ┣ 📜processing.json (describes the code, input parameters, outputs)
+ ┣ 📜subject.json (copied from primary asset)
+ ┣ 📜procedures.json (copied from primary asset)
+ ┣ 📜instrument.json (copied from primary asset)
+ ┣ 📜acquisition.json (copied from primary asset)
+ ┣ 📂<process-label-1>
+ ┃ ┗ 📜<list of files>
+ ┣ 📂<process-label-2>
+ ┃ ┗ 📜<list of files>
+ ┗ 📂<process-label-n>
+ ┃ ┗ 📜<list of files>
+``` 
+
+## File name guidelines
+
+When naming files, we should: 
+
+- use terms from vocabularies defined in aind-data-schema, e.g. 
+  - modalities, institutions 
+  - behavior video file names 
+- use "yyyy-mm-dd" and "hh-mm-ss" in local time zone for dates and times 
+- separate tokens with underscores, and not include underscores in tokens, e.g. 
+  - Do this: EFIP_655568_2022-04-26_11-48-09 
+  - Not this: EFIP-655568-2022_04_26-11_48_09 
+- Do not include illegal filename characters in tokens
+
+## Human-in-the-loop Processing Pipelines
+
+During preliminary phases of processing pipeline development, it is common to defer downstream processing until upstream processing has been manually validated. This is particularly important for pipelines involving expensive processing steps that are sensitive to the quality of upstream results.  
+
+Pipelines that involve human interventions can be treated as regular (if slow-moving) pipelines. As researchers inspect and validate individual processing steps, they construct a processed data asset incrementally, one subfolder at a time.  
+
+Guidelines for constructing a processed data asset with a human in the loop: 
+
+1. It is okay to overwrite the results of the current processing step in-place. 
+2. Always document provenance via an intermediate processing.json within the process subfolder 
+3. Do not revisit previous processing steps. If necessary, delete the asset and start a new one. 
+4. Once complete, generate integrated top-level JSON metadata (processing.json, quality_control.json) 
+
+## FAQ
+
+**My data files already contain some of this metadata. Why store this in additional JSON files?** 
+
+How acquisition formats represent metadata evolves over time and often does not capture everything we need to know to interpret data. These JSON files represent our ground truth viewpoint on what is essential to know about our data in a single location. 
+
+Additionally, JSON files are trivially both human- and machine-readable. They are viewable on any system without additional software to be installed (a text editor is fine). They are easy to parse from code without any heavy dependencies (IGOR, H5PY, pynwb, etc).  
+
+**What are "Institution" and "Group" doing in data_description.json?** 
+
+In the future we may need to tag cloud resources based on the originating group, which may or may not be in AIND, in order to track usage and spending.
+
+**Why are we replicating metadata that we are also tracking in PowerPlatform/LIMS/SLIMS/etc?** 
+
+Database systems such as these are very important for reliable acquisition, however they are also barriers to external interpretability and reproducibility. They have complex schema with extraneous information that make them difficult to interpret. They have query languages (e.g. SQL) that require training to use properly. Information becomes distributed across different locations. They may have security policies that make them difficult to share with the public.  
+
+Files, particularly in cloud storage, are reliable and more persistent. By storing metadata essential to interpreting an acquisition session alongside the acquisition in a human-and-machine-readable format, there will always be an interpretable record of what happened even if e.g. the database stops working. 
+
+**What happened to the "experiment type" and "platform" asset labels?**
+
+Formerly we used a short label called “experiment type” in asset names.This concept was confusing because it was difficult to distinguish from a “modality”. Then we switched to “platform” and introduced a controlled vocabulary, but people did not understand the term and so defaulted to a “primary modality,” which was not helpful. Most of our data contains multiple modalities. A recording session may contain trained behavior event data (e.g. lick times), behavior videos (e.g. face camera), neuropixels recordings, and fiber photometry recordings.