Merge branch 'master' into extra_covariates

Author: vitkl

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,26 @@
+---
+name: Bug report
+about: Cell2location doesn’t do what it should? Please help us fix it!
+title: ''
+labels: bug
+assignees: ''
+---
+
+- [ ] I have confirmed this bug exists on the latest version of cell2location. See https://github.com/BayraktarLab/cell2location#installation
+- [ ] I follow the instructions from the [scvi-tools tutorial](https://cell2location.readthedocs.io/en/latest/notebooks/cell2location_tutorial.html).
+
+---
+
+**Note**: Please read [this guide](https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports) detailing how to provide the necessary information for us to reproduce your bug.
+
+
+### Minimal code sample (that we can run without your data, using public data)
+
+```python
+# Your code here
+```
+
+```pytb
+[Paste the error output produced by the above code here]
+```
+

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,13 @@
+blank_issues_enabled: false
+contact_links:
+  - name: scverse Discorse
+    url: https://discourse.scverse.org/c/ecosytem/cell2location/
+    about: Ask usage questions, how to solve your problems using cell2location and other scvi-tools packages.
+    
+  - name: Frequently asked questions
+    url: https://github.com/BayraktarLab/cell2location/issues?q=is%3Aissue+is%3Aopen+label%3AFAQ
+    about: Before asking a question please check this list (issue with FAQ tag).
+
+  - name: cell2location Community Discussions [deprecated]
+    url: https://discourse.scverse.org/c/ecosytem/cell2location/
+    about: Find previous answers/issues. For new questions please use the link above.

.github/ISSUE_TEMPLATE/enhancement-request.md

@@ -0,0 +1,10 @@
+---
+name: Enhancement request
+about: Anything you’d like to see in cell2location?
+title: ''
+labels: enhancement
+assignees: ''
+---
+
+<!-- Please describe your feature you would like to see below: -->
+...

.github/ISSUE_TEMPLATE/question.md

@@ -0,0 +1,43 @@
+---
+name: Usage Question
+about: Template for posting a question to scverse Discourse.
+title: ''
+labels: question
+assignees: ''
+---
+
+## Please use the template below to post a question to https://discourse.scverse.org/c/ecosytem/cell2location/. 
+
+### Problem
+
+<!-- Please describe your problem below: -->
+...
+
+- [ ] I follow the instructions from the [cell2location tutorial (using on scvi-tools)](https://cell2location.readthedocs.io/en/latest/notebooks/cell2location_tutorial.html).
+- [ ] I have adjusted required hyperparameters to my dataset and tissue `N_cells_per_location` and `detection_alpha`.
+- [ ] I have provided 10X reaction/inlet as `batch_key` for reference NB regression.
+- [ ] I have checked [scverse Discourse](https://discourse.scverse.org/c/ecosytem/cell2location/) and [old Cell2location Community Forum](https://github.com/BayraktarLab/cell2location/discussions), and did not find a solution.
+
+
+### Description of the data input and hyperparameters
+
+<!-- Please briefly describe your : -->
+...
+
+<!-- Please briefly describe your spatial data: -->
+...
+
+#### Single cell reference data: number of cells, number of cell types, number of genes
+
+<!-- Please add this info: -->
+...
+
+#### Single cell reference data: technology type (e.g. mix of 10X 3' and 5')
+
+<!-- Please add this info: -->
+...
+
+#### Spatial data: number of locations numbers, technology type (e.g. Visium, ISS, Nanostring WTA)
+
+<!-- Please add this info: -->
+...

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
     - repo: https://github.com/python/black
-      rev: 20.8b1
+      rev: '22.3.0'
       hooks:
           - id: black
     - repo: https://gitlab.com/pycqa/flake8

README.md

@@ -5,12 +5,17 @@
 ### Comprehensive mapping of tissue cell architecture via integrated single cell and spatial transcriptomics (cell2location model)
 
 [![Stars](https://img.shields.io/github/stars/BayraktarLab/cell2location?logo=GitHub&color=yellow)](https://github.com/BayraktarLab/cell2location/stargazers)
-![Build Status](https://github.com/BayraktarLab/cell2location/actions/workflows/test.yml/badge.svg)
+![Build Status](https://github.com/BayraktarLab/cell2location/actions/workflows/test.yml/badge.svg?event=push)
 [![Documentation Status](https://readthedocs.org/projects/cell2location/badge/?version=latest)](https://cell2location.readthedocs.io/en/stable/?badge=latest)
 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/BayraktarLab/cell2location/blob/master/docs/notebooks/cell2location_tutorial.ipynb)
-[![Docker image on quay.io](https://img.shields.io/badge/container-quay.io/vitkl/cell2location-brightgreen "Docker image on quay.io")](https://quay.io/vitkl/cell2location)
+[![Docker image on quay.io](https://img.shields.io/badge/container-quay.io/vitkl/cell2location-brightgreen "Docker image on quay.io")](https://quay.io/vitkl/cell2location) 
 
-Preprint: https://www.biorxiv.org/content/10.1101/2020.11.15.378125v1 
+If you use cell2location please cite our paper: 
+
+Kleshchevnikov, V., Shmatko, A., Dann, E. et al. Cell2location maps fine-grained cell types in spatial transcriptomics. Nat Biotechnol (2022). https://doi.org/10.1038/s41587-021-01139-4
+https://www.nature.com/articles/s41587-021-01139-4
+
+Please note that cell2locations requires 2 user-provided hyperparameters (N_cells_per_location and detection_alpha) - for detailed guidance on setting these hyperparameters and their impact see [the flow diagram and the note](https://github.com/BayraktarLab/cell2location/blob/master/docs/images/Note_on_selecting_hyperparameters.pdf). Many real datasets (especially human) show within-slide variability in RNA detection sensitivity - requiring you to try both recommended settings of the `detection_alpha` parameter: `detection_alpha=200` for low within-slide technical variability and `detection_alpha=20` for high within-slide technical variability.
 
 Cell2location is a principled Bayesian model that can resolve fine-grained cell types in spatial transcriptomic data and create comprehensive cellular maps of diverse tissues. Cell2location accounts for technical sources of variation and borrows statistical strength across locations, thereby enabling the integration of single cell and spatial transcriptomics with higher sensitivity and resolution than existing tools. This is achieved by estimating which combination of cell types in which cell abundance could have given the mRNA counts in the spatial data, while modelling technical effects (platform/technology effect, contaminating RNA, unexplained variance).
 
@@ -21,11 +26,9 @@ Overview of the spatial mapping approach and the workflow enabled by cell2locati
 
 ## Usage and Tutorials
 
-The tutorial covering the estimation of expresson signatures of reference cell types, spatial mapping with cell2location and the downstream analysis can be found here: https://cell2location.readthedocs.io/en/latest/
-
-You can also try cell2location on [Google Colab](https://colab.research.google.com/github/BayraktarLab/cell2location/blob/master/docs/notebooks/cell2location_tutorial.ipynb) on a smaller data subset containing somatosensory cortex.
+The tutorial covering the estimation of expresson signatures of reference cell types, spatial mapping with cell2location and the downstream analysis can be found here and tried on [Google Colab](https://colab.research.google.com/github/BayraktarLab/cell2location/blob/master/docs/notebooks/cell2location_tutorial.ipynb): https://cell2location.readthedocs.io/en/latest/
 
-Please report bugs via https://github.com/BayraktarLab/cell2location/issues and ask any usage questions in https://github.com/BayraktarLab/cell2location/discussions.
+Please report bugs via https://github.com/BayraktarLab/cell2location/issues and ask any usage questions about [cell2location](https://discourse.scverse.org/c/ecosytem/cell2location/42), [scvi-tools](https://discourse.scverse.org/c/help/scvi-tools/7) or [Visium data](https://discourse.scverse.org/c/general/visium/32) in scverse community discourse.
 
 Cell2location package is implemented in a general way (using https://pyro.ai/ and https://scvi-tools.org/) to support multiple related models - both for spatial mapping, estimating reference cell type signatures and downstream analysis.
 
@@ -58,10 +61,10 @@ bash Miniconda3-latest-Linux-x86_64.sh
 # use prefix /path/to/software/miniconda3
 ```
 
-Before installing cell2location and it's dependencies, it could be necessary to make sure that you are creating a fully isolated conda environment by telling python to NOT use user site for installing packages, ideally by adding this line to your `~/.bashrc` file , but this would also work during a terminal session:
+Before installing cell2location and it's dependencies, it could be necessary to make sure that you are creating a fully isolated conda environment by telling python to NOT use user site for installing packages by running this line before creating conda environment and every time before activatin conda environment in a new terminal session:
 
 ```bash
-export PYTHONNOUSERSITE="someletters"
+export PYTHONNOUSERSITE="literallyanyletters"
 ```
 
 
@@ -76,12 +79,147 @@ Cell2location architecture is designed to simplify extended versions of the mode
 We thank all paper authors for their contributions:
 Vitalii Kleshchevnikov, Artem Shmatko, Emma Dann, Alexander Aivazidis, Hamish W King, Tong Li, Artem Lomakin, Veronika Kedlian, Mika Sarkin Jain, Jun Sung Park, Lauma Ramona, Liz Tuck, Anna Arutyunyan, Roser Vento-Tormo, Moritz Gerstung, Louisa James, Oliver Stegle, Omer Ali Bayraktar
 
-We also thank Krzysztof Polanski, Luz Garcia Alonso, Carlos Talavera-Lopez, Ni Huang for feedback on the package, Martin Prete for dockerising cell2location and other software support.
+We also thank Pyro developers (Fritz Obermeyer, Martin Jankowiak), Krzysztof Polanski, Luz Garcia Alonso, Carlos Talavera-Lopez, Ni Huang for feedback on the package, Martin Prete for dockerising cell2location and other software support.
 
 ## FAQ
 
 See https://github.com/BayraktarLab/cell2location/discussions
 
 ## Future development and experimental features
+Future developments of cell2location are focused on 1) scalability to 100k-mln+ locations using amortised inference of cell abundance (same ideas as used in VAE), 2) extending cell2location to related spatial analysis tasks that require modification of the model (such as using cell type hierarchy information), and 3) incorporating features presented by more recently proposed methods (such as CAR spatial proximity modelling). We are also experimenting with Numpyro and JAX (https://github.com/vitkl/cell2location_numpyro).
+
+## Tips
+
+### Conda environment for A100 GPUs
+
+```bash
+export PYTHONNOUSERSITE="literallyanyletters"
+conda create -y -n test_scvi16_cuda113 python=3.9
+conda activate test_scvi16_cuda113
+conda install -y -c anaconda hdf5 pytables git
+pip install scvi-tools
+pip install git+https://github.com/BayraktarLab/cell2location.git#egg=cell2location[tutorials]
+pip3 install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 -f https://download.pytorch.org/whl/torch_stable.html
+conda activate test_scvi16_cuda113
+python -m ipykernel install --user --name=test_scvi16_cuda113 --display-name='Environment (test_scvi16_cuda113)'
+```
+
+### Issues with package version mismatches often originate from python user site rather than conda environment being used to install a subset of packages
+
+Before installing cell2location and it's dependencies, it could be necessary to make sure that you are creating a fully isolated conda environment by telling python to NOT use user site for installing packages by running this line before creating conda environment and every time before activatin conda environment in a new terminal session:
 
-We also provide an experimental numpyro translation of the model which has improved memory efficiency (allowing analysis of multiple Visium samples on Google Colab) and minor improvements in speed - https://github.com/vitkl/cell2location_numpyro. You can try it on Google Colab [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/vitkl/cell2location_numpyro/blob/main/docs/notebooks/cell2location_short_demo_colab.ipynb) - however note that both numpyro itself and cell2location_numpyro are in very active development. 
+```bash
+export PYTHONNOUSERSITE="literallyanyletters"
+```
+
+### Useful code for reading and combining multiple Visium sections
+
+Keeping info on distinct sections in a csv file (Google Sheet).
+
+```python
+sample_annot = pd.read_csv('./sample_annot.csv')
+
+from glob import glob
+sample_annot['path'] = pd.Series(
+    glob(f'{sp_data_folder}*'),
+    index=[sub('^.+WTSI_', '', sub('_GRCh38-2020-A$', '', i)) for i in glob(f'{sp_data_folder}*')]
+)[sample_annot['Sample_ID']].values
+import os
+sample_annot['file'] = [os.path.basename(i) for i in sample_annot['path']]
+
+sample_annot['Sample_ID'].unique()
+```
+
+Reading and concatenating samples.
+
+```python
+def read_and_qc(sample_name, file, path=sp_data_folder):
+    """
+    Read one Visium file and add minimum metadata and QC metrics to adata.obs
+    NOTE: var_names is ENSEMBL ID as it should be, you can always plot with sc.pl.scatter(gene_symbols='SYMBOL')
+    """
+    
+    adata = sc.read_visium(path + str(file) +'/',
+                           count_file='filtered_feature_bc_matrix.h5',
+                           load_images=True)
+    adata.obs['sample'] = sample_name
+    adata.var['SYMBOL'] = adata.var_names
+    adata.var.rename(columns={'gene_ids': 'ENSEMBL'}, inplace=True)
+    adata.var_names = adata.var['ENSEMBL']
+    adata.var.drop(columns='ENSEMBL', inplace=True)
+    
+    # just in case there are non-unique ENSEMBL IDs
+    adata.var_names_make_unique()
+
+    # Calculate QC metrics
+    sc.pp.calculate_qc_metrics(adata, inplace=True)
+    adata.var['mt'] = [gene.startswith('mt-') for gene in adata.var['SYMBOL']]
+    adata.obs['mt_frac'] = adata[:, adata.var['mt'].tolist()].X.sum(1).A.squeeze()/adata.obs['total_counts']
+    
+    # add sample name to obs names
+    adata.obs["sample"] = [str(i) for i in adata.obs['sample']]
+    adata.obs_names = 's' + adata.obs["sample"] \
+                          + '_' + adata.obs_names
+    adata.obs.index.name = 'spot_id'
+    
+    file = list(adata.uns['spatial'].keys())[0]
+    adata.uns['spatial'][sample_name] = adata.uns['spatial'][file].copy()
+    del adata.uns['spatial'][file]
+    print(adata.uns['spatial'].keys())
+    
+    return adata
+
+def read_all_and_qc(
+    sample_annot, Sample_ID_col, file_col, sp_data_folder, 
+    count_file='filtered_feature_bc_matrix.h5',
+):
+    """
+    Read and concatenate all Visium files.
+    """
+    # read first sample
+    adata = read_and_qc(
+        sample_annot[Sample_ID_col][0], sample_annot[file_col][0], 
+        path=sp_data_folder
+    ) 
+
+    # read the remaining samples
+    slides = {}
+    for i, s in enumerate(sample_annot[Sample_ID_col][1:]):
+        adata_1 = read_and_qc(s, sample_annot[file_col][i], path=sp_data_folder) 
+        slides[str(s)] = adata_1
+
+    adata_0 = adata.copy()
+
+    # combine individual samples
+    #adata = adata.concatenate(list(slides.values()), index_unique=None)
+    adata = adata.concatenate(
+        list(slides.values()),
+        batch_key="sample",
+        uns_merge="unique",
+        batch_categories=sample_annot[Sample_ID_col], 
+        index_unique=None
+    )
+
+    sample_annot.index = sample_annot[Sample_ID_col]
+    for c in sample_annot.columns:
+        sample_annot.loc[:, c] = sample_annot[c].astype(str)
+    adata.obs[sample_annot.columns] = sample_annot.reindex(index=adata.obs['sample']).values
+    
+    return adata
+    
+adata = read_all_and_qc(
+    sample_annot=sample_annot, 
+    Sample_ID_col='Sample_ID', 
+    file_col='file', 
+    sp_data_folder=sp_data_folder, 
+    count_file='filtered_feature_bc_matrix.h5',
+)
+
+adata_incl_nontissue = read_all_and_qc(
+    sample_annot=sample_annot, 
+    Sample_ID_col='Sample_ID', 
+    file_col='file', 
+    sp_data_folder=sp_data_folder, 
+    count_file='raw_feature_bc_matrix.h5',
+)
+```

cell2location/__init__.py

@@ -2,13 +2,11 @@
 from pyro.distributions.transforms import SoftplusTransform
 from torch.distributions import biject_to, transform_to
 
-from .run_c2l import run_cell2location
+from . import models
 from .run_colocation import run_colocation
-from .run_regression import run_regression
 
 __all__ = [
-    "run_cell2location",
-    "run_regression",
+    "models",
     "run_colocation",
 ]