Milo neighborhood groups as in MiloR

[MaximilianNuber]

Description of feature

Dear all,

I would like to ask for a feature in the Milo submodule: Grouping Neighborhoods, as implemented in MiloR (https://github.com/MarioniLab/miloR/blob/master/R/groupNhoods.R).
@emdann suggested that I request the feature for the pertpy implementation (see emdann/milopy#49).

As I am using the feature already to analyse my data I have code suggestions, should you be interested.
I adjusted the code and example to work with pertpy-Milo using the tutorial data, compare the neighborhood grouping results to the MiloR implementation, and included a suggestion for neighborhood group markers based on the pertpy differential expression module.
I would be very thankful for any help to correct/verify my code.

Best regards,
max

Marker genes from find_nhood_groups_markers for neighborhood groups 5 vs 4 in "Python Neighborhood groups":

MiloR neighborhood groups:

Neighborhood group functions:

def find_nhood_groups(
    mdata, 
    SpatialFDR_threshold = 0.1,
    merge_discord = False,
    max_lfc_delta=None,
    overlap=1,
    subset_nhoods: None | str = None,
):
    """Get igraph graph from adjacency matrix. Adjusted from scanpy louvain clustering."""
    nhs = mdata["rna"].obsm["nhoods"].copy()
    nhood_adj = mdata["milo"].varp["nhood_connectivities"].copy()
    da_res = mdata["milo"].var.copy()
    is_da = np.asarray(da_res.SpatialFDR <= SpatialFDR_threshold)

    if subset_nhoods is not None:
        print(da_res.shape)
        mask_da_res = (
            da_res
            .query(subset_nhoods)
            .copy()
        )
        mask = np.asarray([True if x in mask_da_res.index else False for x in da_res.index])
        da_res = da_res.loc[mask, :].copy()
        print(da_res.shape)
        # mask = da_res.index.values.astype(bool)
        print(mask.shape)
        nhs = nhs[:, mask].copy()
        print(mask.shape)
        print(nhood_adj.shape)
        nhood_adj = nhood_adj[:, mask]
        nhood_adj = nhood_adj[mask, :].copy()
        print(nhood_adj.shape)
        # is_da = np.asarray(da_res.SpatialFDR <= SpatialFDR_threshold)
        is_da = is_da[mask].copy()
        # print(is_da.shape)

    # da_res.loc[is_da, 'logFC'].values @ (da_res.loc[is_da, 'logFC']).T.values
    # print(da_res.loc[is_da, 'logFC'])

    if merge_discord is False:
        # discord_sign = np.sign(da_res.loc[is_da, 'logFC'].values @ (da_res.loc[is_da, 'logFC'])) < 0
        # Assuming da.res is a pandas DataFrame and is.da is a boolean array
        discord_sign = np.sign(da_res.loc[is_da, 'logFC'].values @ da_res.loc[is_da, 'logFC'].values.T) < 0
        # print(discord_sign.shape)
        # nhood_adj[is_da, is_da][discord_sign] <- 0
        # print(nhood_adj.shape)
        nhood_adj[(is_da, is_da)][discord_sign] = 0

    if overlap > 1:
        nhood_adj[nhood_adj < overlap] = 0

    if max_lfc_delta is not None:  
        lfc_diff = np.array([da_res['logFC'].values - x for x in da_res['logFC'].values])
        nhood_adj[np.abs(lfc_diff) > max_lfc_delta] = 0

    # binarise
    # nhood_adj = np.asarray((nhood_adj > 0) + 0)
    nhood_adj = (nhood_adj > 0).astype(int)

    g = sc._utils.get_igraph_from_adjacency(nhood_adj, directed = False)

    weights = None
    part = g.community_multilevel(weights=weights)

    groups = np.array(part.membership)
    groups = groups.astype(int).astype(str)

    if subset_nhoods is not None:
        mdata["milo"].var["nhood_groups"] = np.nan
        mdata["milo"].var.loc[mask, "nhood_groups"] = groups
        return None
    mdata["milo"].var["nhood_groups"] = groups

Nhood group markers:

from typing import Sequence
from lamin_utils import logger

class Limma(pt.tl._differential_gene_expression._base.LinearModelBase):

    def _check_counts(self):
        # check_is_integer_matrix(self.data)
        pass
        
    def fit(self, voom = False, **kwargs):

        """Fit model on normalized data using limma.

        Note: this creates its own AnnData object for downstream.

        Args:
            **kwargs: Keyword arguments specific to glmQLFit()
        """

        if voom:
            print("voom selected")
            self.voom_fit(**kwargs)
            return None
        
        from scipy.sparse import issparse
        # For running in notebook
        # pandas2ri.activate()
        # rpy2.robjects.numpy2ri.activate()
        try:
            import rpy2.robjects.numpy2ri
            import rpy2.robjects.pandas2ri
            from rpy2 import robjects as ro
            from rpy2.robjects import numpy2ri, pandas2ri
            from rpy2.robjects.conversion import localconverter
            from rpy2.robjects.packages import importr

            # pandas2ri.activate()
            # rpy2.robjects.numpy2ri.activate()

        except ImportError:
            raise ImportError("edger requires rpy2 to be installed.") from None

        try:
            edger = importr("edgeR")
            limma = importr("limma")
        except ImportError as e:
            raise ImportError(
                "edgeR requires a valid R installation with the following packages:\n"
                "edgeR, BiocParallel, RhpcBLASctl"
            ) from e

        # Convert dataframe
        with localconverter(ro.default_converter + numpy2ri.converter):
            expr = self.adata.X if self.layer is None else self.adata.layers[self.layer]
            if issparse(expr):
                expr = expr.T.toarray()
            else:
                expr = expr.T

        
        expr_r = ro.conversion.py2rpy(pd.DataFrame(expr, index=self.adata.var_names, columns=self.adata.obs_names))

        r_obs = ro.conversion.py2rpy(self.adata.obs)
        # rbase = importr("base")

        with localconverter(default_converter + pandas2ri.converter):
            r_design = ro.conversion.py2rpy(self.design)

        self.fit = limma.lmFit(expr_r, r_design)

        # self.fit = limma.eBayes(fit)

    ### for the sake of completeness
    def voom_fit(self, voom_kwargs = {}, **lmFit_kwargs):

        """Fit model on count data using limma-voom.

        Note: this creates its own AnnData object for downstream.

        Args:
            **kwargs: Keyword arguments specific to glmQLFit()
        """

        print("Fitting limma with voom")
        
        from scipy.sparse import issparse
        # For running in notebook
        # pandas2ri.activate()
        # rpy2.robjects.numpy2ri.activate()
        try:
            import rpy2.robjects.numpy2ri
            import rpy2.robjects.pandas2ri
            from rpy2 import robjects as ro
            from rpy2.robjects import numpy2ri, pandas2ri
            from rpy2.robjects.conversion import localconverter
            from rpy2.robjects.packages import importr

            # pandas2ri.activate()
            # rpy2.robjects.numpy2ri.activate()

        except ImportError:
            raise ImportError("edger requires rpy2 to be installed.") from None

        try:
            edger = importr("edgeR")
            limma = importr("limma")
        except ImportError as e:
            raise ImportError(
                "!!limma requires a valid R installation with the following packages:\n"
                "edgeR, limma, BiocParallel, RhpcBLASctl"
            ) from e

        # Convert dataframe
        with localconverter(ro.default_converter + numpy2ri.converter):
            expr = self.adata.X if self.layer is None else self.adata.layers[self.layer]
            if issparse(expr):
                expr = expr.T.toarray()
            else:
                expr = expr.T

        
        expr_r = ro.conversion.py2rpy(pd.DataFrame(expr, index=self.adata.var_names, columns=self.adata.obs_names))

        r_obs = ro.conversion.py2rpy(self.adata.obs)
        # rbase = importr("base")

        with localconverter(default_converter + pandas2ri.converter):
            r_design = ro.conversion.py2rpy(self.design)

        v = limma.voom(expr_r, r_design, **voom_kwargs)

        self.fit = limma.lmFit(v, r_design, **lmFit_kwargs)
            
    def _test_single_contrast(self, contrast, trend = False, robust = True, **kwargs):
        
        with localconverter(default_converter + pandas2ri.converter):
            r_design = ro.conversion.py2rpy(self.design)

        r_string = '''
        FUN = function(fit, contrast, design) {
            # contrast_matrix <- makeContrasts(contrast, levels=design)
            # print(contrast_matrix)
            contrast <- as.numeric(contrast)
            print(class(contrast))
            
            fit2 = limma::contrasts.fit(fit, contrast)
            print("check")
            efit <- limma::eBayes(fit2)
            print("check")
            # print(coef(efit))
            res <- limma::topTable(efit, coef = 1, n = Inf)
        }
        '''

        r_pkg = STAP(r_string, "r_pkg")

        res = r_pkg.FUN(self.fit, contrast, r_design)

        with localconverter(default_converter + pandas2ri.converter):
            de_res = pandas2ri.rpy2py(res)

        de_res.index.name = "variable"
        de_res = de_res.reset_index()

        de_res =  de_res.rename(columns={"P.Value": "p_value", "logFC": "log_fc", "adj.P.Val": "adj_p_value"})

        return de_res

def get_cells_in_nhoods(mdata, nhood_ids):
    '''
    Get cells in neighbourhoods of interest '''
    in_nhoods = np.array(mdata["rna"].obsm['nhoods'][:,nhood_ids.astype('int')].sum(1))
    ad = mdata["rna"][in_nhoods.astype(bool), :].copy()
    return ad
def _get_groups(
    mdata, 
    sample_col,
    nhood_group_obs = "nhood_groups",
    confounders_obs = [],
    pseudobulk = True,
    func = "sum",
    layer = "counts"
):

    layer = None if layer == "X" else layer
    
    ad_list = []
    groups = mdata["milo"].var[nhood_group_obs].dropna().unique()
    print(groups)

    mdata["milo"].var["_my_groups"] = mdata["milo"].var[nhood_group_obs]

    for g in groups:
        nhood_ids = (
            mdata["milo"].var
            .query('_my_groups == @g')
            .index.to_list()
        )
    
        nhood_ids = np.asarray(nhood_ids)
        cur_ad = get_cells_in_nhoods(mdata, nhood_ids)
        cur_ad.obs[nhood_group_obs] = g
        
        if pseudobulk:
            cur_ad = sc.get.aggregate(cur_ad, by = [sample_col, nhood_group_obs] + confounders_obs, func = func,
                                     layer=layer)
        
        ad_list.append(cur_ad)
    
    ad = sc.concat(ad_list)

    return ad

def find_nhood_group_markers(
    mdata, 
    sample_col = "patient_id",
    nhood_group_obs = "nhood_groups",
    design = "~0+nhood_groups",
    baseline = "non_enriched",
    group_to_compare = "enriched",
    confounders_obs = [],
    pseudobulk = True, 
    func = "mean",
    layer = "X",
    pb_model = Limma,
    return_model = False,
    **kwargs
):
    ad = _get_groups(
        mdata, 
        sample_col = sample_col,
        nhood_group_obs = nhood_group_obs,
        confounders_obs = confounders_obs,
        pseudobulk = pseudobulk, 
        func = func,
        layer = layer
    )

    ad.X = ad.layers[func]

    mod = pb_model(ad, design = design)
    mod.fit(**kwargs)
    res_df = mod._test_single_contrast(
        mod.contrast(nhood_group_obs, baseline = baseline, group_to_compare = group_to_compare)
    )
    if return_model:
        return res_df, mod

    return res_df

Code to reproduce the example plots:

import os
os.environ["R_HOME"] = r_env_path
import rpy2.rinterface_lib.callbacks
# import anndata2ri
import logging

from rpy2.robjects import pandas2ri
from rpy2.robjects import numpy2ri
from rpy2.robjects import r
import rpy2.robjects as ro
from rpy2.robjects.packages import importr
from rpy2.robjects import default_converter
from rpy2.robjects.conversion import localconverter

from rpy2.robjects.packages import STAP

# sc.settings.verbosity = 0
rpy2.rinterface_lib.callbacks.logger.setLevel(logging.ERROR)
%load_ext rpy2.ipython

import numpy as np
import scanpy as sc
import seaborn as sns
# from scipy.stats import median_abs_deviation
import pandas as pd
import numpy as np
import os
import matplotlib.pyplot as plt
import matplotlib as mpl
import pertpy as pt

adata = pt.data.stephenson_2021_subsampled()
adata.obs["COVID_severity"] = adata.obs["Status_on_day_collection_summary"].copy()  # short name
adata.obs[["patient_id", "COVID_severity"]].drop_duplicates().value_counts("COVID_severity")

sc.pl.umap(adata, color=["author_cell_type"], legend_loc="on data")
sc.pl.umap(adata, color=["Site", "Status"], wspace=0.3, size=8)

## Exclude LPS samples
adata = adata[adata.obs["Status"] != "LPS"].copy()

## Initialize object for Milo analysis
milo = pt.tl.Milo()
mdata = milo.load(adata)

from sklearn_ann.kneighbors.annoy import AnnoyTransformer
transformer = AnnoyTransformer(n_neighbors=150)
sc.pp.neighbors(mdata["rna"], use_rep="X_scVI", transformer = transformer)

milo.make_nhoods(mdata["rna"], prop=0.1)

nhood_size = np.array(mdata["rna"].obsm["nhoods"].sum(0)).ravel()
plt.hist(nhood_size, bins=100)
plt.xlabel("# cells in nhood")
plt.ylabel("# nhoods");

milo.count_nhoods(mdata, sample_col="patient_id")

# Reorder categories
# (by default, the last category is taken as the condition of interest)
mdata["rna"].obs["Status"] = mdata["rna"].obs["Status"].cat.reorder_categories(["Healthy", "Covid"])
milo.da_nhoods(mdata, design="~Status")

old_figsize = plt.rcParams["figure.figsize"]
plt.rcParams["figure.figsize"] = [10, 5]
plt.subplot(1, 2, 1)
plt.hist(mdata["milo"].var.PValue, bins=50)
plt.xlabel("P-Vals")
plt.subplot(1, 2, 2)
plt.plot(mdata["milo"].var.logFC, -np.log10(mdata["milo"].var.SpatialFDR), ".")
plt.xlabel("log-Fold Change")
plt.ylabel("- log10(Spatial FDR)")
plt.tight_layout()
plt.rcParams["figure.figsize"] = old_figsize

milo.build_nhood_graph(mdata)

plt.rcParams["figure.figsize"] = [7, 7]
milo.plot_nhood_graph(
    mdata,
    alpha=0.1,  ## SpatialFDR level (1%)
    min_size=1,  ## Size of smallest dot
)

milo.annotate_nhoods(mdata, anno_col="author_cell_type")
plt.hist(mdata["milo"].var["nhood_annotation_frac"], bins=30)
plt.xlabel("celltype fraction")

mdata["milo"].var["nhood_annotation"] = mdata["milo"].var["nhood_annotation"].astype(str)
mdata["milo"].var.loc[mdata["milo"].var["nhood_annotation_frac"] < 0.6, "nhood_annotation"] = "Mixed"

milo.plot_da_beeswarm(mdata, alpha=0.1)

find_nhood_groups(mdata, # subset_nhoods = "nhood_annotation == 'CD83_CD14_mono'"
                 )


### Nhood groups beeswarm plot

milo.plot_da_beeswarm(
    mdata,
    anno_col = "nhood_groups",
)

### Python neighbor groups plot
from legendkit import cat_legend, size_legend, vstack
plt_df = pd.DataFrame(mdata["milo"].varm["X_milo_graph"],
                      columns = ["UMAP_"+str(i+1) for i in range(2)]
                     )

plt_df["nhood_groups"] = mdata["milo"].var["nhood_groups"].values
plt_df["nhood_groups"] = plt_df["nhood_groups"].astype(str)

plt_df["size"] = mdata["milo"].var["Nhood_size"].values
plt_df = plt_df.sort_values("nhood_groups")

label_df = label_df = (
    plt_df
    .groupby("nhood_groups")
    .median()
)

from legendkit import size_legend, cat_legend, vstack


# groups = plt_df["nhood_groups"].unique()
groups = []
[groups.append(n) for n in plt_df["nhood_groups"] if n not in groups]
print(groups)

palette = [color for color in sns.color_palette("Paired", 12) for _ in range(len(groups))]
scale_factor = 0.2
sns.scatterplot(
    x=plt_df["UMAP_1"],
    y=plt_df["UMAP_2"],
    size = plt_df["size"].values * scale_factor, #.apply(lambda x: x*0.2),
    hue = plt_df["nhood_groups"],#.astype("category").cat.codes,
    legend = False,
    linewidth=0,
    palette = sns.color_palette("Paired", 12)
)
sns.despine()
slegend = size_legend(mdata["milo"].var["Nhood_size"].values * scale_factor, loc = "out right center",
           title = "Nhood size", func = lambda x: x*(1/scale_factor))

color = dict(
    colors = sns.color_palette("Paired", 12),
    labels = groups
)

ax = plt.gca()
ax.set_title("Python Neighborhood groups", fontsize = 18)
handles, labels = ax.get_legend_handles_labels()

clegend = cat_legend(**color, title = "Nhood group", handle = "circle")
vstack([clegend, slegend], # title="Stack Vertically",
       loc="out right center", spacing=10, frameon=False, ax=ax)

for _group, row in label_df.iterrows():
    plt.text(row["UMAP_1"], row["UMAP_2"], _group, fontsize = 15)

plt.show()
###############

import anndata2ri
mdata = mdata.copy()
adata = pt.data.stephenson_2021_subsampled()
adata = adata[adata.obs["Status"] != "LPS"].copy()

da_res = mdata["milo"].var.copy()
da_res = (
    da_res.drop("nhood_groups", axis = 1)
)

with localconverter(anndata2ri.converter):
    r.assign("sce", adata)
    r.assign("nhoods", mdata["rna"].obsm["nhoods"])
    r.assign("nhoodCounts", mdata["milo"].X.T)
    r.assign("nhood_adjacency", mdata["milo"].varp["nhood_connectivities"])
    r.assign("da_results", da_res)
    r.assign("nhood_graph", mdata["milo"].varm["X_milo_graph"])

%%R -o da_results
suppressMessages(library(SingleCellExperiment))
suppressMessages(library(scran))
suppressMessages(library(miloR))
suppressMessages(library(Matrix))

milo <- Milo(sce)
nhoodAdjacency(milo) <- nhood_adjacency
nhoodCounts(milo) <- nhoodCounts
nhoods(milo) <- as(nhoods, "CsparseMatrix")
nhoodGraph(milo) <- nhood_graph

colnames(nhoodAdjacency(milo)) <- da_results$index_cell
rownames(nhoodAdjacency(milo)) <- da_results$index_cell

da_results <- groupNhoods(milo, da_results)

### MiloR nhood groups plot

mdata["milo"].var["nhood_groups_miloR"] = da_results["NhoodGroup"]

from legendkit import cat_legend, size_legend, vstack
plt_df = pd.DataFrame(mdata["milo"].varm["X_milo_graph"],
                      columns = ["UMAP_"+str(i+1) for i in range(2)]
                     )

plt_df["nhood_groups"] = mdata["milo"].var["nhood_groups_miloR"].values
plt_df.loc[plt_df.nhood_groups.isnull(), "nhood_groups"] = np.nan
plt_df["nhood_groups"] = plt_df["nhood_groups"].astype(str)

plt_df["size"] = mdata["milo"].var["Nhood_size"].values
plt_df = plt_df.sort_values("nhood_groups", na_position = "first")

label_df = label_df = (
    plt_df
    .groupby("nhood_groups")
    .median()
)

from legendkit import size_legend, cat_legend, vstack


# groups = plt_df["nhood_groups"].unique()
groups = []
[groups.append(n) for n in plt_df["nhood_groups"] if n not in groups]
print(groups)

palette = [color for color in sns.color_palette("Paired", 12) for _ in range(len(groups))]
scale_factor = 0.2
sns.scatterplot(
    x=plt_df["UMAP_1"],
    y=plt_df["UMAP_2"],
    size = plt_df["size"].values * scale_factor, #.apply(lambda x: x*0.2),
    hue = plt_df["nhood_groups"],#.astype("category").cat.codes,
    legend = False,
    linewidth=0,
    palette = sns.color_palette("Paired", 12)
)
sns.despine()
slegend = size_legend(mdata["milo"].var["Nhood_size"].values * scale_factor, loc = "out right center",
           title = "Nhood size", func = lambda x: x*(1/scale_factor))

color = dict(
    colors = sns.color_palette("Paired", 12),
    labels = groups
)

ax = plt.gca()
ax.set_title("MiloR Neighborhood groups", fontsize = 18)
handles, labels = ax.get_legend_handles_labels()

clegend = cat_legend(**color, title = "Nhood group", handle = "circle")
vstack([clegend, slegend], # title="Stack Vertically",
       loc="out right center", spacing=10, frameon=False, ax=ax)

for _group, row in label_df.iterrows():
    plt.text(row["UMAP_1"], row["UMAP_2"], _group, fontsize = 15)

plt.show()

mdata["milo"].var.loc[~mdata["milo"].var["nhood_groups"].isin(["4", "5"]), "nhood_groups"] = np.nan

### uses the above Limma class by default, but can be switched out for pt.tl.EdgeR when a counts layer exists.
res_df, mod = find_nhood_group_markers(mdata, 
                        sample_col = "patient_id",
                        baseline = "4", group_to_compare = "5",
                        return_model = True,
                                      voom = True)

### Volcano Plot:
mod.plot_volcano(res_df, log2fc_thresh = 0.1,legend_pos = (1.3, 1))

[Lilly-May]

Hi @MaximilianNuber, thanks for looking into this! Would you be up for opening a Pull Request with your suggested code modifications? That would simplify reviewing it.

[Zethson]

Dear @MaximilianNuber would you be willing to have a go at it?

[MaximilianNuber]

Dear @Lilly-May , dear @Zethson,

My apologies for my long absence.

I forked pertpy and started to implement and test this (it should be working already).
So far, I added group_nhoods in _milo.py as a function, but will move it into the Milo class later on.

The tests focus on the processing of the adjacency matrix before the actual clustering being equivalent to R.

I would like to ask about going forward with this.

First, I would like to add differential expression between neighborhood groups, which can be applied to the group_nhoods or user defined groups. I would like to keep this as close as possible to
Dann, E., Cujba, AM., Oliver, A.J. et al. Precise identification of cell states altered in disease using healthy single-cell references. Nat Genet 55, 1998–2008 (2023). https://doi.org/10.1038/s41588-023-01523-7

Therefore the question, can I add a glmGamPoi class to the differential expression module to use this, or rather call it directly without relying on the differential expression module?

Second, for Python to R conversions (which I would need for the differential expression only) I made some convenience wrappers for lazy imports and conversions, which I use in all of my code. I put them in the branch on my fork under pertpy.utils...
I don´t intend to keep this there, but would like to ask if I could include this as a dependency if I put it into a mini package? Or should this be included somewhere else? Or should I rely on rpy2 directly?
If unclear, I would remove this entirely of course.

Thank you again for your patience.

Best,
Max

[Zethson]

@MaximilianNuber no worries! I'm glad to hear that you're still interested in contributing this.

Therefore the question, can I add a glmGamPoi class to the differential expression module to use this, or rather call it directly without relying on the differential expression module?

Am I correct in assuming that glmGamPoi currently does not exist in Python. Therefore, we'd need to call a R package? Ideally, we'd have an implementation that uses pydeseq2 under the hood if possible.
It is totally fine to add this class to the DE module. Ideally, we just wouldn't require more dependencies.

Second, for Python to R conversions (which I would need for the differential expression only) I made some convenience wrappers for lazy imports and conversions, which I use in all of my code. I put them in the branch on my fork under pertpy.utils...
I don´t intend to keep this there, but would like to ask if I could include this as a dependency if I put it into a mini package? Or should this be included somewhere else? Or should I rely on rpy2 directly?
If unclear, I would remove this entirely of course.

Ideally, your version would also support a pure Python version using statsmodels or pydeseq2. I recently added a pydeseq2 flavor to Milo and since then, R isn't even required to run Milo. Now if this is not possible in your case and you need R dependencies, then I'd like to kindly ask you to use the existing structure of how we deal with R dependencies in Milo. We're working with rpy2. Let me know if this is unclear.

Generally, feel free to work on this incrementially with a draft PR. We'll later also add it to our tutorials etc but let's do it step by step.

Thank you!

[MaximilianNuber]

Dear @Zethson ,

I am slowly getting somewhere, and while I add and run the last tests I would like to make sure everything is up to your guidelines and goals. Most important: which test dataset containing raw counts to use in test_milo.py?
A description is below.

TL;DR:

• Added 3 functions Milo().group_nhoods(...), Milo().annotate_cells_from_nhoods(...), Milo().find_nhood_group_markers(...).
• PyDeseq2 is default for differential expression, optionally edgeR.
• For testing differential expression between neighborhood groups in test_milo.py I would need to load a dataset with counts. From what I see, neither the Milo tutorial dataset nor sc.datasets.pbmc3k() have raw counts. Any preferred dataset? I have looked through some in pertpy and scanpy and all I saw were missing raw counts.
• I think Milo().annotate_cells_from_nhoods(...) solves Milo: annotate adata from mdata #751

If there is anything to add, leave out, or change, please let me know.

(This is already the description for the draft PR. Will submit after the last test is added with an appropriate dataset containing counts)

There are 3 new user-facing functions:

1. Milo().group_nhoods(...)
2. Milo().annotate_cells_from_nhoods(...)
3. Milo().find_nhood_group_markers(...)

milo.group_nhoods takes the neighborhood connectivities sparse matrix already calculated by milo, removes edges based on milo.da_nhoods SpatialFDR and the number of overlapping cells in neighborhoods, and then performs graph_object.community_multilevel.
I added tests to confirm that the preparation of the neighborhood connectivities in mdata["milo"].varp["nhood_connectivities"] before the clustering is equivalent to the R version. The R code in pertpy.tests.tools.test_milo.py is taken directly from the miloR GitHub repository to test quivalence of the R and Python versions (https://github.com/MarioniLab/miloR/blob/master/R/groupNhoods.R).

For finding neighborhood group marker genes, the neighborhood grouping can be based on milo.group_nhoods or specified manually for custom comparisons, as done in both miloR and Dann et al. 2023.

In both miloR and Dann et al. 2023, the findNhoodGroupMarkers, or Python equivalent respectively, combine the transfer of neighborhood group labels from the neighborhood "milo" object (mdata["milo"]) to the single-cell object (mdata["rna"]) with the differential expression analysis between the specified neighborhood groups.
I deliberately separated the annotation of cells from neighborhoods, as it enables custom differential expression analyses, the neighborhood grouping info becomes a column in mdata["rna"].obs.
The annotation of cells from nhoods needs to solve the problem of cells being members more than one neighborhood group. miloR solves this by assigning the neighborhood group label of the last neighborhood group in the assignment loop to each cell. Dann et al. 2023 solved this by excluding overlapping cells. I implemented both versions, usable through the mode-argument as milo.annotate_cells_from_nhoods(mdata, mode = "last_wins") and milo.annotate_cells_from_nhoods(mdata, mode = "exclude_overlaps").

Then I implemented the rest of the differential expression between neighborhood groups in milo.find_nhood_group_markers. After some checks the pseudobulk is performed using sc.get.aggregate based on the sample column, the neighborhood groups column, and including any desired covariates. As milo.annotate_cells_from_nhoods introduces NAs, and NAs are necessary in specifying custom neighborhood groups, any NAs are filtered before pseudobulking.

Dann et al. 2023 uses scuttle::logNormCounts and scran::modelGeneVar for filtering to a specified number of highly variable genes. I implemented this, which introduces new R dependencies scuttle, scran, and SingleCellExperiment, all imported through Milo()._try_import_bioc_library for respective errors if the R package is not installed.

However, the default for filtering genes is scanpy, using scanpy.pp.highly_variable_genes as a Python version of Dann et al. 2023.
Additionally I added edger::filterByExpr as an option for gene filtering, as edgeR might already be installed if solver = "edger" was used in milo.da_nhoods.

The default for differential expression in milo.find_nhood_group_markers is PyDeseq2, optionally edgeR, again because it is likely to be installed if if solver = "edger" was used in milo.da_nhoods.
The function takes the optional arguments baseline and group_to_compare, for targeted comparison of two neighborhood groups. If both arguments are None, a one vs. all comparison is performed for every neighborhood group, as implemented in miloR.
If desired, I would like to add glmGamPoi at a later time, as it was used in Dann et al. 2023.

As the Milo tutorial data contains on normalized counts, I originally implemented both Limma and pt.tl.Statsmodels as backends for find_nhood_group_markers. However, two issues came up.
The log-fold-changes were larger by a magnitude of about 200 as compared to PyDeseq2 and EdgeR (taking a dataset with counts and using sc.pp.normalize_total(adata, target_sum = 1e4)). I presume this occurs due to the scaling factor target_sum, where no scaling factor is used in scuttle::logNormCounts, which is performed before using limma for finding neighborhood group markers in miloR. But this is a question I would like to clarify further, as I am not certain right now.

The second issue was a circular import when importing the Statsmodels class from pertpy differential expression. For this reason I took out the function using statsmodels, and left in Limma for now.
This does not exclude us from using pt.tl.Statsmodels on normalized data, as we can perform custom differential expression after milo.annotate_cells_from_nhoods.

New dependencies:

• scran, scuttle and SingleCellExperiment in R if using the respective filtering method in milo.find_nhood_group_markers
• formulaic_contrasts, for creating the design matrices and contrast vectors in milo.find_nhood_group_markers
• edgeR, if milo.da_nhoods(..., solver = "pydeseq2") was previously used by the user.
  Except for formulaic_contrasts, the dependencies are optional.

pt.data.stephenson_2021_subsampled only contains normalized counts, so I used scvelo.datasets.gastrulation. for testing, and the default testing dataset in tests_milo.py

[Zethson]

Thank you! I read everything but will only respond to a few things:

1. Can we make formulaic_constrasts an optional dependency that is only imported when needed? https://github.com/scverse/pertpy/blob/main/pyproject.toml#L77 it's already a part of the DE extra
2. https://scanpy.readthedocs.io/en/1.11.x/generated/scanpy.datasets.pbmc3k.html#scanpy.datasets.pbmc3k is a count matrix dataset. The X is sparse but I only see counts.

The second issue was a circular import when importing the Statsmodels class from pertpy differential expression. For this reason I took out the function using statsmodels, and left in Limma for now.

I am not quite sure whether I follow here but we shouldn't regress existing functionality.

I'm fine with only having a pydeseq2 backend and not a statsmodels backends if it doesn't seem to work well.

[MaximilianNuber]

I am sorry, I just created a full pull request, but closed immediately because I meant a draft PR, which I submitted now.

https://scanpy.readthedocs.io/en/1.11.x/generated/scanpy.datasets.pbmc3k.html#scanpy.datasets.pbmc3k is a count matrix dataset. The X is sparse but I only see counts.

I am sorry about this, too, I missed the obvious one.

Following changes I did before the draft PR:

• Removed formulaic_contrasts as a dependency.
• Removed all remnants of limma and statsmodels code
• Implemented tests with the pbmc3k dataset
• All tests pass, however warnings are still emitted

I am now going through the warnings in test_milo.py to see which I can resolve. Apologies if this takes a bit, I am new to this.

I also added plotting functions, please see the draft PR for this;

Best,
Max

[Zethson]

No worries! I'm very grateful for your contribution and we'll figure things out together :)