render_labels color plot no longer working

[mezwick]

Morning,

Trying to make a side-by-side render_image and render_label plot for my sdata object.

My code worked with earlier versions of spatialdata-plot, but since updating to 0.2.8 have found some issues. (e.g. for render_image, needed to switch from specifying vmin and vmax, to doing this via norm = matplit.colors.Normalize()). My render_image code is now working.

For render_labels though, I seem to get a 'KeyError' when I provide a color parameter. The function works fine when not providing a color parameter. Not sure why, the color name does seem to be a column in sdata.tables['table'].obs

Any pointers?
Am sure this code worked with earlier versions for spatialdata-plot.

Thanks

Code

# TODO: TROUBLESHOOT CODE

from matplotlib import colors

# Generate plot for chosen coords
coord = 'slide'
lbl = 'bulk4_ldn_n50_r0pt15_anno'
# lbl = 'bulk4_ldn_n50_r0pt15'

sdata.tables['table'].uns[f'{lbl}_colors'] = glasbey.create_palette(len(sdata.tables['table'].uns[f'{lbl}_colors']))

fig, axs = plt.subplots(nrows = 1, ncols = 2, dpi=600, figsize = (10, 5))
axs_rvl = axs.ravel()

## render_images, axis 0
# Set vmin and vmax, with clip
norm = colors.Normalize(vmin=0, vmax=200, clip=True)
sdata.pl.render_images(element = f'{coord}_images', channel='DNA1-Ir191', cmap='gray', norm=norm).pl.show(coordinate_systems=coord, title="DNA", colorbar=False, ax=axs_rvl[0])

## render_labels, axis 1
groups = list(sdata.tables['table'].obs[lbl].unique())
palette = sdata.tables['table'].uns[f'{lbl}_colors']

# Works, no color by label call
# sdata.pl.render_labels(element = f'{coord}_labels', table_name='table').pl.show(coordinate_systems=coord, ax=axs_rvl[1], title='bulk clustering', legend_loc='lower left', legend_fontsize=10)

# Errors on the color call
sdata.pl.render_labels(element = f'{coord}_labels', table_name='table', color=lbl, groups=groups, palette=palette).pl.show(coordinate_systems=coord, ax=axs_rvl[1], title='bulk clustering', legend_loc='lower left', legend_fontsize=10)

# Previously working
# sdata.pl.render_labels(element = f'{coord}_labels', color=lbl, groups=groups, palette=palette).pl.show(coordinate_systems=coord, ax=axs_rvl[1], title='bulk clustering', legend_loc='lower left', legend_fontsize=10)

# plt.suptitle(coord)
plt.tight_layout()

Error

Output exceeds the [size limit](command:workbench.action.openSettings?[). Open the full output data [in a text editor](command:workbench.action.openLargeOutput?5f6746ed-c6f8-44ba-95f7-6663ee9aeeae)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File c:\Users\U062951\Miniconda3\envs\imc_ome_v1_spatial\lib\site-packages\pandas\core\indexes\base.py:3805, in Index.get_loc(self, key)
   3804 try:
-> 3805     return self._engine.get_loc(casted_key)
   3806 except KeyError as err:

File index.pyx:167, in pandas._libs.index.IndexEngine.get_loc()

File index.pyx:196, in pandas._libs.index.IndexEngine.get_loc()

File pandas\\_libs\\hashtable_class_helper.pxi:7081, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas\\_libs\\hashtable_class_helper.pxi:7089, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'bulk4_ldn_n50_r0pt15_anno'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In[95], line 28
     22 palette = sdata.tables['table'].uns[f'{lbl}_colors']
     24 # Works, no color by label call
     25 # sdata.pl.render_labels(element = f'{coord}_labels', table_name='table').pl.show(coordinate_systems=coord, ax=axs_rvl[1], title='bulk clustering', legend_loc='lower left', legend_fontsize=10)
     26 
...
   3815     #  InvalidIndexError. Otherwise we fall through and re-raise
   3816     #  the TypeError.
   3817     self._check_indexing_error(key)

KeyError: 'bulk4_ldn_n50_r0pt15_anno'

[mezwick]

Morning.

Adding a minimal code example here to demo based on scverse/napari-spatialdata#324 (comment)

I've tried adding a 'strings' .obs column, with the colors as both a list and a dictionary.

If I use a dictionary, the colors come through with napari_spatialdata.Interactive() viewer (but only for napari_spatialdata 0.5.4post0, these fail with 0.5.5). However, i can't call the colors via .render_labels() from spatialdata_plot.

Incidentally, I thought colors were supposed to be a list of hexcodes? Isn't that what scanpy would expect if pulling the same colors rather than a dictionary?

import spatialdata as sd
import spatialdata_plot
import napari_spatialdata
from spatialdata.datasets import blobs_annotating_element
import matplotlib.pyplot as plt

print(f"spatialdata version: {sd.__version__}")
print(f"spatialdata_plot version: {spatialdata_plot.__version__}")
print(f"napari_spatialdata version: {napari_spatialdata.__version__}")

sdata = blobs_annotating_element("blobs_labels")

# Add new obs id
obs = sdata["table"].obs
obs["strings"] = ["A", "A", "B", "B", "C"]

# Add colors by dictionary
sdata["table"].uns["strings_colors"] = {
    "A": "#FF5733",  # red
    "B": "#3498DB",  # blue
    "C": "#2ECC71",  # green
}

# # Add colors by list
# sdata.tables["table"].uns["strings_colors"] = [
#     "#FF5733", #red
#     "#3498DB", #blue
#     "#2ECC71" #green
#     ]

napari_spatialdata.Interactive(sdata)

fig, axs = plt.subplots(ncols=2)
sdata.pl.render_images('blobs_image').pl.show(ax=axs[0])
sdata.pl.render_labels('blobs_labels', color='strings').pl.show(ax=axs[1])
plt.show()

The error output is

c:\Users\U062951\Miniconda3\envs\spatialdata_3pt0\lib\site-packages\dask\dataframe\__init__.py:31: FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning.
  warnings.warn(
spatialdata version: 0.3.0
spatialdata_plot version: 0.2.9
napari_spatialdata version: 0.5.4.post0
c:\Users\U062951\Miniconda3\envs\spatialdata_3pt0\lib\site-packages\anndata\_core\anndata.py:401: FutureWarning: The dtype argument is deprecated and will be removed in late 2024.
  warnings.warn(
c:\Users\U062951\Miniconda3\envs\spatialdata_3pt0\lib\site-packages\spatialdata\models\models.py:1053: UserWarning: Converting `region_key: region` to categorical dtype.
  return convert_region_column_to_categorical(adata)
c:\Users\U062951\Miniconda3\envs\spatialdata_3pt0\lib\site-packages\spatialdata\models\models.py:1053: UserWarning: Converting `region_key: region` to categorical dtype.
  return convert_region_column_to_categorical(adata)
Output exceeds the [size limit](command:workbench.action.openSettings?[). Open the full output data [in a text editor](command:workbench.action.openLargeOutput?5327ed03-eabe-489b-b670-861c1ab368c6)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File c:\Users\U062951\Miniconda3\envs\spatialdata_3pt0\lib\site-packages\pandas\core\indexes\base.py:3805, in Index.get_loc(self, key)
   3804 try:
-> 3805     return self._engine.get_loc(casted_key)
   3806 except KeyError as err:

File index.pyx:167, in pandas._libs.index.IndexEngine.get_loc()

File index.pyx:196, in pandas._libs.index.IndexEngine.get_loc()

File pandas\\_libs\\hashtable_class_helper.pxi:2606, in pandas._libs.hashtable.Int64HashTable.get_item()

File pandas\\_libs\\hashtable_class_helper.pxi:2630, in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 0

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In[2], line 35
     33 fig, axs = plt.subplots(ncols=2)
     34 sdata.pl.render_images('blobs_image').pl.show(ax=axs[0])
---> 35 sdata.pl.render_labels('blobs_labels', color='strings').pl.show(ax=axs[1])
     36 plt.show()
...
   3815     #  InvalidIndexError. Otherwise we fall through and re-raise
   3816     #  the TypeError.
   3817     self._check_indexing_error(key)

KeyError: 0

[LucaMarconato]

Hi, thanks for the minimal example.
If you add obs["strings"] = obs["strings"].astype('category') after obs["strings"] = ["A", "A", "B", "B", "C"] spatialdata-plot doesn't raise errors and makes this plot:

This is the full code

You can add a header

You can add text within a collapsed section.

You can add an image or a code block, too.

import spatialdata as sd
import spatialdata_plot
import napari_spatialdata
from spatialdata.datasets import blobs_annotating_element
import matplotlib.pyplot as plt

print(f"spatialdata version: {sd.__version__}")
print(f"spatialdata_plot version: {spatialdata_plot.__version__}")
print(f"napari_spatialdata version: {napari_spatialdata.__version__}")

sdata = blobs_annotating_element("blobs_labels")

# Add new obs id
obs = sdata["table"].obs
obs["strings"] = ["A", "A", "B", "B", "C"]
obs["strings"] = obs["strings"].astype('category')

# Add colors by dictionary
sdata["table"].uns["strings_colors"] = {
    "A": "#FF5733",  # red
    "B": "#3498DB",  # blue
    "C": "#2ECC71",  # green
}

napari_spatialdata.Interactive(sdata)

fig, axs = plt.subplots(ncols=2)
sdata.pl.render_images('blobs_image').pl.show(ax=axs[0])
sdata.pl.render_labels('blobs_labels', color='strings').pl.show(ax=axs[1])
plt.show()

@timtreis @Sonja-Stockhaus I think the error message/or check for type should take into account into this. Specifically, if the column is not converted to a categorical column, then the code gets an exception here:

spatialdata-plot/src/spatialdata_plot/pl/utils.py

Line 824 in 478ab27

cols = cmap_params.cmap(cmap_params.norm(color_vector))

A check on the dtype of the column could be added there, and if it's a string, we could try to convert it to categorical and raise a warning saying that the user should convert to categorical before.

[LucaMarconato]

I could reproduce the napari bug, I linked this issue to this one: scverse/napari-spatialdata#324.

[LucaMarconato]

@mezwick btw tests are being added in this PR #413

[mezwick]

Thanks for your help. .astype('category') does solve the minimal code example for me and render_labels now works.

Though I would expect render_labels(colors='strings') to render in the red, green, blue, colors specified in .uns['strings_colors'] rather than default seaborn categorical green, orange, blue?

They do show as red, green, blue in the Interactive() napari?

[LucaMarconato]

Good point, yes the bug in spatialdata-plot is still present.