Tileset info not found with custom data

[yangmu3]

Hi,

I tried the example code from https://docs.higlass.io/jupyter.html#numpy-matrix.

import numpy as np

dim = 2000
I, J = np.indices((dim, dim))
data = (
    -(J + 47) * np.sin(np.sqrt(np.abs(I / 2 + (J + 47))))
    - I * np.sin(np.sqrt(np.abs(I - (J + 47))))
)

from  clodius.tiles import npmatrix
from higlass.tilesets import Tileset

ts = Tileset(
    tileset_info=lambda: npmatrix.tileset_info(data),
    tiles=lambda tids: npmatrix.tiles_wrapper(data, tids)
)

display, server, viewconf = higlass.display([
    View([
        Track(track_type='top-axis', position='top'),
        Track(track_type='left-axis', position='left'),
        Track(track_type='heatmap',
              position='center',
              tileset=ts,
              height=250,
              options={ 'valueScaleMax': 0.5 }),

    ])
])
display

And I got this

And this is when I try ts.tileset_info()

Could you give me some suggestions on how to solve this?
Thank you!

[pkerpedjiev]

Hi there. Are you per chance behind a firewall? The higlass.display function will start a server in the background and it looks like the client is having trouble accessing it.

I just tried the exact same example and it worked fine.

[yangmu3]

Yes, that might be the reason. What should I do if I happen to be behind a firewall?

[pkerpedjiev]

Hmmm, to be honest I'm not sure. HiGlass needs a server component to access data. You could open up a port on your firewall and pass that port number to the display function:

display, server, viewconf = higlass.display([
    View([
        Track(track_type='top-axis', position='top'),
        Track(track_type='left-axis', position='left'),
        Track(track_type='heatmap',
              position='center',
              tileset=ts,
              height=250,
              options={ 'valueScaleMax': 0.5 }),

    ])
], server_port=8061)

[Shiyuanjun001225]

Hi, I have the same question, have you solve this question?

[tcege]

Hi everyone,
I am encountering the same problem while trying to visualize a .mcool file in HiGlass-Python. My goal is to load and display a Hi-C heatmap using Jupyter Notebook and the higlass-python library. I am not using the HiGlass server or Docker—just the Python library in a local environment. I am using a Conda environment specifically set up for it, with Python 3.9 and all required dependencies installed in this environment.

Setup: Environment: Conda environment (Python 3.9)
Installed Libraries:
higlass-python (latest version)
higlass-widget
higlass-schema
All dependencies such as ipywidgets, notebook<7.0, etc., are installed and working.
Notebook: Jupyter Notebook with widgetsnbextension enabled.

Code is:
import higlass as hg
tileset = hg.cooler("/home/ltao/DATA2/tege/MicroC_cochlea/merge/merge_matrix_10kb.mcool::/resolutions/10000")
track = tileset.track("heatmap")
view = hg.view(track, width=12)
view

When running the code, the Jupyter Notebook widget renders, but I get this error:

Tileset info not found. Server: [http://localhost:39279/tilesets/api/v1/] tilesetUid:

The heatmap widget appears, but no data is shown.

I have verified that the .mcool file exists at the specified path and is accessible. I have checked that the resolution path (/resolutions/10000) in the .mcool file is valid and available. I have ensured that Jupyter Notebook is running in the correct environment, with all necessary extensions enabled.

Does anyone know what could be causing this issue or if there's a step I might be missing? Any help would be greatly appreciated.

[pkerpedjiev]

Hi there, can you try without the ::/resolutions/10000 in the filename? So this:

import higlass as hg
tileset = hg.cooler("/home/ltao/DATA2/tege/MicroC_cochlea/merge/merge_matrix_10kb.mcool")
track = tileset.track("heatmap")
view = hg.view(track, width=12)
view

I just tried this with a local file and the latest higlass-python and it worked for me:

[tcege]

Thank you for the suggestion! Here's what I did step by step, including how I activated my environment, launched Jupyter Notebook, and tested your proposed code:

conda activate higlass_2 # activated my conda environment

jupyter notebook --no-browser --NotebookApp.token='' --ip=0.0.0.0 --port=8888 # launched Jupyter Notebook

http://alpha:8888/ #opened the Notebook in my browser, my data is stored on a server (not on my local machine)

I tried your suggested code:

Unfortunately, I still encountered the same problem.
Thank you again for your help!

[pkerpedjiev]

That's odd. Could you open your browser developer tools console and post the output of that?

[tcege]

Thank you for your help in troubleshooting this issue. Here is the output :

[tcege]

[nvictus]

The hg server is running on the same machine as your notebook server but a different machine than your notebook frontend. The higlass client in your browser doesn't know that -- you can see it is trying to fetch tiles from localhost.

There is an undocumented solution to turn on jupyter-server-proxy:

import higlass as hg
hg.server.enable_proxy()

But it may not work for a variety of reasons. The alternative workaround is to configure your tilesets to declare their endpoint on the remote higlass server (assuming the server is accessible), but it currently requires some massaging:

import functools
import hashlib
import pathlib
from higlass.tilesets import LocalTileset

# Fix a port number you know is exposed and free, different from the port jupyter is running on
HIGLASS_PORT = 8889

# We currently need this to bypass the automatic port setting by hg.cooler()
def cooler_tileset(filepath: str):
    try:
        from clodius.tiles.cooler import tiles, tileset_info
    except ImportError:
        raise ImportError(
            'You must have `clodius` installed to use "matrix" data-server.'
        )
        
    abspath = pathlib.Path(filepath).absolute()
    uid = hashlib.md5(str(abspath).encode()).hexdigest()

    return LocalTileset(
        datatype="matrix",
        tiles=functools.partial(tiles, filepath),
        info=functools.partial(tileset_info, filepath),
        uid=uid,
    )

tileset = cooler_tileset(path)
hg.server.add(tileset, port=HIGLASS_PORT)

track = tileset.track("heatmap")
view = hg.view(track)
view

[tcege]

Thank you for your detailed suggestions! I tried your recommended solutions and pasted the results with screenshots.

[nvictus]

Doesn't help much if I can't see the full traceback, but I think I know what I forgot:

tileset = cooler_tileset(path)
tileset_resource = hg.server.add(tileset, port=HIGLASS_PORT)

track = tileset_resource.track("heatmap")

[nvictus]

Sigh. I just remembered, there is no exposed API to set the host name before the local server starts, so this only works if you are port forwarding your jupyter server connection over localhost to a local port. You will also have to forward the remote port of the higlass server to (the same) local port.

The short of it is, if hg.server.enable_proxy() doesn't work, this setup isn't officially supported right now. The plan is to drop the need to spawn a separate server from Jupyter entirely: #144

cc @manzt

[tcege]

Thank you for your suggestion! I tried using hg.server.enable_proxy() and port forwarding, but I still have issues. The problem seems related to the remote server setup.

Thank you again for your help!