Tensor Factorization "liana::decompose_tensor" in R

[junghpark]

Thanks for this powerful package!

I've been following your LIANA x Tensor-cell2cell Quickstart (R) using a virtual environment with Python 3.11.4.

After running liana_tensor_c2c, I've got a tensor as follows:

tensor$shape
[[1]]
[1] 16

[[2]]
[1] 1766

[[3]]
[1] 37

[[4]]
[1] 37

summary(tensor)
<cell2cell.tensor.tensor.PreBuiltTensor object at 0x7f7c39ef8210>

However, when I run liana::decompose_tensor using the same tensor, decomposing takes weirdly long, regardless of setting the rank as NULL (Estimating ranks...
0%| | 0/25 [11:18:58<?, ?it/s]) or not (Decomposing the tensor...
0%| | 0/100 [00:00<?, ?it/s]), and the tf_optimization as robust or regular. After running for a couple of days, R just crashed.

I've also run the same code with your covid dataset, and the estimation time for decomposing appeared to be 74 hours but didn't get completed. I wonder if you've ever encountered this type of issue.

Thank you!

[earmingol]

Wow, that's pretty slow. On what kind of computer are you running it (CPU, RAM, etc)?
If you have access to a GPU, I recommend running the protocol using it to substantially speed up the analysis.

Also it could be an issue of your environment, have you tried generating a new one with the instructions we provide here?

[junghpark]

Thank you for your feedback! I’m running it on the HPC, where I have access to a GPU. As you recommended, I created a new environment following the instructions at env_setup and ran "liana::decompose_tensor". However, it's been running for 20 hrs so far, and showed following at around 11hr point: Estimating ranks... 0%| | 0/25 [00:00<?, ?it/s] 4%|▍ | 1/25 [11:15:22<270:09:02, 40522.62s/it] I'll update how it goes!

…

________________________________ From: Erick Armingol ***@***.***> Sent: Saturday, February 1, 2025 12:57 PM To: saezlab/ccc_protocols ***@***.***> Cc: Park, Jung Hyun ***@***.***>; Author ***@***.***> Subject: [External] Re: [saezlab/ccc_protocols] Tensor Factorization "liana::decompose_tensor" in R (Issue #21) This message was sent from a non-IU address. Please exercise caution when clicking links or opening attachments from external sources. Wow, that's pretty slow. On what kind of computer are you running it (CPU, RAM, etc)? If you have access to a GPU, I recommend running the protocol using it to substantially speed up the analysis. Also it could be an issue of your environment, have you tried generating a new one with the instructions we provide here<https://github.com/saezlab/ccc_protocols/tree/main/env_setup>? — Reply to this email directly, view it on GitHub<#21 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A3XNASAY3DKLR5FPQFLEF4D2NUKINAVCNFSM6AAAAABWJPCE2GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMRZGA3TAMJVGI>. You are receiving this because you authored the thread.

[earmingol]

This is still slow. When running the pipeline (function liana::decompose_tensor), which init parameter are you using? it sounds like you may be using the init='svd' option? I would change it to init='random'.

Additionally, if this is taking too much, I would change the tf_optimization='robust' to tf_optimization='regular'. That should speed things up. Read here for more info about these parameters: https://saezlab.github.io/liana/reference/decompose_tensor.html

If this is not working, I would try to subset the tensor and use fewer LR pairs (maybe prioritize by a process that you are interest in?) and keep ~ 400- 500 LR pairs max. You have to think that elements in your tensor are 16 x 1766 x 37 x 37 (~39 millions), so if you reduce your LR pairs to ~500, you would have ~10 millions.

I hope this helps

[junghpark]

I already have applied tf_init='random' (it's still running!), but I'll also utilize the 'regular' option and subset of the tensor. Thank you!

…

________________________________ From: Erick Armingol ***@***.***> Sent: Monday, February 3, 2025 2:01 PM To: saezlab/ccc_protocols ***@***.***> Cc: Park, Jung Hyun ***@***.***>; Author ***@***.***> Subject: [External] Re: [saezlab/ccc_protocols] Tensor Factorization "liana::decompose_tensor" in R (Issue #21) This message was sent from a non-IU address. Please exercise caution when clicking links or opening attachments from external sources. This is still slow. When running the pipeline (function c2c.analysis.run_tensor_cell2cell_pipeline()), which tf_init parameter are you using? it sounds like you may be using the tf_init='svd' option? I would change it to tf_init='random'. Additionally, if this is taking too much, I would change the tf_optimization='robust' to tf_optimization='regular'. That should speed things up. If this is not working, I would try to subset the tensor and use fewer LR pairs (maybe prioritize by a process that you are interest in?) and keep ~ 400- 500 LR pairs max. You have to think that elements in your tensor are 16 x 1766 x 37 x 37 (~39 millions), so if you reduce your LR pairs to ~500, you would have ~10 millions. I hope this helps — Reply to this email directly, view it on GitHub<#21 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A3XNASF5BJBXUJQEDKN2UYL2N7DJRAVCNFSM6AAAAABWJPCE2GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMZRHE2TAMBVGE>. You are receiving this because you authored the thread.

[junghpark]

As an update, the estimated time for decomposing the tensor of size 16 x 1766 x 37 x 37 (around 39 million) including 'sca' and 'natmi' was approximately 50 hours. To speed up the process, I merged some clusters, and the resulting tensor of size 16 x 3616 x 13 x 13 (around 9 million) employing only 'sca' took 18 hours with a GPU.

[junghpark]

By the way, I used the 'liana_tensor_c2c' function for the decomposition in this tutorial (https://saezlab.github.io/liana/articles/liana_cc2tensor.html) because the 'liana::decompose_tensor' function from another tutorial (https://ccc-protocols.readthedocs.io/en/latest/notebooks/ccc_R/QuickStart.html) wasn't working for me for some reason. Could you explain the differences between these two methods? Thank you!