Feature request
hierarchical_topics() builds parent topic names by concatenating the top 5 c-TF-IDF keywords — e.g., "wear_safety_PPE_worker_work". Meanwhile, leaf topics get rich labels from the representation model (e.g., "PPE Non-Compliance Incidents" via an LLM). This creates a jarring inconsistency in visualize_hierarchy().
Add a use_representation_model parameter that runs the representation pipeline on parent topics:
# Current: parent nodes show "wear_safety_PPE_worker_work"
hierarchy = topic_model.hierarchical_topics(docs)
# Proposed: parent nodes get proper labels from the representation model
hierarchy = topic_model.hierarchical_topics(docs, use_representation_model=True)
Motivation
Root cause: _extract_words_per_topic is called with calculate_aspects=False for parent nodes, so no aspect models (including LLM labeling) run on them.
When using LLM-based representation models, the hierarchy visualization becomes unusable because leaf labels (e.g., "PPE Non-Compliance Incidents") are at a completely different abstraction level than parent labels (e.g., "wear_safety_PPE_worker_work"). Users must manually relabel parent nodes, which defeats the purpose of the representation pipeline.
Your contribution
I can submit a PR that adds use_representation_model: bool = False to hierarchical_topics(). When enabled, a post-processing step runs the full representation pipeline on all parent topics in a single batch call — not inline during the merge loop (which would make N-1 separate LLM calls).
Default is False — existing behavior unchanged. The batch approach keeps LLM cost bounded.
Note: If the shared helpers refactoring (#2497) lands first, this PR can reuse _aggregate_documents(). Otherwise, the aggregation is inlined — either way works.
I've already been prototyping this in my fork, so I can open a PR quickly if the approach looks good to you.
Feature request
hierarchical_topics()builds parent topic names by concatenating the top 5 c-TF-IDF keywords — e.g.,"wear_safety_PPE_worker_work". Meanwhile, leaf topics get rich labels from the representation model (e.g.,"PPE Non-Compliance Incidents"via an LLM). This creates a jarring inconsistency invisualize_hierarchy().Add a
use_representation_modelparameter that runs the representation pipeline on parent topics:Motivation
Root cause:
_extract_words_per_topicis called withcalculate_aspects=Falsefor parent nodes, so no aspect models (including LLM labeling) run on them.When using LLM-based representation models, the hierarchy visualization becomes unusable because leaf labels (e.g.,
"PPE Non-Compliance Incidents") are at a completely different abstraction level than parent labels (e.g.,"wear_safety_PPE_worker_work"). Users must manually relabel parent nodes, which defeats the purpose of the representation pipeline.Your contribution
I can submit a PR that adds
use_representation_model: bool = Falsetohierarchical_topics(). When enabled, a post-processing step runs the full representation pipeline on all parent topics in a single batch call — not inline during the merge loop (which would make N-1 separate LLM calls).Default is
False— existing behavior unchanged. The batch approach keeps LLM cost bounded.Note: If the shared helpers refactoring (#2497) lands first, this PR can reuse
_aggregate_documents(). Otherwise, the aggregation is inlined — either way works.I've already been prototyping this in my fork, so I can open a PR quickly if the approach looks good to you.