Skip to content

Add top-down (divisive) strategy for hierarchical_topics() #2508

Description

@pidefrem

Feature request

hierarchical_topics() builds hierarchies exclusively bottom-up using scipy's agglomerative linkage on c-TF-IDF vectors. Add a top-down (divisive) alternative:

# Current behavior (default)
hierarchy = topic_model.hierarchical_topics(docs, strategy="agglomerative")

# New: top-down recursive splitting
hierarchy = topic_model.hierarchical_topics(docs, strategy="divisive")

Motivation

Agglomerative linkage has known limitations:

  • Merge quality degrades at higher levels — late merges combine dissimilar topics because linkage minimizes global distance, not local semantic coherence
  • No per-split representation — parent nodes get concatenated keywords
  • #1907 — confirmed bug when 3+ topics have identical c-TF-IDF distances, breaking the hierarchy

A top-down approach avoids these issues by recursively splitting topics where each split is locally optimal.

Your contribution

I can submit a PR that adds a strategy parameter to hierarchical_topics(). The divisive path recursively splits topics using c-TF-IDF weighted NMF decomposition, building a tree where each parent-child relationship reflects a meaningful topic subdivision.

Default is "agglomerative" — existing behavior unchanged.

I've already been prototyping this in my fork (working implementation with tests). Since the divisive path is a new algorithm to maintain, I'm happy to discuss scope (e.g. landing it as experimental first) before opening the PR.


Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions