Skip to content

approximate_distribution: add target-based min_similarity ("at most N unassigned documents") #2502

Description

@pidefrem

Feature request

approximate_distribution() requires manually specifying min_similarity. Like the threshold parameter in reduce_outliers, users must iterate to find a value that produces an acceptable number of documents with zero topic distribution.

Add an outliers_nb_target parameter that auto-searches for the optimal min_similarity:

# Current: manual trial-and-error
distributions = topic_model.approximate_distribution(docs, min_similarity=0.1)
# Count zeros... too many. Try 0.05... too few. Try 0.08...

# Proposed: specify the target directly
distributions = topic_model.approximate_distribution(docs, outliers_nb_target=50)

Motivation

Users know their target — "I want at most 50 documents without any topic assignment" — not the internal min_similarity value that achieves it. The same trial-and-error UX problem as reduce_outliers (see related issue).

Your contribution

I can submit a PR that adds:

  • outliers_nb_target (int): target number of zero-distribution documents
  • Performance optimization: refactor the internals so that the expensive similarity matrix is computed once and reused across all binary search iterations

Cannot set both min_similarity and outliers_nb_target. Backward compatible: defaults to None.

I've already been prototyping this in my fork, so I can open a PR quickly if this looks like a good direction.


Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions