Feature request
reduce_outliers() requires manually specifying a threshold parameter. Users must run the method repeatedly, inspecting results each time, to find a threshold that achieves their desired outlier rate.
Add an outliers_percentage_target parameter that auto-searches for the optimal threshold:
# Current: manual trial-and-error
new_topics = topic_model.reduce_outliers(docs, topics, strategy="probabilities", threshold=0.3)
# Check outlier rate... too high. Try 0.2... too low. Try 0.25...
# Proposed: specify the target directly
new_topics = topic_model.reduce_outliers(docs, topics, strategy="probabilities",
outliers_percentage_target=0.05) # "at most 5% outliers"
Motivation
The trial-and-error workflow is evident in the 10+ open issues asking about reduce_outliers behavior:
- #1785 —
reduce_outliers result not updated in model
- #1843 — persistent zero probability after
reduce_outliers
- #1520 — clusters changed after reducing outliers
- #2114 —
reduce_outliers removes stop_words/ngram_range effects
In production, users typically know their target — "I want at most 5% outliers" — not the internal threshold that achieves it.
Your contribution
I can submit a PR that adds an outliers_percentage_target parameter (float, 0–1): auto-search for the threshold that achieves the target outlier percentage using binary search. Works with all 4 existing strategies (probabilities, distributions, c-tf-idf, embeddings) and requires no changes to the underlying reduction logic.
Cannot set both threshold and outliers_percentage_target — raises ValueError. Backward compatible: defaults to None, existing behavior unchanged.
I've already been prototyping this in my fork, so I can open a PR quickly if the approach looks good to you.
Feature request
reduce_outliers()requires manually specifying athresholdparameter. Users must run the method repeatedly, inspecting results each time, to find a threshold that achieves their desired outlier rate.Add an
outliers_percentage_targetparameter that auto-searches for the optimal threshold:Motivation
The trial-and-error workflow is evident in the 10+ open issues asking about
reduce_outliersbehavior:reduce_outliersresult not updated in modelreduce_outliersreduce_outliersremoves stop_words/ngram_range effectsIn production, users typically know their target — "I want at most 5% outliers" — not the internal threshold that achieves it.
Your contribution
I can submit a PR that adds an
outliers_percentage_targetparameter (float, 0–1): auto-search for the threshold that achieves the target outlier percentage using binary search. Works with all 4 existing strategies (probabilities, distributions, c-tf-idf, embeddings) and requires no changes to the underlying reduction logic.Cannot set both
thresholdandoutliers_percentage_target— raisesValueError. Backward compatible: defaults toNone, existing behavior unchanged.I've already been prototyping this in my fork, so I can open a PR quickly if the approach looks good to you.