Skip to content

Allow recalculating representative docs with a configurable count (currently hardcoded to 3) #2500

Description

@pidefrem

Feature request

_save_representative_docs hardcodes nr_samples=500 and nr_repr_docs=3. Users cannot control how many representative documents are stored per topic, nor the sampling pool size.

Add a public method to recalculate representative documents after training with a configurable count, following the API direction suggested in #1767:

# Recalculate with more representative docs after training
topic_model.recalculate_representative_docs(nr_repr_docs=10, nr_samples=1000)

Motivation

This is a frequently requested feature:

  • #552 — "get more than 3 representative docs"
  • #848 — "Ability to change # of documents pulled per topic"
  • #1767 — "How to get more than 3 representative docs per topic"

In #1767, @MaartenGr noted:

"It might be nice to have an additional function that re-calculates the representative documents since this question seems to appear frequently."

PR #2130 attempted an nr_repr_docs constructor parameter, but @MaartenGr's feedback was that he prefers a standalone public function that recalculates representative docs after training.

Use cases: production users who need more docs for human review, small datasets where 500 exceeds topic size, users who want to iterate on representative doc quality post-training.

Your contribution

I can submit a PR following @MaartenGr's preferred API direction:

  1. New public method: recalculate_representative_docs(nr_repr_docs=3, nr_samples=500) — recalculates and replaces representative_docs_ after training.
  2. Constructor defaults nr_repr_docs (default 3) and nr_repr_docs_nr_samples (default 500) — used during fit_transform().

I've already been prototyping this in my fork along the lines you suggested, so I can open a PR quickly if this direction works for you.

Backward compatible: defaults reproduce the current behavior exactly.

Note: Works standalone. Pairs well with the sampling/indexing fix for _extract_representative_docs (#2495) — that fix corrects the sampling this method configures.


Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions