V1.0 (highly experimental)

[MaartenGr]

[HIGHLY EXPERIMENTAL] / [UNLIKELY TO MERGE]

This PR is a branch off of #2467 and something that I have been thinking about for a while now.

BERTopic has been a fun project, but it's time to take it to the next level. A big part of that is applying standardization in a way that allows for the development of new features. In general, I want to:

• Standardize the codebase to the point where things can be even more modular and stable
• Introduce new sets of features that revolve around Agents/LLMs on the one hand and statistical analyses on the other
• Be prepared for the influx of AI-driven PRs that are filling up my inbox currently 😅

All of the above is meant to speed up development in a way that makes my life easier, that of contributors, and even that of Agents.

Standardization

The idea of modularity in BERTopic has turned out to be a compelling point for using the package, but did leave it with several bugs that were hard to squash. The code was also written in the early days of my career and I would do some things differently now.

In particular, I think I can improve stability and allow for a lot of cool new features by introducing a couple of data classes:

• bertopic._corpus.Corpus - This class tracks all information on a document-level and concerns the input data (e.g., documents, images, embeddings) as well as data generated during a fit (e.g., reduced embeddings, topic assignments, probabilities)
• bertopic_topics.Topic - This class tracks all information on a topic-level and can contain any number of documents or their sub-components (e.g., topic_id, various topic_representations, topic_embeddings)
• bertopic_topics.Topics - This class tracks all information on a multi-topic-level and contains references to multiple topics that are somehow related to each other (e.g., hierarchical_topics, dynamic_topics, topic_taxonomy)

Features

Alongside the above standardization, I want to make sure that BERTopic is ready for the LLM/Agentic era where we do much more than just name the clusters. Creating summarizations, sentiment-based clusters, agent-driving clusters, etc. should all be possible with BERTopic if it is truly modular.

A big part of that is rewriting the topic representation classes as they currently are. I made it simple for myself at the start by assuming keywords and hacking labels on top of that, but it should change.

There is much more that could be enabled with this standardization, but it will depend on how quickly (and how stable) this new standardization will be.

AI-only PRs

There are an increasing number of AI-written PRs that I have a lot of issues keeping up with. They are often out of scope, too large, create a lot of unnecessary code, or do not adhere to the practices of this repo. That's not to say that we shouldn't have AI write PRs, it just means that it is my responsibility to create clear guidelines on what will be expected. Due to my exceedingly limited bandwidth at the moment, that turns out to be quite the challenge.