Add multiple Agents/Models to the same conversation

[alexx-ftw]

What's happening?

Self explanatory

Where does this happen?

None

Impact on your workflow

None

Environment (if applicable)

No response

Additional context

No response

[enricoros]

@alexx-ftw The ticket needs more info, such as:

• what's the desired outcome for this - i.e. why shall we do it
• what should we do, i.e. in which way one would use this feature, if you could walk through it
• shall the agents/personas interact with each other, pass messages, respond in any way, shall there be a summarization of the answer
• how would the ui look like with this, and are there any tradeoffs for mobile

Any info on the above will help. ☝️

[alexx-ftw]

This would allow multiple agents (same model or different model) to have a conversation which the human user can see and interact as well. This is very useful when brainstorming ideas, to identify the weaknesses in the plans, playing devil's advocate, etc.

The interface could work in many different ways, but one idea is to have a + symbol here:

similar to how Beam works.

Agents should have full context of the chat. They could be set up to chat in a turn-based (round robin or random) style or only when @tagged by any other agent or human. Names could be randomly generated at session runtime and edited by the human.

Maybe there could also be a button here:

for quickly starting a new multi-agent session preset

Agents could have different colors for their chat bubbles and different profile pictures, and of course different system messages and initial prompts.

Wrt. UI, I dont think anyting relevant would need to be changed.

Each model would summarize the chat when needed and continue based on its own private summarized conversation with newer messages appended to it, individually.

[alexx-ftw]

I am willing to contribute a PR if you're open to it

[enricoros]

I am open to PRs! Your PR will show the way and the UX you have in mind, then I can take and clean it up to make sure it has no side effects and works for everyone (i.e. on mobile, etc).

I believe people would ask for model to model communication, and longer term execution (agentic loops). They'll be interested to know if they happen side by sie, or in separate messages, if you delete the ones you don't like, etc.
Show me what you mean with a good PR!

[alexx-ftw]

Draft PR opened for this work: #1031

[DoS007]

@alexx-ftw Very interesting feature.

a) So, the whole interaction would still be a linear conversation, right? Just for confirmation.
b) How does the orchestration work? Each model gets a aim/objective prompt beforehand? And then until user starts a chain or more and they go on until they have finished or the user stops them (or implicit by writing a new message)? An alternative to that would be a moderator ai.

@enricoros : I like to think of the Beam-Mode as a Zoom in one message in a message graph (which is linear most of the time and only temporarly not linear while beaming, at least for now). This issue seems to be another useful way of using many ais together.

[alexx-ftw]

@DoS007

As it sits in my PR it is just a multi-agent linear turn-based conversation:
Human -> Agent1 -> ... -> AgentX --> Human...

There is no orchestration now other than a very simple prompt telling the agents that other agents are in the chat.

This could serve as a base for introducing more advanced features.

I am already working in a consensus mode where agents go through multiple turns until they all agree on something the Human wants

[alexx-ftw]

Also worth adding that in Beam mode Agents can NOT read eachothers messages/interact with eachothers. It's basically 2 forks of the same chat

[enricoros]

Update: thanks for the terrific work in #1052. I am adding the lower layer plumbing for Agents / Tools, which will be followed by abstraction of execution modes, so that any new mode will be easy to develop, fast, and will work even when in Beam or other modalities. Expected a few weeks on Tools / Agent mode.