[BUG] Fabricated-Observation recovery in process_llm_response is dead code since #2483 — real tool calls silently discarded for models without stop-word support

[ritsth]

Description

process_llm_response() in lib/crewai/src/crewai/utilities/agent_utils.py contains a recovery block that is meant to handle models that don't support the stop parameter (use_stop_words=False: the gpt-5 family, the o1 family, and any custom BaseLLM that ignores stop sequences). Without a "\nObservation:" stop sequence, the LLM generates straight past the real tool call and fabricates the rest of the ReAct loop in a single completion:

Thought: ...
Action: web_search
Action Input: {"search_query": "..."}
Observation: <fabricated — the tool never ran>
Final Answer: <fabricated>

The recovery block is supposed to discard the fabricated continuation so the real Action executes:

if not use_stop_words:
    try:
        format_answer(answer)
    except OutputParserError as e:
        if FINAL_ANSWER_AND_PARSABLE_ACTION_ERROR_MESSAGE in e.error:
            answer = answer.split("Observation:")[0].strip()

This block has been dead code since April 2025. It catches an error that can no longer be raised:

• From Removing LangChain and Rebuilding Executor #1322 (e77442cf, Oct 2024), CrewAgentParser.parse() raised FINAL_ANSWER_AND_PARSABLE_ACTION_ERROR_MESSAGE whenever a response contained both a parseable Action and a Final Answer, and the executor caught it and truncated at "Observation:". This worked — the real tool executed.
• Feat/individual react agent #2483 (efe27bd5, Apr 2025) removed the raise from the parser (commit log: "Remove redundant error handling for action and final answer in CrewAgentParser") and deleted its regression test test_action_and_final_answer_error — while, in the same PR, moving the catch block into the newly created process_llm_response(). The raise was not redundant; it was the trigger for that catch. Since then parse() returns the fabricated AgentFinish whenever Final Answer: appears anywhere in the text, and the recovery can never run.
• (It is doubly unreachable: format_answer() also swallows every exception — that's the separate, orthogonal issue [BUG] format_answer() catches OutputParserException, breaking retry logic for malformed LLM outputs #3771/[BUG] format_answer() catches OutputParserException, breaking retry logic #4113. Fixing that alone would not revive this recovery, because the error message it looks for no longer exists anywhere in the codebase.)

Other leftovers of the original intent are still in the tree:

• FINAL_ANSWER_AND_PARSABLE_ACTION_ERROR_MESSAGE still sits in crewai/agents/constants.py, imported only by the dead catch.
• The experimental executor has an unreachable warning block (experimental/agent_executor.py, "LLM returned 'Final Answer:' but parsed as AgentAction") that only makes sense with the recovery alive.

User-visible impact: this is, as far as I can tell, the root cause of the many reports collected in #3154 ("Agent does not actually invoke tools, only simulates tool usage with fabricated output"). That thread independently observed the exact correlation this explains: gpt-4.1 works (stop words supported → generation stops before a fabricated Observation:), while gpt-5 / o-series fabricate (supports_stop_words() returns False for both, so nothing stops the completion and the dead recovery never rescues the real tool call).

Steps to Reproduce

from crewai.utilities.agent_utils import process_llm_response

# What a model without stop-word support typically returns (shape from #3154):
answer = """Thought: I should use the Web Search Tool to investigate trends in AI.
Action: Web Search Tool
Action Input: {"search_query": "trends in AI 2025"}
Observation: The search results show that the main trends include multimodal models.
Thought: I now have enough information.
Final Answer: The main AI trends are multimodal models."""

result = process_llm_response(answer, use_stop_words=False)
print(type(result).__name__)  # AgentFinish — the fabricated answer wins

Expected behavior

AgentAction(tool="Web Search Tool", ...) — the fabricated Observation/Final Answer continuation is discarded and the real tool call executes, as it did before #2483.

Screenshots/Code snippets

See above; the dead block is at lib/crewai/src/crewai/utilities/agent_utils.py (in process_llm_response), current main (2b90117e).

Operating System

macOS (reproduces on any OS)

Python Version

3.13

crewAI Version

current main (1.15.2a2, commit 2b90117e)

crewAI Tools Version

n/a

Virtual Environment

uv

Evidence

Regression timeline verified with git log -S FINAL_ANSWER_AND_PARSABLE_ACTION_ERROR_MESSAGE: raise introduced in e77442cf (#1322), removed in efe27bd5 (#2483) together with its test, catch block moved (already dead) into process_llm_response in the same commit.

Possible Solution

Replace the dead except block with an explicit position check — no exception plumbing, so it is immune to format_answer()'s exception swallowing and independent of #3771:

if not use_stop_words and FINAL_ANSWER_ACTION in answer:
    action_match = ACTION_INPUT_REGEX.search(answer)
    final_answer_idx = answer.find(FINAL_ANSWER_ACTION)
    if action_match and action_match.start() < final_answer_idx:
        observation_idx = answer.find(
            "Observation:", action_match.start(), final_answer_idx
        )
        if observation_idx != -1:
            answer = answer[:observation_idx].strip()

return format_answer(answer)

Only the exact fabrication shape (Action → Observation → Final Answer, with stop words unsupported) is affected; every other response shape, and all behavior for stop-word-supporting models, is unchanged. This also fixes all four call sites at once (experimental AgentExecutor, CrewAgentExecutor, LiteAgent, step_executor) and makes the experimental executor's existing warning reachable.

I have this implemented with regression tests (fails-before/passes-after) and will open a PR referencing this issue.

Additional context

Found by code inspection while tracing why the recovery path never fires. AI-assisted analysis (Claude); every claim above was verified by hand against git history and a local reproduction.

[0xbrainkid]

This is a strong bug report because it separates the parser failure from the trust failure users actually experience: the transcript says a tool produced evidence, but the runtime never executed the tool.

One useful invariant for the fix is that a model-authored Observation: should never be accepted as tool evidence unless it is joined to an actual tool execution record. For this specific path, I would want the regression test to assert both parse shape and evidence shape:

model output contains Action + fabricated Observation + Final Answer
use_stop_words = false
parser returns AgentAction, not AgentFinish
tool execution record is created before any Observation is appended
final answer can only cite an observation that has execution_id / tool_call_id

The important distinction is:

claimed observation -> text generated by the model
runtime observation -> result emitted after a real tool invocation

If those collapse into the same transcript field, downstream memory, evaluation, and reputation systems cannot tell whether the agent actually used a tool or only simulated tool use.

From adjacent AgentFolio/SATP work, the portable lesson is evidence layering, not a dependency: identity, tool authority, execution receipt, and final-answer attribution should stay separate but linkable. A minimal receipt for this class of bug could be as small as:

agent_run_id
tool_call_id
tool_name
canonical_args_hash
execution_status
observation_hash
created_at

Then the final answer can point to tool_call_id / observation_hash, and a verifier can reject model-authored observations that lack a matching runtime receipt.

So I would treat the proposed position check as the immediate compatibility fix, and add a second negative control that protects the evidence boundary: fabricated Observation: text must not become a valid observation artifact until the tool runner has produced it.

Boundary note: architecture feedback only. No claim of integration, endorsement, certification, or production readiness.

[mayank609]

@ritsth This is a good example of why successful completion isn't the same as successful execution. An Action being generated shouldn't implicitly mean the tool was invoked. Having explicit execution checkpoints (Action generated → Tool invoked → Observation received) would make it much easier to identify where the workflow diverged instead of discovering it from the final response.

[hermes6941]

This looks like a useful failed-agent debugging case.

If you can share a sanitized snippet showing what the agent/tool was expected to do, what actually happened, and the trace around the failure, I can run it through Rootlyze and send back a proofpack with likely cause, evidence, next action, prevention guidance, and limitations.

Please remove secrets, API keys, tokens, customer data, and private URLs first.