Agent not good at paging to solve truncated results.

[rxgrant]

Agent is not very good at paging to solve truncated results, despite having seen current stock instructions on how to do so.

Calling tool git_diff(status: "staged", paths: ["src/app.rs"])

I'm noticing the diff is truncated and there are gaps between the hunks I've
already seen. I need to examine the full file region between lines 1356 and 1542
to catch any changes I might have missed, but I'll need to see both the old and
new versions to understand what's actually changed.

The diff tool truncates. Let me read the app.rs region between the hunks I
haven't seen (lines 1356–1542) to find any change unrelated to #64's grouping.

Calling tool fs_read_file(path: "src/app.rs", start_line: 1356, end_line: 1545)

There is some tension between workflows that want long results and those that can be solved with searches instead. Every tool-refusal (haha) generates round-trip delays and nondeterministic paging-protocol dropout, which is noticable in cases where searches were not the more desirable approach.

Three paths appear evident right now:

• in the reference tooling, keep the existing nudge towards targeted searches, and if a workflow wants long results then configure another tool that does not truncate;
• make paging a much more foundational concept in JP and its tool integration, so that it can really be drilled into the models with early foundational instructions; or
• muddle on until models follow existing instructions.

[rxgrant]

I think I found a bug where my config for git-reading.toml was missing the line to enable git_diff_file. This is probably pain from pulling jp_tools on a different schedule than updating my config from the upstream config, or just from manual error while updating config.

[rxgrant]

Weirdly, when I did enable it after repairing git-reading.toml and doing jp q -c skill/git-reading [...] mid-session, the LLM tried to fight it:

I see the git_diff_file tool is referenced in the hints but not in my available
functions, though the user says they've enabled it. Let me try calling it anyway
with a staged status check on a specific file like src/number.rs to see the full
diff.

It managed to execute git_diff_file and saw results. Does a late skill-enable need more LLM handholding?

[JeanMertz]

Weirdly, when I did enable it after repairing git-reading.toml and doing jp q -c skill/git-reading [...] mid-session, the LLM tried to fight it:

I see the git_diff_file tool is referenced in the hints but not in my available
functions, though the user says they've enabled it. Let me try calling it anyway
with a staged status check on a specific file like src/number.rs to see the full
diff.

It managed to execute git_diff_file and saw results. Does a late skill-enable need more LLM handholding?

Interesting.

It shouldn't, no. We essentially build the list of tools, and if it differs from earlier, then the cache is busted and a new list is sent. However, I have seen cases where removing a tool mid-conversation can result in confusion since the LLM sees it ran the tool in its history and tries to re-run it again even though the tool is no longer listed in the list of available tools.

This seems to be the inverse, which I've never seen happen. The only thing I can think of is that it reasoned to itself earlier in the conversation which tools are/are not available at a time when the tool is indeed not listed, and it uses that same conclusion later in the conversation when the tool is in the list, to convince itself that it isn't.

If you have JP_DEBUG=1, you should be able to inspect (or share) the trace logs, which includes the full request sent to the LLM provider, which can verify whether the tool is indeed in the list or not.

[rxgrant]

It definitely reasoned about the tool being missing before I enabled it. Have debug.

[JeanMertz]

It definitely reasoned about the tool being missing before I enabled it. Have debug.

Yeah, that might possibly be the culprit.

I think eventually the right approach is to use RFD 010: System Notification Queue, which would allow us to inject notifications such as "tool XYZ is now available to use" whenever a tool is enabled mid-conversation.

This is likely to help the LLM to understand what it can do, since the problem I suspect with the current design is that our skills toml files inject a new system prompt section explaining the tool, but there is already reasoning somewhere in the conversation history from when the model couldn't find/use the tool, so the LLM first reads the system prompt now stating the tool is available, and then reading its own reasoning that it can't use/find the tool, resulting the outcome that it already concluded that it can't use the tool, even if that reasoning content is now outdated. If we had injected "tool XYZ is now available to use" as part of the latest query, I suspect the chances of the assistant using the tool go up significantly.

Have debug.

Feel free to share them with me.

Also, note that you can pass --tool-use (-u) to force the assistant to call a specific tool. E.g. jp q -u fs_modify_file. That doesn't prevent the assistant from calling any other tools, it just has to call that tool before the end of the turn. There are some caveats with that, since e.g. Anthropic doesn't officially support this anymore in newer models, but we have some built-in logic to still try and force the model to call the correct tool.

You can also set assistant.tool_choice to one of auto (default), none (don't allow any tool use), required (forced tool use, but assistant can pick which one), or any other name, matching a tool name (e.g. fs_modify_file), which forces the assistant to call that tool in the current turn. However, this will persist the choice across turns, so you'd have to reset it to assistant.tool_choice=auto to avoid the assistant from calling the same tool repeatedly each turn.