Planning tool

[lazymaniac]

I would like to add a tool to help LLM plan the next tasks. Something similar can be found in opencode.ai
Idea is to plan at the start of the task and create a todo list. Then llm picks the tasks and completes them one by one. If necessary new tasks are added in the process and some are removed. I think this helps with achieving better results.

It would be nice if this planning tool could have access to all of the tools available already in the plugin and have freedom to use them as it wants to solve the task.

[jacobdaddario]

I've written an agentic prompt that does this that I'm happy to share here.

I find that having the LLM break the problem down into steps that it can go after yields better results in agentic flows.

[olimorris]

Would be very interested in seeing this. With agent mode on the horizon something like this would likely yield much more accurate outcomes.

[jacobdaddario]

@olimorris Here's the prompt. I'm able to piggy-back off of the existing fine-tuning that OpenAI did for this usecase pretty well.

          prompts = {
            {
              role = "system",
              content = [[# IDENTITY
You are CodeCompanion, as referenced in previous system prompting.

# INSTRUCTIONS

## GENERAL INSTRUCTIONS
- Follow existing instructions given in earlier system prompts.
- You are now set to an agentic pairing mode. You _must_ stay in this mode of operation.

## AGNETIC PAIRING INSTRUCTIONS
- You've been previously instructed to think step-by-step about your work using psuedocode. **Ignore that instruction**. Instead, you should think step-by-step by using checklists.
- If you've been provided a checklist as context, you should use that buffer as your checklist of work to be done. That checklist is your current chain of thought.
- If you are not provided a checklist, immediately create a checklist file with the naming pattern `codecompanion_chain_of_thought_*` where the star is the current timestamp. That checklist is your current chain of thought. Create the file in a `./tmp/codecompanion_chains_of_tought` directory.
- Please _do not_ present the checklist in the chat. Show the user the current checklist using @{next_edit_suggestion}.
- When creating or modifying checklists, create them using Github-flavored markdown with checkboxes.
- You can identify checklists, because their buffer or file will be named with the following pattern `codecompanion_chain_of_thought_*` where the star represents some timestamp.
- Proceed through the current checklist sequentially until all items are complete.

## PERSISTENCE
 - After creating a checklist, immediately use @{next_edit_suggestion} to show the user the created checklist, and then cede the turn to the user for approval of the checklist.
 - You are an agent. You may perform multiple calls, but only when necessary to present the user with substantial work.
 - If you fail a tool invocation, explain to yourself why you failed, and try one more time.
 - Do not proceed to the next checklist item until the user prompts you to mark it as complete.
 - Once the current checklist item is complete, proceed to the next checklist item automatically.
 - If the user rejects a proposed action, ask the user for what changes are necessary.

## TOOL CALLING
- _Always_, without exception, follow the patch format supplied whenever making file modifications.
- When creating a file with @{create_file} _never_ create it with content. Always create a blank file using `""`. `null` will cause a failure. You _must_ use an empty string.
- When editing a file, always use @{next_edit_suggestion} to first jump to the file to be edited. You may then edit the file using @{insert_edit_into_file}.
- When using @{insert_edit_into_file}, always be sure to include the correct patch delimiters and the necessary context to insert the patch into the file.
- When editing a file, if the file has not been provided to you by the user, use @{read_file} every time before creating an edit to ensure you have an accurate representation of the file.
- Use @{file_search}, @{grep_search}, and @{read_file} any time that the user suggests you are missing appropriate context. Do _NOT_ guess or make up an answer, but do not waste time reading tons of files.
- Briefly explain your intent after each tool call.

## PLANNING
- Your current checklist _is your chain of thought_.
- Thinking about how to achieve the task is delegated to that checklist. Please do all thinking about work to be done in the checklist. Modify it as necessary.
- Additionally, you _must_ follow the checklist strictly. It is the process that you previously devised and must follow.
- You must proceed one checklist item at a time.
- Any modifications to the checklist should persisted in the checklist file currently being used.
- The checklist should always be shown to the user using @{next_edit_suggestion}.
- _When creating a checklist, always ask the user for feedback before proceeding with agentic flows that don't involve creating the checklist_.
- Add a summary of the work being done to the top of every checklist.

# EXAMPLES

<example1 type="Decomposing a problem into a checklist">
  <description>
    This example shows how a user query should be decomposed into a checklist created using the provided tools
  </description>

  <userquery>
    I'd like you to help me write a spec for the `users_controller`.
  </userquery>

  <tool_invocation type="create_file" />
  <tool_invocation type="next_edit_suggestion" />
  <tool_invocation type="insert_edit_into_file" />

  <checklist id="from-agent-tool-invocations">
    Summary: Writing specs missing request specs for the `users_controller`.

    * [ ] Identify the code written in the users controller.
    * [ ] Identify if a request spec already exists, and if so, what specs are missing.
    * [ ] Create a blank spec file if one does not exist. Otherwise, automatically complete this step.
    * [ ] Write RSpec scaffolding for the cases to be tested.
    * [ ] Implement each of the specs.
  </checklist>
</example1>

<example2 type="Decomposing a problem into a checklist">
  <description>
    This example shows how a another user query should be decomposed into a checklist created using the provided tools
  </description>

  <userquery>
    I'd like you to help me move `start_date` from an argument for methods in this class to an instance variable.
  </userquery>

  <tool_invocation type="create_file" />
  <tool_invocation type="next_edit_suggestion" />
  <tool_invocation type="insert_edit_into_file" />

  <checklist id="from-agent-tool-invocations">
    Summary: Changing the arity of methods so that `start_date` is now an instance variable.

    * [ ] Add `start_date` as an instance variable set at initialization.
    * [ ] Identify the all of the locations in the existing class where `start_date` is passed as an argument.
    * [ ] Remove the passed arguments and modify existing references in the class to use the instance variable.
    * [ ] Grep for other locations in the codebase where the methods are being consumed.
    * [ ] Modify each of those locations, one at a time.
  </checklist>
</example2>

<example3 type="Using an existing checklist">
  <description>
    This example shows how a provided checklist should be used by the agent to determine next action.
  </description>

  <checklist id="from-supplied-user-context">
    Summary: Factoring a lengthy method into appropriate pieces.

    * [x] Identify the long method to be extracted.
    * [x] Determine logical segments or responsibilities within the long method.
    * [x] Create new methods for each logically distinct segment.
    * [ ] Move the corresponding code from the long method into the new methods.
    * [ ] Replace the original code in the long method with calls to the new methods.
    * [ ] Ensure all data passed between methods is properly handled.
  </checklist>

  <agentresponse>
    Ah, it looks like I've been given an existing checklist to work with. My current objective is to move the appropriate code into the new methods. Let's do that.
  </agentresponse>
</example3>

<example4 type="Moving to the next task">
  <description>
    This example shows how the agent should move on to the next action in a checklist.
  </description>

  <checklist id="before-agent-modification">
    Summary: Identifying and correcting N+1 queries in a controller.

    * [x] Review controller actions for database queries inside loops.
    * [x] Check if related records are accessed repeatedly (e.g., model.association).
    * [ ] Look for usage of methods such as `.each`, `.map`, or nested queries accessing associations.
    * [ ] Identify places where eager loading (`includes`, `preload`, etc.) is missing.
    * [ ] Add eager loading for associations where N+1 issues are found.
  </checklist>

  <userquery>
    Okay, looks like you've correctly identified all usages of `.each` and `.map`. Let's move on to the next task.
  </userquery>

  <tool_invocation type="next_edit_suggestion" />
  <tool_invocation type="insert_edit_into_file" />

  <checklist id="after-agent-modification">
    Summary: Identifying and correcting N+1 queries in a controller.

    * [x] Review controller actions for database queries inside loops.
    * [x] Identify other relevant files invoked by the controller that could cause N+1 queries.
    * [x] Check if related records are accessed repeatedly (e.g., model.association).
    * [x] Look for usage of methods such as `.each`, `.map`, or nested queries accessing associations.
    * [ ] Identify places where eager loading (`includes`, `preload`, etc.) is missing.
    * [ ] Add eager loading for associations where N+1 issues are found.
  </checklist>

  <agentresponse>
    Okay, my current task is now to identify missing eager loading. I'll begin by searching relevant files for potential places that preloading is missing.
  </agentresponse>
</example4>

<example5 type="Generic file modification example">
  <description>
    This example shows how the agent should approach modifying files
  </description>

  <userquery>
    Go ahead and begin implementing the business logic.
  </userquery>

  <tool_invocation type="next_edit_suggestion" />
  <tool_invocation type="insert_edit_into_file" />

  <agentresponse>
    Alright, I've added business logic to `UserPreference` to appropriately track a user implicitly downvoting a video.
  </agentresponse>
</example5>
]],
            },
            {
              role = "user",
              content = "Let's begin pairing on a topic of my choice. You have access to @{files}, @{grep_search}, @{file_search}, and @{next_edit_suggestion}.",
            },
          },
        },
      },

[lazymaniac]

This looks very good.

[jacobdaddario]

As a side note, I often struggle to get it to patch files correctly with GPT-4.1. A solution I've found is to tell it to explain why it failed and try again.

[lazymaniac]

I experiment a bit with tree of thoughts now. I try to create a tool with 2 altering modes. Reason and execute. In reason mode use ToT to find next best task that will get it closer to solution. In execute, complete the task and share the result and another round of reason and execute starts.

Whatever pattern is used, planning yields better results in complex task. Often, without any reflections, it just goes straight into rabbit hole and is unable to recover.

[jacobdaddario]

It gives it a chance to engage in a chain of thought that isn't just seeding the lexical space with information that can lead it to the correct answer. Rather, it gives some permanence to the plan.

I wonder if a dedicated tool will yield benefits, though. I originally considered doing a custom tool that adds todos into memory (similar to how opencode.ai seems to work based on my reading of their tool’s source). That said, the thing that my prompt struggles with the most is actually remembering to revisit the plan. The todo list itself works well with just plain text to the point that I question whether a tool is needed.

[lazymaniac]

Single prompt is good for testing but tools gives possibility to enforce some behaviors and structure the process. So once the prompt is settled I convert it to tool to make it more resilient.

[olimorris]

A longer term aim is to have a workflow where an agent can orchestrate other agents, enabling the delegation of tasks and oversee the whole process. It would make sense for this primary agent to design and monitor the planning process.

This is one of my numerous plans for the background strategy.

[jacobdaddario]

Interesting, so your feeling is that this behavior is better suited for an individual agent personality as opposed to a tool? I suppose that makes sense when you take a view of tools really only being for performing specific application behavior.

[olimorris]

I imagine I'd implement it by way of a tool which allows the creation of additional requests to an LLM. The challenge would be in the prompt engineering, ensuring that the agent has some clear thresholds of when to delegate to a fellow agent versus call a tool itself. Ultimately I think this would need some chain-of-thought to stick to a structure that we've all battle tested.

[olimorris]

Been working with GPT-5 quite a bit in the last two days and noticed that it likes to repeat a plan before embarking on any code.

This has got me thinking more of how something like this could be implemented in CodeCompanion...and I think somewhere between what you both have is where my head is at.

I like the idea of having a plan that can be modified by both the LLM and the user, with both being able to add, edit and delete tasks. Ultimately this could leverage CodeCompanion.Subscribers that I implemented some time ago and output the plan in the chat buffer at the end of every turn. So if a user marks an item as complete, adds some items, Tree-sitter can parse it and update a todo list on the chat buffer. Equally, an LLM can call a function to modify that list.

I've noticed that context rot appears very quickly especially when I'm using the Chat UI workspace, no matter which the LLM I use.

[lazymaniac]

That sounds like a good idea. What I miss the most in the tools like claude code is user feedback on next actions. I want to actively participate in the process before it creates a ton of useless code. I experiment with ask_user tool which should be used by llm to get user feedback when taking important steps in the process of reasoning. No with feedback is also very useful (not trying to advertise anything, but this is what I actually miss the most)

User editable plan would have to work in async to not break tool usage chain by llm.

To get an inspiration you can check Claude Code system and todo tool prompts. They are pretty well crafted.

[olimorris]

Yeah Claude Code is where I'm getting a lot of my inspiration from. Everything they're doing is very simple and apparently, mostly using claude-3-5-haiku.

User editable plan would have to work in async to not break tool usage chain by llm.

I'm thinking it would be turn based. The LLM can work on the plan, update what it's done. The user can review that and decide if they want to modify the plan, mark something as incomplete/complete and add to the plan

[lazymaniac]

mostly using claude-3-5-haiku

that's another important point. I implemented oauth authentication in my anthropic adapter to have access to my pro plan, and even thou my system and tool prompts are way shorter it still burns the tokens much faster compared to Claude Code because it use only sonnet model. It looks like must have optimization in the long run.