Naive ideas for making gptel simpler yet more powerful

[grothesque]

Edit: Originally, the title of this issue was “Please consider not making prompt prefixes markdown/org headings by default”. I changed it to better reflect the way into which the discussion had evolved.

---

Thank you very much for this impressive tool that is helping me discover a new world of possibilities.

I have the impression that the default choice of prompt prefixes (gptel-prompt-prefix-alist and gptel-response-prefix-alist) being headings is sub-optimal both in principle and in practice:

In principle it means that the first line of each prompt is marked-up as a heading, while the following lines are regular text. This distinction, however, is not in any way visible to the model, and has therefore no effect. Often the first line of a multi-line prompt will not have any particular importance - why mark it up differently?

In practice, if one tries to use other headings in a M-x gptel buffer, the fixed "prompt headings" tend to mess up the structure. At least with org, the possibility to structure the document as a tree is at the heart of what it is about. Gptel's readme file suggests a solution: https://github.com/karthink/gptel?tab=readme-ov-file#use-branching-context-in-org-mode-tree-of-conversations. How about making this the actual default? Or something else that does not introduce an inconsistency between visuals and semantics, and that does not interfere with the document structure.

[karthink]

In principle it means that the first line of each prompt is marked-up as a heading, while the following lines are regular text. This distinction, however, is not in any way visible to the model, and has therefore no effect. Often the first line of a multi-line prompt will not have any particular importance - why mark it up differently?

I agree! It doesn't work well, I'd like to change it. The reasons it's still the default are:

1. It works for very simple question/answer style interactions. This is all many users use gptel for.
2. Users have "notebooks" of stored gptel chats. If they use the default prefixes and I change them,
   • The heading characters will be sent to the LLM in subsequent interactions. (The choice of prefixes are not stored in the markdown files.)
   • The continuation of the chat will be formatted differently from the existing exchanges.
3. Any change is going to confuse a lot users. You'd be surprised at how many gptel users believe the current heading/body prefixes are part of gptel's "syntax" for differentiating between prompts and responses. Most users, even Emacs users, don't look at customization options or read the README beyond the bare minimum needed to install the package[1].
4. I don't know what good defaults should be. If change it to @user: and @assistant: or something like that, this will make problem 3 worse, as this looks even more like special, required syntax.

Point 3 continued: More than changing the defaults, I'd like some way to transparently indicate to the users that the prefixes are cosmetic as far as the LLM is concerned.

1 and 2 are not minor issues, we can live with these breakages. If you have suggestions for 3 and 4 however, please let me know.

How about making this the actual default? Or something else that does not introduce an inconsistency between visuals and semantics, and that does not interfere with the document structure.

This tree-of-conversations feature only works in Org mode, not markdown.

• To make this behavior the default, markdown should support it too, otherwise the behavior is inconsistent. PRs for adding this to markdown chat buffers are welcome.
• Sometimes the model replies with headings in the response. The "thinking" models like o1 and gemini-flash-thinking do this often. This messes up the hierarchy. This breakage is okay if branching context is an option you've turned on yourself, but not if this is gptel's default.
• Branching context is a confusing idea to explain. The current model is very straightforward: "everything up to the cursor is sent".

As for "something else", again suggestions are welcome. The only constraint is that I don't plan to switch gptel to using syntax to track responses instead of invisible text-properties.

[1]: I know this because this is also my default behavior when installing Emacs packages. There's just too much to read considering how many Emacs packages we use, and I like reading documentation.

[grothesque]

I agree! It doesn't work well, I'd like to change it. The reasons it's still the default are:

1. It works for very simple question/answer style interactions.  This is all many users use gptel for.

2. Users have "notebooks" of stored gptel chats.  If they use the default prefixes and I change them,

   * The heading characters will be sent to the LLM in subsequent interactions.  (The choice of prefixes are not stored in the markdown files.)
   * The continuation of the chat will be formatted differently from the existing exchanges.

3. Any change is going to confuse a lot users.  You'd be surprised at how many gptel users believe the current heading/body prefixes are part of gptel's "syntax" for differentiating between prompts and responses.  Most users, even Emacs users, don't look at customization options or read the README beyond the bare minimum needed to install the package[1].

4. I don't know what good defaults should be.  If change it to `@user: ` and `@assistant: ` or something like that, this will make problem 3 worse, as this looks even more like special, required syntax.

These are all important points. In my opinion all of them could be solved by one - unforunately - huge change:

• Give up on saving gptel--bounds in a “Local Variables” magic comment. While that approach is clever, it is very brittle: editing the chat-log with any (Heaven forbid!) non-Emacs will break it silently. Even using Emacs will break it (in a subtle and difficult to discover way - or how many people know “C-u C-x =”?), unless the user remembers to M-x gptel-mode or has found and set up how to enable it automatically.

• Instead, use something like you name in 4 and give the markers actual meaning (the model would still not see them). I think this should be standardized and part of the “format” (one could allow to modify it through local variables if this is really useful). I know that gptel is supposed to be free-form and work everywhere in Emacs. But these gptel-mode notebooks are different in that they already have a format anyway. The alternative of 4 is more robust and transparent.

• gptel-mode could make the different regions stand out in some way, and/or fontify the magic prefixes.

• Backwards-compatibility could be assured by notifying the user and automatically transforming from the old to the new format when gptel-mode discovers a notebook where gptel-bounds are set.

This is some pain and some work, but the result would be a huge improvement IMHO.

How about making this the actual default? Or something else that does not introduce an inconsistency between visuals and semantics, and that does not interfere with the document structure.

This tree-of-conversations feature only works in Org mode, not markdown.

I’m aware of the tree-of-conversations feature, but I didn’t even mean it here. (I also agree that it can be confusing.)

My thinking is that it is often useful to have (org or markdown) document structure in a “gptel-notebook” even without tree-of-conversations. This could be as simple as having a few sections that split up the conversation into subsequent sections. The prompt prefixes should not interfere with this.

[grothesque]

OK, I understand now that gptel also uses text properties in other buffers, not only in gptel-mode "notebooks". This complicates the story, since in generic buffers one does not want to introduce arbitrary markings.

But I think that this is not a problem: currently, text properties are only preserved in notebooks (through the local variables mechanism), not in other files. If the gptel "response" text property was preserved through markings instead of storing gptel-bounds under "Local Variables:" this would only mean replacing one preservation mechanism by another, more robust one.

I think that having special markers @gptel-user: and @gptel-assistant: that gptel-mode highlights visually would address your points 3 & 4. This change would be self-explaining and it could be introduced in a backwards-compatible way: gptel-mode could offer to convert old notebooks to the new format.

[grothesque]

After playing a bit with the effect of gptel text properties in non-notebook buffers, I fear that gptel's current model of using hidden text properties is more confusing than useful:

For example let's say that the user is in a programming mode buffer and asks the LLM to write a function. The function code is inserted and invisibly marked as a response. Now if the user modifies this function by introducing some code in the middle, and later asks some question to the LLM about this function, gptel will split up the code of the function into "user" and "assistant" chunks, thus totally confusing the LLM. To make things slightly more confusing, this splitting up is not operational if one uses gptel's rewrite mechanism.

I understood all of this only because I enabled inspection.

My current thinking is that gptel's hidden text properties are not worth the complexity and potential confusion. Here is an idea about how I believe the operation of gptel could be made easier to understand and more powerful at the same time:

• Back-and-forth conversations would be reserved to explicit gptel-mode conversation buffers that use explicit markers for distinguishing replies and responses.

• Outside of that, gptel would operate without markers. It should be possible to unify regular directives and gptel-rewrite for an easier to understand and at the same time more powerful mode of operation: any gptel directive would work in the same way: the LLM receives the (possibly narrowed-down) buffer. If a region is active, the LLM is asked to replace it. If no region is active, the LLM is asked to add text at point. Thus, most (if not all) "system messages" could work in completion or replacement mode.

For example, if the system message is "Assist in composing a letter. Only output letter text, without any explainations." and the user wants some section to be edited, they would mark the section, and the directive (knowing that region is active), would emit an additional request to "reformulate the highlighted region". If no region is active, the request would be to suggest some text based on the surroundings.

Note that in this new way of operating, Emacs' narrow-down functionality would be used to restrict what the LLM gets to see of the buffer. The region is used to indicate what part should be replaced. Using both narrow-down and region/point is the key to allowing more powerful/unified/consistent directives in a transparent way (=you get what you see).

The way static directives could be specified is by the user providing three strings: one general instruction, one additional instruction for the case that the region is active and one for when it is not. The second or third string could be nil meaning that this directive requires an active region (or requiring the region not to be active). In this way a consistent user interface could be established for all directives.

It seems to me that these simple directives would already allow to cover many use cases in a transparent way. They would allow to unify rewrite and non-rewrite. Explain-the-thing-at-point could be realized as a directive as well. Of course gptel-request is much more powerful, but the point here is only to improve on gptel-send.

Note that with current gptel-send, even with dynamic directives, something like explain-the-thing-at-point is difficult, because gptels-send will either send all text up to point, or send the region. And gptel-rewrite, which already is not all-to-well integrated with old-style gptel-send, makes it difficult to provide context around the region to be rewritten.

Note also that with my proposal it would be still possible to use gptel quickly in any buffer. For example replacing a region of the minibuffer would work very naturally - it would no longer require pressing i in the transient.

I'm sorry if my ideas are too radical or unrealistic. (Sometimes, ideas of a newcomer can be useful.) In any case, I would appreciate your comments on these ideas.

[karthink]

There are many suggestions here, so I'll respond in detail when I have time. There are several prior discussions about all these topics, I suggest searching the issues and discussions page for "tracking", "track", "whitespace", "prefix" and org-mode, among other terms.

See also #249, #321, #343 among others. (Feel free to skip past the many parts that don't have to do with the issues you bring up in this thread.)

[grothesque]

Thanks for the pointers. (Looking forward to your response!)

Here is just some more context for what I wrote above:

I tried to use gptel-split-buffer-directive that you suggested in another issue, but I noted that now (gptel 0.9.7) gptel-send seems always to add the region to the query, so I modified it to:

(defun my-gptel-split-buffer-directive (&optional system-prompt)
  (list (or system-prompt (alist-get 'default gptel-directives))
        (format "I'll provide a document with a highlighted section.
The code is in %s.
The answer should be specific to the highlighted section,
but use the rest of the text as context to understand the patterns and intent."
                (gptel--strip-mode-suffix major-mode))
        "What is before the highlighted section?"
        (buffer-substring-no-properties (point-min) (region-beginning))
        "What is after the highlighted section?"
        (buffer-substring-no-properties (region-end) (point-max))
        "What is in the highlighted section?  (I will work on this text.)"))

This seems to work (assuming that the region is active), but in interactive usage I haven’t found a way to change the system message which is always the default one. It seems like I would have to provide different versions of this directive for different roles.

Such experiments made me thinking about how gptel-send and the associated “directives” could be made more orthogonal. I noted in particular:

• gptel-rewrite feels like an afterthought. It reuses the same “directives” mechanism, but directives for rewrite do not necessarily work with regular gptel-send and the other way around.

• In your introductory video, when outside of a dedicated chat buffer, you use gptel-send almost always with an active region.

• Outside of chat, without an active region, actions like “tell me something about the thing at point” or “suggest something to be added at point” seem natural, but the choice of never providing context after point (motivated by the chat functionality) seems to be restrictive here.

• It could be useful to - at the same time - indicate a region for rewriting and restrict the context within a buffer. Emacs’ own narrowing functionality (C-x n n) seems to be a very natural answer to this problem.

[karthink]

Addressing a couple of your points here (out of many, so I'll respond more later).

...but in interactive usage I haven’t found a way to change the system message which is always the default one. It seems like I would have to provide different versions of this directive for different roles.

I don't follow. Do you mean that you haven't been able to change the system message dynamically, from gptel's menu?

gptel-rewrite feels like an afterthought. It reuses the same “directives” mechanism, but directives for rewrite do not necessarily work with regular gptel-send and the other way around.

All "directives" are the same kind of object: a system message + a canned conversation. That some are useful for rewriting and some for other kinds of interaction doesn't make them different in kind. You can have as many rewrite directives as you want, fit for different purposes. Here's a rewrite directive I use when the task needs more context:

  (defun my/gptel-code-infill ()
    "Fill in code at point based on buffer context.  Note: Sends the whole buffer."
    (let ((lang (gptel--strip-mode-suffix major-mode)))
      `(,(format "You are a %s programmer and assistant in a code buffer in a text editor.

Follow my instructions and generate %s code to be inserted at the cursor.
For context, I will provide you with the code BEFORE and AFTER the cursor.


Generate %s code and only code without any explanations or markdown code fences.  NO markdown.
You may include code comments.

Do not repeat any of the BEFORE or AFTER code." lang lang lang)
       nil
       "What is the code AFTER the cursor?"
       ,(format "AFTER\n```\n%s\n```\n"
               (buffer-substring-no-properties
                (if (use-region-p) (max (point) (region-end)) (point))
                (point-max)))
       "And what is the code BEFORE the cursor?"
       ,(format "BEFORE\n```%s\n%s\n```\n" lang
               (buffer-substring-no-properties
                (point-min)
                (if (use-region-p) (min (point) (region-beginning)) (point))))
       ,@(when (use-region-p) "What should I insert at the cursor?"))))

(You could achieve the same effect by adding the buffer to gptel's context and using the default rewrite directive, but that gets tedious.)

That said, the discussion in #375 hasn't concluded and meain's idea of templated interactions hasn't been implemented in full yet, so the way that instructions are specified for rewrites will probably change some more.

Such experiments made me thinking about how gptel-send and the associated “directives” could be made more orthogonal.

Yes, gptel-send is quite rigid in how it does things. My original idea was that gptel-send should work in an unambiguous, "obvious" way. For more bespoke functionality, the user can write a little elisp helper with gptel-request, which makes no assumptions about workflow or UI. While people have written many helper functions of this kind (and even separate packages), there's clearly some dissatisfaction with the rigidity of gptel-send. The discussion about templates in #375 is an attempt to address this. Feel free to add to that discussion as well.

[grothesque]

...but in interactive usage I haven’t found a way to change the system message which is always the default one. It seems like I would have to provide different versions of this directive for different roles.

I don't follow. Do you mean that you haven't been able to change the system message dynamically, from gptel's menu?

No, I mean that I can activate my split directive, and it works, but then I’m not able to vary the system message, because choosing the system message and choosing a directive is the same menu point.

Here are the details:

In order to enable the directive function listed above, I use: (add-to-list 'gptel-directives '(split . my-gptel-split-buffer-directive)).

Now let’s try it out on some short shell script. I do the following:

• Set region to one line of the script that I’d like to understand,
• fire up gptel-menu,
• press s p to select my “split” directive,
• press d to add instructions: “Please explain”,
• press e to see the response in the echo area.

And I get a useful response, as expected.

For reference, the following query is sent in the background:

(:model "gpt-4o-mini" :messages
        [(:role "system" :content "You are a large language model living in Emacs and a helpful assistant. Respond concisely.

Please explain. ")
         (:role "user" :content "I'll provide a document with a highlighted section.
The code is in Shell.
The answer should be specific to the highlighted section,
but use the rest of the text as context to understand the patterns and intent.")
         (:role "assistant" :content "What is before the highlighted section?")
         (:role "user" :content "#!/bin/sh

user=$1
shift
")
         (:role "assistant" :content "What is after the highlighted section?")
         (:role "user" :content "xhost +si:localuser:$user
su $user -c \"$*\"
")
         (:role "assistant" :content "What is in the highlighted section?  (I will work on this text.)")
         (:role "user" :content "trap \"xhost -si:localuser:$user\" EXIT INT TERM")]
        :stream t :temperature 1.0)

So far so good, but I’d like to be able to change the system message in the query to something else. Perhaps I feel a bit strange and would like to tell the LLM “You are a poet. Reply only in verse.”, but otherwise keep everything as above.

I understand that the comment from which I took gptel-split-buffer-directive says that it is for use with gptel-request, like in:

(gptel-request (read-string "Rewrite instructions: ")
  :system 'gptel-split-buffer-directive
  :in-place t)

Thus, since I added my version of the directive to gptel-directives, there’s no point in keeping the (&optional system-prompt) argument - this is simply work in progress.

My point is that there doesn’t seem to be a way to specify the system message independently from the directive.

---

Note that the unfinished directive my-gptel-split-buffer-directive does not at all implement my proposal of interpreting an active region as text to be replaced. It’s just an exercise with the capabilities of current gptel.

---

Thanks for my/gptel-codef-infill, it looks very useful. But it also illustrates my above point, I believe. I think that orthogonally to selecting a directive like yours it would be useful to be able to set the “system message”. For example one might want to specify that the generated code should contain a lot of comments, or no comments at all, without the need for two separate but very similar directives.

Going further, gptel already varies the default system message based on the major mode of the buffer. It would be great if there was a code-infill-directive that would work both for email text and for Python code, because the mode-specific bit would be automatically set through the system message, and one could still choose a directive (and also modify the system message if one so chooses).

[karthink]

No, I mean that I can activate my split directive, and it works, but then I’m not able to vary the system message, because choosing the system message and choosing a directive is the same menu point.

Please use #375 to discuss this particular idea/workflow. We can continue in this thread the discussion of other topics you've brought up.

[grothesque]

No, I mean that I can activate my split directive, and it works, but then I’m not able to vary the system message, because choosing the system message and choosing a directive is the same menu point.

Please use #375 to discuss this particular idea/workflow. We can continue in this thread the discussion of other topics you've brought up.

Thanks. I will have a closer look at #375.

Within this issue, it is not my purpose to discuss templates and dynamic directives. It is rather that I got into playing with gptel's dynamic directives as a vehicle for experimenting with my ideas for an alternative mechanism for static directives.

Still, here is one short question which IMHO fits best into this thread: is the example directive my/gptel-code-infill that you show above meant to be added to gptel-directives? If yes, I am a bit confused: when I use your directive in this way, the first half of the buffer (before point) will appear in the query twice: a first time because the directive injects it, and a second time because gptel adds the contents of the buffer up to point if region is not active. Is this by design? Or am I using it incorrectly?

[grothesque]

Returning to the topic of exploring alternative mechanisms for static directives, let me try to summarize my points - I fear that they got a bit mixed up in the conversation above, in particular because my understanding of gptel evolved during the discussion. This comment tries to distill my previous ideas, but contains also some new ones towards the end.

Disclaimer: for simplicity, I frame the following as a concrete proposal. I am aware that I've been using gptel only for a couple of days, so I am not in any way trying to impose anything. I see this just as a form of brainstorming, and hope that others here may find some of these ideas useful. I would greatly appreciate getting feedback about this proposal.

I really like gptel’s basic premise of being a simple and general tool! I think that this fits Emacs and LLMs very well. My proposal aims to get even better at this.

• I believe to understand the rationale for how gptel-send works: the hidden text properties and the basic behavior of gptel-send (without active region) are just what is needed to implement a basic chat in any buffer. When region is active the behavior is different: this allows queries concerning specific parts of the buffer.

• However, I tend to agree with @jwr that chat outside of dedicated chat buffers is not particularly useful. In any case I haven't seen convincing examples of chat outside of a dedicated chat buffer. Even if there were such examples, chat in general buffers would be highly confusing, because the invisible text markings that are necessary for chat do not survive writing the file to disk, and are also invisible.

• Now comes the crucial point: if we accept that chat should take place only inside dedicated buffers, we can use a more reliable and transparent mechanism for separating answers and replies. This has a double advantage: (1) it avoids the reliability problems with current chat buffers that I mentioned previously (e.g. that the user needs to remember to enable gptel-mode or that editing a chat log outside of Emacs will corrupt in an invisible way), and (2) it frees us from the constraints that supporting chat everywhere has on gptel's way of working outside of chat buffers (text is sent only up to point, invisible text properties play a role).

• If one accepts the above points, one can design two modes of operation for gptel (where the hope is that each is simpler and better):

(1) gptel-chat-mode: a mode for conversations with an LLM. Answers and replies are marked up with some clear and visible markers. C-u RET is bound to a command that sends all the text up to point, split-up according to the markers. For backwards compatibility, gptel-mode, whenever entered, could propose converting old-style chat buffers to gptel-chat-mode.

(2) A general-purpose command similar to gptel-menu: Since it does not have to support chat, we can think of what would be a simple yet powerful general-purpose way of interfacing with a LLM in Emacs. Ideas for this command follow after the separator.

---

Let’s call this command gptel-interact.

• Since gptel-interact does not have to support chat, the transient menu can be shown by default.

• Of course gptel-interact can be used to good effect inside gptel-chat-mode buffers. (And this is actually useful, since the directive for the chat has potentially nothing to do with the directive for gptel-interact even if used inside the chat buffer.) gptel-intract in chat buffers does not treat the chat markers in any special way.

• gptel-interact supports both static and dynamic "directives" (not sure whether this is a fitting term, but let’s use it here). Dynamic directives are clearly nice to have, but let’s try to hash out a useful mode of operation for static directives, which are simpler to understand and to create (no elisp required, can be created on the fly).

• Since gptel-interact does not have to support chat, it is not obliged to send text only up to point. Often, text beyond the point can be useful to see for the LLM.

• As @meain observes, a conversation with a LLM tends to produce better results than a system message followed by a single chunk of input. It would be great if we could take advantage of this even with static directives.

• It is often useful to limit the attention of the LLM to only a part of the buffer. Emacs built-in narrowing functionality seems perfect for this. It works even in the minibuffer, and it is very clear (literally what you see is what you get). This lets us use the region for a different purpose than limiting attention to part of the buffer.

• What purpose could an active region have? To me the natural application is to focus the attention of the LLM on a part of the visible section of the buffer.

• If the region is not active in Emacs, the natural bit of additional information is the position of point. This could be used as the place to insert some new text (for example completion), or as a particular point of interest. (But sometimes it could be also useful if point plays no role.)

• Like now the transient would offer options to put the output of the LLM in-place, or externally (echo area, other buffer). In inplace mode, if region is active, it would be replaced by the output (with support for diffing etc. like gptel-rewrite), if region is not active, the output would be inserted at point. Instead of a single command might be better to have two separate commands, perhaps called gptel-inside and gptel-outside. This would simplify the interface because gptel-inside would need no option for where the output should go. gptel-outside could default to the echo area. Furthermore, each command could have different default directives and could also remember the last directive separately. For example the directive "suggest" is a good default for gptel-inside, and the directive "explain" is a good default for gptel-outside. Now if the user is writing prose, they could set gptel-outside to "synonyms and antonyms" while maintaining the autocomplete behavior of "suggest" with gptel-inside.

• In addition to directives, which describe what to do, there would be a "role" message which describes the setting. This message would be set by default depending on the major mode, e.g. "You are an Emacs lisp programmer." or "You are a text writing assistant.", but could be modified by the user.

• The directives would be as orthogonal to the role messages as possible, i.e. ideally most role messages would make sense with most directives. The default directive could be "suggest" . Other directives could be "explain". But I’m not sure about this point. Perhaps it’s better to combine roles and directives like this is done currently.

• As mentioned in comments above, static directives could be simply pairs of strings: one string to be used if region is active, the other when not.

[karthink]

@grothesque Thanks for putting your thoughts together into a single post, that makes it easier to follow. I'm currently busy addressing bugs and issues opened in the past week, and couldn't get to this topic (which deserves a long response) in time this weekend. Posting here to mention that I will reply -- haven't lost track of this!

Re: some of your points, you may want to look at gptel-track-response. If you haven't set gptel-expert-commands yet, you should do that too: (setq gptel-expert-commands t) and use the transient menu.

[lispy-ai]

Wrote #565 in partial response to this thread.

[grothesque]

@grothesque Thanks for putting your thoughts together into a single post, that makes it easier to follow. I'm currently busy addressing bugs and issues opened in the past week, and couldn't get to this topic (which deserves a long response) in time this weekend. Posting here to mention that I will reply -- haven't lost track of this!

Sure, take your time. I'm very interested in your thoughts on the topics I raised. Even if you don't want to alter the way gptel works, I might be able to write something myself that uses gptel-request as the backend.

Re: some of your points, you may want to look at gptel-track-response. If you haven't set gptel-expert-commands yet, you should do that too: (setq gptel-expert-commands t) and use the transient menu.

I've set gptel-expert-commands to t already so that I can review queries. (That said, I think that it would be even better if inspecting queries would work by creating a new chat buffer - ideally this would allow to save and later replay complete queries, so the system message would also have to be saved in the chat buffer.)

Thanks for the pointer to gptel-track-response. This might be useful to me, but I'm even more interested to understand in which ways chats in regular buffer (say code) can be useful.