gptel [63/105]

file:///nix/store/kmwlm9nmvszrcacs69fj7hwpvd7wwb5w-emacs-30.1/share/emacs/30.1/lisp/eshell/esh-mode.el.gz#MissingReference

From the dry-run output menu, and perhaps even from the transient menu.

Should there be a gptel-debug command?

file:///home/karthik/.local/share/git/elpaca/repos/gptel-main/gptel-transient.el#MissingReference

Unfortunately it’s not (easily) possible to track the suffixes hidden/disabled groups when exporting the prefix. For example, in the below example if the “w t” suffix is hidden because the group is hidden, “w” cannot be exported even if it was selected before hiding the group. (To hide the group, use the second “z” option.)

(setq toggle nil)
(transient-define-prefix testprefix ()
  "TODO"
  :refresh-suffixes t
  :transient-suffix #'transient--do-call
    [:description "Test"
     :class transient-subgroups
     :setup-children
     (lambda (_)
       (transient-parse-suffixes
        'testprefix
        ;; (list [(:info "Test1")])
        (list [:if (lambda () toggle) ("w t" "Test1" "w")]
              [("z" "Test2" (lambda () (interactive) (cl-callf not toggle)))]
              [("y" "Test3" "y")
               ("RET" "Done" (lambda (args) (interactive (list (transient-args transient-current-command)))
                               (message "%S" args)))])))])

These are intended to be different sections of the same manual (single document).

The user manual should contain

• [ ] a brief explanation of what gptel is
• [ ] a brief overview of how LLM APIs work
• [ ] explanations of gptel’s features
• [ ] along with customization relevant to each feature
• [ ] explanation of (WIP) RAG and tool use

The developer manual should include

• [ ] building on gptel: examples of gptel-request usage
• [ ] a brief explanation of gptel’s design goals, focusing on
  • its scope: what kinds of features will/won’t be added
  • philosophy: integrate with Emacs, reduce cognitive burden, don’t introduce syntax
• [ ] a brief overview of gptel’s main control flow and state management
• [ ] data structures (plists, plists everywhere!)

Resources:

• Uniline for drawing line diagrams for info?

[BROKEN LINK: No match for fuzzy expression: copy-marker tracking-marker nil)))))))]

https://github.com/mwolson/emacs-shared/blob/master/init/shared-init.el#L699

I also ended up writing small wrappers starting here to assign each of the following to different keybinds, some of which could potentially be made a bit more streamlined:

• Add current function to context (uses the mark-function that I wrote in the gist)
• Add the current file to context
• View all context (note - it would also be nice if the cursor was positioned a bit forward so that hitting “d” would immediately mark the first item, currently “d” just moves to the first item without marking it)
• When the view context UI is opened, assign a key to apply changes and quit without further prompts
• Add the current function to context and immediately open a split gptel window off to the right to ask a question about it
• Start gptel itself without any transient menus, immediately getting into a gptel buffer for the default backend and model

#556 Customization of `gptel-backend` causes recursive require error, prevents gptel from loading #529 Instructions for adding to the list of Gemini models

The idea is to allow gptel-backend to be a string (that’s the name of the backend). In gptel-request, we can check the type of the variable:

(let ((backend
       (or (cl-etypecase gptel-backend
             (gptel-backend gptel-backend)
             (string (alist-get gptel-backend gptel--known-backends
                                nil nil #'equal)))
           (user-error "Backend \"%s\" is not known to be defined"
                       gptel-backend))))
  ...)

We just have to be sure that we do this everywhere gptel-backend is let-bound. Alternatively, we can use gptel--backend as the actual backend and gptel-backend can always be a string.

We may also need a gv-settable gptel-get-backend that can be used in situations like:

(push 'gpt-4o (gptel-backend-models (gptel-get-backend "ChatGPT")))

Disadvantages of using text-properties for this:

• Roles are invisible
• If the text-properties are sticky, self-inserted text in a response region is assigned the response role, but yanked or otherwise inserted text isn’t.
• If the text-properties are non-sticky, there’s no way to modify the response text at all.
• Persistence across Emacs sessions is fragile (requires gptel-mode to be turned on)

There are several proposals for improving this:

• #321 How should gptel distinguish between LLM responses and user text?
• #343 Add response color-coding & role setting for gptel buffer
• #565 Use zero-width delimiters for role tracking in gptel-mode
• #546 Naive ideas for making gptel simpler yet more powerful

In short, they suggest:

• Using syntax, i.e. user-defined prompt and response prefixes in chat buffers.
• Using zero-width delimiters
• Providing response-highlighting and role-setting commands
• Using overlays

This is required for packages like gptel-openai-assistant to reuse gptel-menu.

The idea is to allow a little before-string before each context overlay, then incldue that before the chunk when creating the context. I’m not sure yet how to make this feature discoverable.

• Calling gptel-add with a prefix arg, or using a keybinding at the context overlay, should allow us to attach a description or instructions
  • gptel-context-annotate bound to C-c C-a
  • gptel-context-clear bound to C-c C-k
• This annotation should then be included with the chunk when creating the context string.
• This annotation doesn’t need to be stored as a variable, we can store it in the overlay and display it as a before string, maybe.

Claude

JSON array of strings that will cause the model to stop generating text if encountered, specified at the top level of the API request(?)

[BROKEN LINK: No match for fuzzy expression: I don't want to populate the menu with even more options, at least for now. Dumping the full prompt if it's a new gptel session sounds like a good compromise to me, I]

Tool use [0/0]​

https://old.reddit.com/r/emacs/comments/1i0vsz9/tech_demo_completing_functions_using_gptel/

https://old.reddit.com/r/emacs/comments/1i0vsz9/tech_demo_completing_functions_using_gptel/mc8ponx/

There is one small thing that would make it slightly easier to use some of the transient-wrapped functions in gptel-rewrite without undergoing too complex a change. Using gptel–suffix-rewrite as an example, it would be nice to have the code from (interactive …) onwards moved into its own named function, so that the transient-define-suffix is just a key, description, and function call. This would provide easier access from .emacs config and other libs to be able to directly call those named functions and skip the initial transient entry point. I basically forked that function to get that kind of access.

It will be interesting to see how the context changes turn out. Hopefully there will remain a good way to visualize and edit context outside of a chat buffer for flows that want to skip having one (or at least, not show one immediately to the user).

What I meant specifically about opening a chat session is that just calling the gptel function will prompt about which backend to use, which I currently have to write a small wrapper to be able to skip. It has to do: * Get the backend name and format it with “%s” * Call (gptel formatted-backend-name nil “###”) * Call switch-to-buffer since the buffer doesn’t get raised when called non-interactively (and args can’t get passed interactively AFAICT)

Maybe a helper function named like gptel-start or similar could help differentiate the programmatic and interactive cases? Variables to control backend prompting is another possibility.

Since I don’t want to make automatic network requests to fetch model information from the API,

#447 Enumerate Ollama models on the server pointed to by :host directive #567 Determine which models exist with API calls rather than hard coding them.

here’s an idea.

• Define a cl-defgeneric that dispatches on the backend type, and updates the model list for gptel-backend or any backend.
• Define a new command, gptel-update-models, that calls these methods.

Now the user can run this whenever/wherever they want to update the list of models, like in a hook.

The github mcp server can still fail to start without triggering the error state in the connection.

#261 Losing chunks of replies

Retrieval Augmented Generation, i.e. automatically fetching context for gptel requests.

It’s not clear yet how to do this. Just like we want to make gptel-tools work with externally defined tool collections, I think it will be good to make RAG a plug-in option. There are many sources for RAG.

file:///tmp/APAC_meetup.html

So some of the code involving formatting spaces in headers and annotation functions can be simplified using format.

#288 Make `gptel-expert-commands` customizable

#235 Feature request: setup an undo-boundary

The problem with this is coordinating async insertions into the undo change group – it’s tough to do.

#283 feature request: additional curl arguments

We can ask them if they want to contribute it to gptel.

https://github.com/certainty/madmacs/blob/main/modules/tools/madmacs-tools-ai.el

So that the user doesn’t need to press “d” to get started.

Different context sets are useful in different chat buffers (or regular buffers), so it would be good to have a way to specify this. With a single global set of contexts, you often have to engage in the add-context/dump-context dance.

• #481 Media (and other) links.
• #475 Per-gptel-buffer contexts

The plan we’ve arrived on is to add buffer-local context via links to files in Org/Markdown mode buffers – this doesn’t work for buffers or regions, unfortunately, unless we add a link type.

• [ ] Add buffer-local context support
• [ ] Find a way to indicate that a link will be included in the context, perhaps the activate-func Org link property, and something bespoke for Markdown?
• [ ] Send text file links along with media files

tool-test.org

Jester7’s minimal gptel tools Jester7’s gptel tools

• [ ] Update Juri Linkov about gptel-request tools callback API change
• [ ] An explicit indication of how many tools were called (Where do we put this?)
• [ ] Allow editing tool calls from the confirmation prompt
• [ ] Support (gptel-get-tool '(("emacs" ("read_buffer" "open_file"))))
• [ ] README: Add instructions for tool-use with OpenWebUI.
• [X] Go through the model list and assign tool capabilities correctly.
• [ ] Then ensure that the tool options in the menu only show up when appropriate.
• [X]

  Log arguments as well: (truncated arguments)

  Changed the tool call API:

  • tool call confirmation: callback called with ((tool arg-values callback) …)
  • tool call result: callback called with ((name arg-values result) …)
• [X] Juri Linkov’s tool-use logs, reply
• [X] Decide on whether we want t, auto and nil, or force, t and nil for tool-use options.
• [X] Ollama: Turn off streaming automatically if tools are used This is done in a hacky way right now. Eventually I need a better method.
• [X] Tool call failures for async calls, should we handle them? This is difficult to handle.
• [X] PrivateGPT tool use with/without streaming Doesn’t support, AFAICT
• [X] README: Add demos for tool use.
• [X] README: Add explanation of tool use.
• [X]

  Handle tool use confirmation and result inclusion in the custom callbacks used by gptel--suffix-send. Perhaps we can pop up a buffer that displays the confirmation? Or just use the minibuffer? We can try to reuse the existing mechanism used in dedicated chat buffers, perhaps with some temporary function advice.

  Ended up using the minibuffer. It’s good enough for now.

• [X] Handle non-string, structured return values from tools. Tools can return only simple types or JSON objects/arrays. These are returned as plists/arrays, processed via gptel--json-read.
• [X] tool specification for tools that don’t need to be run We just specify :confirm t in the tool specification. You can still send it if required that way.
• [X] How do we control including tools in gptel? (gptel-use-tools)
• [X] Configuring tool calls: auto, always, any etc
• [X] Handle tool call failures (wrap with condition-case-unless-debug?)
• [X] Test OpenAI compatible APIs like Groq
• [X] Don’t run post response hook etc when a request ends and we’re waiting on a tool call
• [X] Ollama tool path with streaming This is not supported, but we need to turn off streaming automatically when tools are used. (Ollama returns a non-streaming JSON response.)
• [X] Tool creation/appending code in gptel-request
  • Check for model-tool capability
• [X]

  Ollama tool path without streaming (see note) Gives me the following error:

    "message": "tools.0.type: Extra inputs are not permitted"
  

  But it works once in a while! What the hell: Error in tool object construction, now fixed.

• [X] Gemini tool path without streaming
• [X] Gemini tool path with streaming
• [X] Anthropic tool path without streaming
• [X] Anthropic tool path with streaming
• [X] OpenAI tool path without streaming
• [X] OpenAI tool path with streaming
• [X] Kagi tool path without streaming: No Kagi support
• [X] Kagi tool path with streaming: No Kagi support

https://github.com/karthink/gptel/issues/323#issuecomment-2632054061

• [X] Reply to Joe’s comment about FIM completion

• Move all the networking code to gptel-request.el (renamed from gptel-curl.el) and make it standalone.
• gptel.el will contain all the non-transient UI code.

This will make it easier for third party packages to use only gptel-request via (require 'gptel-request), and they can safely ignore the rest of gptel. This can be spun out into another package at some point, although naming issues will cause trouble.

No change to the other files.

The idea is to allow entries in gptel-directives to be

• strings (as usual)
• lists (for templates)
• functions that return strings or lists of strings

This allows for

To allow for features like https://github.com/karthink/gptel/issues/328, always create the prompt in a temp buffer. That way, we can add a list of filters to transform the text before sending, like Org’s export filters.

One commonly requested feature is removing the PROPERTIES drawers from the prompt in Org buffers before sending the request.

Claude

Via tool-use

OpenAI

Via tool-use or a “structured-output” mode

We need a way to specify per-run pre- and post-response hooks, as well as transformation hooks.

This is an oft-requested feature.

• How Aider does it, and Aider’s edit formats

https://imgur.com/a/anCrybB

ert#erts files

#592 Include DeepSeek’s reasoning_content when present

To handle multi-turn requests, like those required for tool-use, we need something more robust than shoving function calls into callbacks. Unfortunately we don’t have access to a good promise API or to org-async right now¹.

I may need to write a process executor ☹️.

• [X] Merge!
• [X] Create a gptel-send / gptel--suffix-send specific transition table
• [X] Separate the user messaging in gptel-send from the callbacks Almost – only the “Typing…” instruction is part of the process filter.
• [X] Reuse the WAIT state after calling a tool
• [X] Make gptel-request return just the fsm, no callback.
• [X] Make gptel-request standalone.
• [X] Rename fsm callbacks to fsm handlers
• [X] Use a Finite State Machine (fsm) to drive gptel-request.

There are a few things we need.

This is a collection of features supported by different APIs that I’d like to support on a model-level basis in gptel. I need to figure out how to structure the symbol-plist of gptel models (which will now be symbols, not strings)

If all APIs support a feature (like stop_tokens) we don’t need specialized behavior for it, we can just add it to the transient menu directly.

As a precursor to adding vision support, function calling support, fill-in-middle support etc, we need a more granular way to specify model capabilities. The idea is to

• make gptel-model a symbol instead of a string,
• whose symbol-plist will list the capabilities of that model,
• which will be consulted when preparing the prompt

This will also require more prompt construction method for the various APIs.

#299 Perplexity: citations

Instead of writing new response parsers wholesale for Perplexity, it might be possible to piggyback on the OpenAI parsers using cl-call-next-method. Writing a response parser is significantly more work now that we support tool-use.

Email from Anthropic Support: Claude can now use tools

• [-] Integration with gptel-complete
• [X] Branching conversations in Org

• Note taken on [2023-12-19 Tue 22:44]
  string-edit is Emacs 29.1+ only.

• Note taken on [2024-03-12 Tue 00:16]
  Unfortunately this doesn’t work: there are too many places in the call chain that refer to gptel-backend et al, so it’s not sufficient to add extra defmethods for gptel--parse-buffer and gptel--request-data. Some of the other places include gptel-send and gptel--suffix-send (for messaging the user), and gptel--url-get-response and gptel-curl--get-response.

  Another problem with this approach: The org properties will have to be scanned for again by each defmethod.

  The advice method is probably the best we can do.

In the feature-org branch: Currently the prompt parsing functions in gptel are advised to find the gptel parameters as Org properties if in Org mode. We can do away with this by using the qualifiers provided by cl-defmethod, see elisp#Generic Functions. This does the same thing as advice but is cleaner in the code. Might be harder to debug though.

• Note taken on [2024-04-23 Tue 18:49]
  Switched to the Ollama chat API so there is no longer a context vector.

#279 (Ollama) Old context information being sent after switching to a different model

https://github.com/karthink/gptel/discussions/156#discussioncomment-8896600

This can be an add-on package – gptel-copilot.

The main challenge is going to be finding a way to include enough context.

• C-g will abort the whole thing.
• C-= will launch a diff/ediff.
• Doing anything else will accept the completion.

[BROKEN LINK: No match for fuzzy expression: ;;; hierarchy.el — Library to create and display hierarchical structures -*- lexical-binding: t; -*-]

https://github.com/karthink/gptel/issues/683

#709 Unable to determine the end of reasoning

I check for the end of the reasoning block like this:

(when-let* ((content (plist-get delta :content))
            ((not (eq content :null))))
  (if (plist-member delta :reasoning_content) ;Check for reasoning model
      (plist-put info :reasoning-block t) ;End of streaming reasoning block
    (plist-put info :reasoning-block 'done)) ;Not using a reasoning model
  (throw 'done t))

This requires that the response continue to have :reasoning_content blocks after the reasoning content is done, with the value of the key set to :null. This is how we check if a reasoning model is being used. However, this is not the case in the attached log, where there is no :reasoning_content key at all. As a result, :reasoning-block is never being set to t.

log around end of reasoning block:

data: {"choices":[{"delta":{"content":"","reasoning_content":".","role":"assistant"},"index":0}],"created":1742127218,"id":"021742127218161a94c19b4b22e17b450210441606b508df776fc","model":"deepseek-r1-250120","service_tier":"default","object":"chat.completion.chunk","usage":null}

data: {"choices":[{"delta":{"content":"","reasoning_content":"\n","role":"assistant"},"index":0}],"created":1742127218,"id":"021742127218161a94c19b4b22e17b450210441606b508df776fc","model":"deepseek-r1-250120","service_tier":"default","object":"chat.completion.chunk","usage":null}

data: {"choices":[{"delta":{"content":"","role":"assistant"},"index":0}],"created":1742127218,"id":"021742127218161a94c19b4b22e17b450210441606b508df776fc","model":"deepseek-r1-250120","service_tier":"default","object":"chat.completion.chunk","usage":null}

data: {"choices":[{"delta":{"content":"\n\n","role":"assistant"},"index":0}],"created":1742127218,"id":"021742127218161a94c19b4b22e17b450210441606b508df776fc","model":"deepseek-r1-250120","service_tier":"default","object":"chat.completion.chunk","usage":null}

#452 Claude error: (HTTP/2 400) messages: text content blocks must contain non-whitespace text

Other packages can find it very useful to pick and choose components from gptel menus for their own purposes. However there doesn’t seem to be a way to do it.

https://github.com/magit/transient/discussions/299

Here’s how the backtick parsing works in Markdown:

1. A verbatim/src block can begin with any number of backticks.
2. If it begins with one backtick
   • this backtick is not followed by another backtick.
   • The contents of the verbatim block can include any number of backticks.
   • The end of the block is signalled by a backtick that is not next to a backtick.
3. If it begins with multiple backticks
   • Any sequence of fewer backticks is part of the verbatim block
   • A sequence of as many backticks ends the verbatim block

So here’s how we set in-src-block:

If we see a backtick, check if the char-after is a backtick

If yes,

• hold on until we see something other than a backtick,
• then set in-src-block. Keep count of the number of backticks.
• If we see this many backticks again, turn off in-src-block.

If no,

• set in-src-block.
• If we see another backtick, wait to make sure that it’s isolated – i.e. not preceeded/followed by a backtick.
• When this is the case, turn off in-src-block

gpt-4 has started using ### headings in its answers. Convert to Org headings or do something with them.

https://github.com/karthink/gptel/issues/184

And the ones I’m not adding and why.

It would be great to be able to (e)diff the last thing added to the buffer by the LLM. It would be even better to ediff across all previous states, but I’m not sure how to do this – some kind of integration with undo?

For one-step ediffs, here’s the info we need:

1. state of the buffer before gptel killed some text, if any.
2. state of the buffer after the response.

Other options: hook into after-change-functions – bad idea?

Since gptel’s edits to the buffer are local: (optionally) kills text and adds text at point, we can get away without diffing the whole buffer. (There may be user-driven edits in other parts of the buffer in the meantime that we don’t care about anyway.)

So:

• If killing text: keep the text in a buffer-local var
• In post-response-functions, add (at the end, at position 99), store the text in another buffer-local var. Or it could be a single cons containing the previous string (if any), along with other metadata.
• In the transient menu, we provide an option to (e)diff last LLM response in this buffer. When called… ?

To implement the feature described, the strategy seems plausibly efficient. Here are some potential steps:

• The functionality to (e)diff between two content states can be implemented as a separate function, perhaps gptel-ediff-last-response().
• You can make use of two buffer-local variables to keep track of the content state before and after the LLM’s edit. You can name these variables as gptel-last-pre-edit-content and gptel-last-post-edit-content respectively. Be aware to accurately update these variables whenever an edit operation happens.
• You would probably want a third buffer-local variable to keep track of the position where the LLM made its last edit (e.g., gptel-last-edit-position).
• Additionally, you might want to make these states persistent across emacs sessions for the users. You can use standard emacs lisp constructs like save-excursion to preserve point and other state in the process.
• In the gptel-ediff-last-response(), create two temporary buffers (or diff buffers). You can insert the content of gptel-last-pre-edit-content into the first buffer, and gptel-last-post-edit-content into the second buffer.
• After storing the content, use ediff-buffers or similar functionality to perform the diffing.

It will also be nice to have a mechanism to clear the tracked state, maybe a function gptel-clear-diff-state(). Also, consider adding a new key to the gptel’s transient menu which can be used to trigger this ediff operation. Users can also bind this function to a keystroke if they use it frequently.

By allowing users to trigger this diff function it would make it easier for them to visually see and understand what the LLM has changed in their content.

Remember to handle error states and edge cases gracefully - for example, what happens if a user tries to ediff when no edit operation has been made by the LLM yet (i.e., the buffer-local variables are null or undefined)? Effective user feedback and good documentation will be key to making this feature successful.

Finally, an additional feature to consider for the future might be to allow users to ediff between any two arbitrary past states, not just the most recent one. For this feature, a history of past states must be maintained. That could be a more significant design and performance endeavor, but it could also be a powerful tool for users.

• Note taken on [2024-02-02 Fri 19:48]
  

  buffer

  the buffer where the response is inserted

  position

  the position in :buffer= where the response is inserted

It’s not clear what these keyword args do, or even what they should do exactly.

Documentation: Rest API: https://ai.google.dev/tutorials/rest_quickstart Model parameters: https://ai.google.dev/docs/concepts#model_parameters

• Note taken on [2023-10-24 Tue 18:39]
  Settled for using cl-structs instead. Might still use defgeneric if the parsing differs considerably between backends.

This will make it easier to add support for

• Azure
• Dall-E
• Local LLMs, etc

https://github.com/karthink/gptel/issues/4

[BROKEN LINK: No match for fuzzy expression: ;; TODO: Handle this for ollama, see the new code in `gptel-curl–stream-cleanup']

Mainly the :header (lambda () bearer-token-stuff) needs to be automated, so that it can work with and without a specified key for that backend.

Email from Andrew Hyatt: LLM Experiments, Part 1: Corrections Email from Psionic K.: LLM Experiments, Part 1: Corrections

Email from Eli Zaretskii: Re: bug#68692: 29.1; File-local variable storage for strings with newlines

Email from Andrew Hyatt: Re: LLM Experiments, Part 1: Corrections

It includes the response prefix.

#261 Losing chunks of replies Raw data from filter-function reads

Email from Jan Rychter: Re: [karthink/gptel] Separate system prompts and directives (Issue #249)

How is gptel more complex than the web UI?

See this issue.

https://github.com/magit/transient/issues/287

#296 Org mode response formatting

Email from Johan Ekenberg: 31.0.50; Fontification malfunction

[BROKEN LINK: No match for fuzzy expression: :parts 0 :text)))]

#237 Weird interplay with purpose mode

Thank you, didnt know that. Mind you add this as an example to the docs, please? The docs are a bit misleading as it says:

The :key can be a function that returns the key (more secure).

So i was thinking i had to develop my own function.