gptel [51/83]

tool-test.org

Jester7’s minimal gptel tools Jester7’s gptel tools

• [ ] Allow editing tool calls from the confirmation prompt
• [ ] Support (gptel-get-tool '(("emacs" ("read_buffer" "open_file"))))
• [ ] If stop reason in max_tokens, send a message.
• [ ] README: Add instructions for tool-use with OpenWebUI.
• [X] Go through the model list and assign tool capabilities correctly.
• [ ] Then ensure that the tool options in the menu only show up when appropriate.
• [X]

  Log arguments as well: (truncated arguments)

  Changed the tool call API:

  • tool call confirmation: callback called with ((tool arg-values callback) …)
  • tool call result: callback called with ((name arg-values result) …)
• [X] Decide on whether we want t, auto and nil, or force, t and nil for tool-use options.
• [X] Ollama: Turn off streaming automatically if tools are used This is done in a hacky way right now. Eventually I need a better method.
• [X] Tool call failures for async calls, should we handle them? This is difficult to handle.
• [X] PrivateGPT tool use with/without streaming Doesn’t support, AFAICT
• [X] README: Add demos for tool use.
• [X] README: Add explanation of tool use.
• [X]

  Handle tool use confirmation and result inclusion in the custom callbacks used by gptel--suffix-send. Perhaps we can pop up a buffer that displays the confirmation? Or just use the minibuffer? We can try to reuse the existing mechanism used in dedicated chat buffers, perhaps with some temporary function advice.

  Ended up using the minibuffer. It’s good enough for now.

• [X] Handle non-string, structured return values from tools. Tools can return only simple types or JSON objects/arrays. These are returned as plists/arrays, processed via gptel--json-read.
• [X] tool specification for tools that don’t need to be run We just specify :confirm t in the tool specification. You can still send it if required that way.
• [X] How do we control including tools in gptel? (gptel-use-tools)
• [X] Configuring tool calls: auto, always, any etc
• [X] Handle tool call failures (wrap with condition-case-unless-debug?)
• [X] Test OpenAI compatible APIs like Groq
• [X] Don’t run post response hook etc when a request ends and we’re waiting on a tool call
• [X] Ollama tool path with streaming This is not supported, but we need to turn off streaming automatically when tools are used. (Ollama returns a non-streaming JSON response.)
• [X] Tool creation/appending code in gptel-request
  • Check for model-tool capability
• [X]

  Ollama tool path without streaming (see note) Gives me the following error:

    "message": "tools.0.type: Extra inputs are not permitted"
  

  But it works once in a while! What the hell: Error in tool object construction, now fixed.

• [X] Gemini tool path without streaming
• [X] Gemini tool path with streaming
• [X] Anthropic tool path without streaming
• [X] Anthropic tool path with streaming
• [X] OpenAI tool path without streaming
• [X] OpenAI tool path with streaming
• [X] Kagi tool path without streaming: No Kagi support
• [X] Kagi tool path with streaming: No Kagi support

So that the user doesn’t need to press “d” to get started.

• Move all the networking code to gptel-request.el (renamed from gptel-curl.el) and make it standalone.
• gptel.el will contain all the non-transient UI code.

This will make it easier for third party packages to use only gptel-request via (require 'gptel-request), and they can safely ignore the rest of gptel. This can be spun out into another package at some point, although naming issues will cause trouble.

No change to the other files.

These are intended to be different sections of the same manual (single document).

The user manual should contain

• [ ] a brief explanation of what gptel is
• [ ] a brief overview of how LLM APIs work
• [ ] explanations of gptel’s features
• [ ] along with customization relevant to each feature
• [ ] explanation of (WIP) RAG and tool use

The developer manual should include

• [ ] building on gptel: examples of gptel-request usage
• [ ] a brief explanation of gptel’s design goals, focusing on
  • its scope: what kinds of features will/won’t be added
  • philosophy: integrate with Emacs, reduce cognitive burden, don’t introduce syntax
• [ ] a brief overview of gptel’s main control flow and state management
• [ ] data structures (plists, plists everywhere!)

Resources:

• Uniline for drawing line diagrams for info?

Since I don’t want to make automatic network requests to fetch model information from the API,

[BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWU0NDc=] [BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWU1Njc=]

here’s an idea.

• Define a cl-defgeneric that dispatches on the backend type, and updates the model list for gptel-backend or any backend.
• Define a new command, gptel-update-models, that calls these methods.

Now the user can run this whenever/wherever they want to update the list of models, like in a hook.

[BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWU1NTY=] [BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6cHVsbHJlcTUyOQ==]

The idea is to allow gptel-backend to be a string (that’s the name of the backend). In gptel-request, we can check the type of the variable:

(let ((backend
       (or (cl-etypecase gptel-backend
             (gptel-backend gptel-backend)
             (string (alist-get gptel-backend gptel--known-backends
                                nil nil #'equal)))
           (user-error "Backend \"%s\" is not known to be defined"
                       gptel-backend))))
  ...)

We just have to be sure that we do this everywhere gptel-backend is let-bound. Alternatively, we can use gptel--backend as the actual backend and gptel-backend can always be a string.

We may also need a gv-settable gptel-get-backend that can be used in situations like:

(push 'gpt-4o (gptel-backend-models (gptel-get-backend "ChatGPT")))

The idea is to allow entries in gptel-directives to be

• strings (as usual)
• lists (for templates)
• functions that return strings or lists of strings

This allows for

Different context sets are useful in different chat buffers (or regular buffers), so it would be good to have a way to specify this. With a single global set of contexts, you often have to engage in the add-context/dump-context dance.

• [BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWU0ODE=]
• [BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWU0NzU=]

The plan we’ve arrived on is to add buffer-local context via links to files in Org/Markdown mode buffers – this doesn’t work for buffers or regions, unfortunately, unless we add a link type.

• [ ] Add buffer-local context support
• [ ] Find a way to indicate that a link will be included in the context, perhaps the activate-func Org link property, and something bespoke for Markdown?
• [ ] Send text file links along with media files

To allow for features like https://github.com/karthink/gptel/issues/328, always create the prompt in a temp buffer. That way, we can add a list of filters to transform the text before sending, like Org’s export filters.

One commonly requested feature is removing the PROPERTIES drawers from the prompt in Org buffers before sending the request.

Disadvantages of using text-properties for this:

• Roles are invisible
• If the text-properties are sticky, self-inserted text in a response region is assigned the response role, but yanked or otherwise inserted text isn’t.
• If the text-properties are non-sticky, there’s no way to modify the response text at all.
• Persistence across Emacs sessions is fragile (requires gptel-mode to be turned on)

There are several proposals for improving this:

• #321 How should gptel distinguish between LLM responses and user text?
• [BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6cHVsbHJlcTM0Mw==]
• [BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWU1NjU=]
• [BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWU1NDY=]

In short, they suggest:

• Using syntax, i.e. user-defined prompt and response prefixes in chat buffers.
• Using zero-width delimiters
• Providing response-highlighting and role-setting commands
• Using overlays

Claude

Via tool-use

OpenAI

Via tool-use or a “structured-output” mode

We need a way to specify per-run pre- and post-response hooks, as well as transformation hooks.

[BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWUyODg=]

The idea is to allow a little before-string before each context overlay, then incldue that before the chunk when creating the context. I’m not sure yet how to make this feature discoverable.

• Calling gptel-add with a prefix arg, or using a keybinding at the context overlay, should allow us to attach a description or instructions
  • gptel-context-annotate bound to C-c C-a
  • gptel-context-clear bound to C-c C-k
• This annotation should then be included with the chunk when creating the context string.
• This annotation doesn’t need to be stored as a variable, we can store it in the overlay and display it as a before string, maybe.

[BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWUyOTk=]

Instead of writing new response parsers wholesale for Perplexity, it might be possible to piggyback on the OpenAI parsers using cl-call-next-method. Writing a response parser is significantly more work now that we support tool-use.

This is an oft-requested feature.

• How Aider does it, and Aider’s edit formats

Retrieval Augmented Generation, i.e. automatically fetching context for gptel requests.

It’s not clear yet how to do this. Just like we want to make gptel-tools work with externally defined tool collections, I think it will be good to make RAG a plug-in option. There are many sources for RAG.

Claude

JSON array of strings that will cause the model to stop generating text if encountered, specified at the top level of the API request(?)

[BROKEN LINK: No match for fuzzy expression: I don't want to populate the menu with even more options, at least for now. Dumping the full prompt if it's a new gptel session sounds like a good compromise to me, I]

[BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWUyNjE=]

[BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWUyMzU=]

The problem with this is coordinating async insertions into the undo change group – it’s tough to do.

[BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWUyODM=]

We can ask them if they want to contribute it to gptel.

https://github.com/certainty/madmacs/blob/main/modules/tools/madmacs-tools-ai.el

ert#erts files

To handle multi-turn requests, like those required for tool-use, we need something more robust than shoving function calls into callbacks. Unfortunately we don’t have access to a good promise API or to org-async right now¹.

I may need to write a process executor ☹️.

• [X] Merge!
• [X] Create a gptel-send / gptel--suffix-send specific transition table
• [X] Separate the user messaging in gptel-send from the callbacks Almost – only the “Typing…” instruction is part of the process filter.
• [X] Reuse the WAIT state after calling a tool
• [X] Make gptel-request return just the fsm, no callback.
• [X] Make gptel-request standalone.
• [X] Rename fsm callbacks to fsm handlers
• [X] Use a Finite State Machine (fsm) to drive gptel-request.

There are a few things we need.

This is a collection of features supported by different APIs that I’d like to support on a model-level basis in gptel. I need to figure out how to structure the symbol-plist of gptel models (which will now be symbols, not strings)

If all APIs support a feature (like stop_tokens) we don’t need specialized behavior for it, we can just add it to the transient menu directly.

As a precursor to adding vision support, function calling support, fill-in-middle support etc, we need a more granular way to specify model capabilities. The idea is to

• make gptel-model a symbol instead of a string,
• whose symbol-plist will list the capabilities of that model,
• which will be consulted when preparing the prompt

This will also require more prompt construction method for the various APIs.

Email from Anthropic Support: Claude can now use tools

• [-] Integration with gptel-complete
• [X] Branching conversations in Org

• Note taken on [2023-12-19 Tue 22:44]
  string-edit is Emacs 29.1+ only.

• Note taken on [2024-03-12 Tue 00:16]
  Unfortunately this doesn’t work: there are too many places in the call chain that refer to gptel-backend et al, so it’s not sufficient to add extra defmethods for gptel--parse-buffer and gptel--request-data. Some of the other places include gptel-send and gptel--suffix-send (for messaging the user), and gptel--url-get-response and gptel-curl--get-response.

  Another problem with this approach: The org properties will have to be scanned for again by each defmethod.

  The advice method is probably the best we can do.

In the feature-org branch: Currently the prompt parsing functions in gptel are advised to find the gptel parameters as Org properties if in Org mode. We can do away with this by using the qualifiers provided by cl-defmethod, see elisp#Generic Functions. This does the same thing as advice but is cleaner in the code. Might be harder to debug though.

• Note taken on [2024-04-23 Tue 18:49]
  Switched to the Ollama chat API so there is no longer a context vector.

[BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWUyNzk=]

https://github.com/karthink/gptel/discussions/156#discussioncomment-8896600

This can be an add-on package – gptel-copilot.

The main challenge is going to be finding a way to include enough context.

• C-g will abort the whole thing.
• C-= will launch a diff/ediff.
• Doing anything else will accept the completion.

[BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWU0NTI=]

Other packages can find it very useful to pick and choose components from gptel menus for their own purposes. However there doesn’t seem to be a way to do it.

https://github.com/magit/transient/discussions/299

Here’s how the backtick parsing works in Markdown:

1. A verbatim/src block can begin with any number of backticks.
2. If it begins with one backtick
   • this backtick is not followed by another backtick.
   • The contents of the verbatim block can include any number of backticks.
   • The end of the block is signalled by a backtick that is not next to a backtick.
3. If it begins with multiple backticks
   • Any sequence of fewer backticks is part of the verbatim block
   • A sequence of as many backticks ends the verbatim block

So here’s how we set in-src-block:

If we see a backtick, check if the char-after is a backtick

If yes,

• hold on until we see something other than a backtick,
• then set in-src-block. Keep count of the number of backticks.
• If we see this many backticks again, turn off in-src-block.

If no,

• set in-src-block.
• If we see another backtick, wait to make sure that it’s isolated – i.e. not preceeded/followed by a backtick.
• When this is the case, turn off in-src-block

gpt-4 has started using ### headings in its answers. Convert to Org headings or do something with them.

https://github.com/karthink/gptel/issues/184

And the ones I’m not adding and why.

It would be great to be able to (e)diff the last thing added to the buffer by the LLM. It would be even better to ediff across all previous states, but I’m not sure how to do this – some kind of integration with undo?

For one-step ediffs, here’s the info we need:

1. state of the buffer before gptel killed some text, if any.
2. state of the buffer after the response.

Other options: hook into after-change-functions – bad idea?

Since gptel’s edits to the buffer are local: (optionally) kills text and adds text at point, we can get away without diffing the whole buffer. (There may be user-driven edits in other parts of the buffer in the meantime that we don’t care about anyway.)

So:

• If killing text: keep the text in a buffer-local var
• In post-response-functions, add (at the end, at position 99), store the text in another buffer-local var. Or it could be a single cons containing the previous string (if any), along with other metadata.
• In the transient menu, we provide an option to (e)diff last LLM response in this buffer. When called… ?

To implement the feature described, the strategy seems plausibly efficient. Here are some potential steps:

• The functionality to (e)diff between two content states can be implemented as a separate function, perhaps gptel-ediff-last-response().
• You can make use of two buffer-local variables to keep track of the content state before and after the LLM’s edit. You can name these variables as gptel-last-pre-edit-content and gptel-last-post-edit-content respectively. Be aware to accurately update these variables whenever an edit operation happens.
• You would probably want a third buffer-local variable to keep track of the position where the LLM made its last edit (e.g., gptel-last-edit-position).
• Additionally, you might want to make these states persistent across emacs sessions for the users. You can use standard emacs lisp constructs like save-excursion to preserve point and other state in the process.
• In the gptel-ediff-last-response(), create two temporary buffers (or diff buffers). You can insert the content of gptel-last-pre-edit-content into the first buffer, and gptel-last-post-edit-content into the second buffer.
• After storing the content, use ediff-buffers or similar functionality to perform the diffing.

It will also be nice to have a mechanism to clear the tracked state, maybe a function gptel-clear-diff-state(). Also, consider adding a new key to the gptel’s transient menu which can be used to trigger this ediff operation. Users can also bind this function to a keystroke if they use it frequently.

By allowing users to trigger this diff function it would make it easier for them to visually see and understand what the LLM has changed in their content.

Remember to handle error states and edge cases gracefully - for example, what happens if a user tries to ediff when no edit operation has been made by the LLM yet (i.e., the buffer-local variables are null or undefined)? Effective user feedback and good documentation will be key to making this feature successful.

Finally, an additional feature to consider for the future might be to allow users to ediff between any two arbitrary past states, not just the most recent one. For this feature, a history of past states must be maintained. That could be a more significant design and performance endeavor, but it could also be a powerful tool for users.

• Note taken on [2024-02-02 Fri 19:48]
  

  buffer

  the buffer where the response is inserted

  position

  the position in :buffer= where the response is inserted

It’s not clear what these keyword args do, or even what they should do exactly.

Documentation: Rest API: https://ai.google.dev/tutorials/rest_quickstart Model parameters: https://ai.google.dev/docs/concepts#model_parameters

• Note taken on [2023-10-24 Tue 18:39]
  Settled for using cl-structs instead. Might still use defgeneric if the parsing differs considerably between backends.

This will make it easier to add support for

• Azure
• Dall-E
• Local LLMs, etc

https://github.com/karthink/gptel/issues/4

[BROKEN LINK: No match for fuzzy expression: ;; TODO: Handle this for ollama, see the new code in `gptel-curl–stream-cleanup']

Mainly the :header (lambda () bearer-token-stuff) needs to be automated, so that it can work with and without a specified key for that backend.

Email from Andrew Hyatt: LLM Experiments, Part 1: Corrections Email from Psionic K.: LLM Experiments, Part 1: Corrections

Email from Eli Zaretskii: Re: bug#68692: 29.1; File-local variable storage for strings with newlines

Email from Andrew Hyatt: Re: LLM Experiments, Part 1: Corrections

It includes the response prefix.

[BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWUyNjE=] Raw data from filter-function reads

Email from Jan Rychter: Re: [karthink/gptel] Separate system prompts and directives (Issue #249)

How is gptel more complex than the web UI?

See this issue.

https://github.com/magit/transient/issues/287

[BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWUyOTY=]

Email from Johan Ekenberg: 31.0.50; Fontification malfunction

[BROKEN LINK: No match for fuzzy expression: :parts 0 :text)))]

[BROKEN LINK: orgit-topic:Z2l0aHViLmNvbTpSX2tnRE9KRngwZGc6aXNzdWUyMzc=]

Thank you, didnt know that. Mind you add this as an example to the docs, please? The docs are a bit misleading as it says:

The :key can be a function that returns the key (more secure).

So i was thinking i had to develop my own function.

Tool use [0/0]​

https://old.reddit.com/r/emacs/comments/1i0vsz9/tech_demo_completing_functions_using_gptel/