Amr Gharbeia
e3e62140ff
Some checks failed
Deploy (Gitea) / deploy (push) Failing after 2s
Stream-chunk protocol: SSE streaming via provider-openai-stream, cascade-stream with fboundp guard in think(). TUI renders live. Stream interrupt: Esc during streaming marks [interrupted], finalizes msg. SSE cancel infrastructure: *stream-cancel* check in read loop. Markdown inline: **bold**, *italic*, `code` via parse-markdown-spans. Code blocks: parse-markdown-blocks + syntax-highlight (keywords/strings/fns). URL detection + Tab-to-activate: https:// URLs in dim, Tab opens. Watchdog: 30s stall detection via Dexador read-timeout. [streaming] indicator in status bar. Pre-existing TUI test fixes (7): first→aref, nil→zerop, add-msg arg. Core: 65/65 Neuro: 13/13 TUI View: 22/22 TUI Main: 65/65 Total: 165 tests, 0 failures.
411 lines
20 KiB
Org Mode
411 lines
20 KiB
Org Mode
#+TITLE: SKILL: Unified LLM Backend (org-skill-unified-llm-backend.org)
|
|
#+AUTHOR: Agent
|
|
#+FILETAGS: :skill:model:provider:llm:
|
|
#+PROPERTY: header-args:lisp :tangle ../lisp/neuro-provider.lisp
|
|
|
|
* Architectural Intent
|
|
|
|
~system-model-provider~ is the universal LLM client. It speaks the OpenAI-compatible ~/v1/chat/completions~ protocol, which covers every modern provider — OpenRouter, OpenAI, Anthropic, Groq, Gemini, DeepSeek, NVIDIA NIM, plus any local engine (Ollama, vLLM, LM Studio, llama.cpp) when running behind an OpenAI-compatible adapter.
|
|
|
|
One function, eight (and counting) providers. The same JSON payload, the same response format, the same error handling. Adding a new provider is a one-line config entry: a keyword, a base URL, an API key env var name, and a default model.
|
|
|
|
Providers register themselves at boot. No API key? That provider doesn't register. No local URL set? The local entry stays dormant. Only the providers you actually configure appear in ~*probabilistic-backends*~ at runtime. The old code assumed Ollama was always available; this code requires an env var like everything else.
|
|
|
|
=*provider-cascade*= defaults to cloud-only (all providers except ~:local~ and ~:ollama~). If you want a local fallback, set ~LOCAL_BASE_URL~ in your env and add ~:local~ to the ~PROVIDER_CASCADE~ list.
|
|
|
|
** Contract
|
|
|
|
1. (provider-config provider): returns the configuration plist for a
|
|
provider keyword, or nil if unregistered.
|
|
2. (provider-available-p provider): returns T if the provider's API key
|
|
or base URL is configured.
|
|
3. (provider-openai-request prompt system-prompt &key model provider):
|
|
executes an OpenAI-compatible /v1/chat/completions request. Returns
|
|
~(:status :success :content ...)~ or ~(:status :error :message ...)~.
|
|
4. (provider-openai-request prompt system-prompt &key model provider tools):
|
|
when ~:tools~ is provided (a list of plist tool definitions), the request
|
|
body includes ~"tools"~ and ~"tool_choice": "auto"~ fields. Parses
|
|
~tool_calls~ from the response: extracts ~function.name~ and
|
|
~function.arguments~ (decoded from JSON string to alist). Returns
|
|
~(:status :success :tool-calls ((:name <str> :arguments <alist>)))~
|
|
when the LLM returns a tool call, or the existing ~:content~ path otherwise.
|
|
4. (provider-cascade-initialize): reads ~PROVIDER_CASCADE~ from env and
|
|
sets ~*provider-cascade*~.
|
|
5. (provider-openai-stream prompt system-prompt callback &key model provider tools):
|
|
v0.7.1 — executes a streaming OpenAI-compatible /v1/chat/completions
|
|
request. Sends ~"stream": true~ in the request body. Reads Server-Sent
|
|
Events (SSE) from the response stream, parsing ~data: ...~ lines. For
|
|
each delta with content, calls CALLBACK with the delta string. After
|
|
all deltas, calls CALLBACK with ~""~ to signal end-of-stream. Returns
|
|
~(:status :success)~ on completion or ~(:status :error :message ...)~.
|
|
If ~*stream-cancel*~ is set to T (by another thread), exits the SSE
|
|
loop and calls CALLBACK with ~""~.
|
|
6. (parse-sse-line line): parses an SSE line. Returns the data content
|
|
for ~data: <content>~ lines, ~:done~ for ~data: [DONE]~, and ~nil~
|
|
for comment lines (starting with ~:~), empty lines, or non-data lines.
|
|
|
|
* Implementation
|
|
|
|
** Provider registry
|
|
#+begin_src lisp
|
|
(in-package :passepartout)
|
|
|
|
(defparameter *provider-configs*
|
|
'((:local . (:base-url nil :key-env nil :url-env "LOCAL_BASE_URL" :default-model "llama3"))
|
|
(:openrouter . (:base-url "https://openrouter.ai/api/v1" :key-env "OPENROUTER_API_KEY" :default-model "openrouter/auto"))
|
|
(:openai . (:base-url "https://api.openai.com/v1" :key-env "OPENAI_API_KEY" :default-model "gpt-4o-mini"))
|
|
(:anthropic . (:base-url "https://api.anthropic.com/v1" :key-env "ANTHROPIC_API_KEY" :default-model "claude-3-5-sonnet-20241022"))
|
|
(:groq . (:base-url "https://api.groq.com/openai/v1" :key-env "GROQ_API_KEY" :default-model "llama-3.1-70b-versatile"))
|
|
(:gemini . (:base-url "https://generativelanguage.googleapis.com/v1beta/openai" :key-env "GEMINI_API_KEY" :default-model "gemini-2.0-flash"))
|
|
(:deepseek . (:base-url "https://api.deepseek.com/v1" :key-env "DEEPSEEK_API_KEY" :default-model "deepseek-chat"))
|
|
(:nvidia . (:base-url "https://integrate.api.nvidia.com/v1" :key-env "NVIDIA_API_KEY" :default-model "meta/llama-3.1-405b-instruct"))))
|
|
#+end_src
|
|
|
|
** Provider config lookup
|
|
#+begin_src lisp
|
|
(defun provider-config (provider)
|
|
"Returns the configuration plist for a provider keyword."
|
|
(cdr (assoc provider *provider-configs*)))
|
|
#+end_src
|
|
|
|
** Availability check
|
|
#+begin_src lisp
|
|
(defun provider-available-p (provider)
|
|
"Checks if a provider is configured. Checks API key or URL env vars."
|
|
(let* ((config (provider-config provider))
|
|
(key-env (getf config :key-env))
|
|
(url-env (getf config :url-env))
|
|
(base-url (getf config :base-url)))
|
|
(cond (key-env (let ((key (uiop:getenv key-env))) (and key (> (length key) 0))))
|
|
(url-env (let ((url (uiop:getenv url-env))) (and url (> (length url) 0))))
|
|
(base-url t))))
|
|
#+end_src
|
|
|
|
** Unified request execution
|
|
#+begin_src lisp
|
|
(defun provider-openai-request (prompt system-prompt &key model (provider :openrouter) tools)
|
|
"Executes a request against any OpenAI-compatible API endpoint.
|
|
When :tools is provided, includes function-calling tool definitions in the request."
|
|
(let* ((config (provider-config provider))
|
|
(base-url (getf config :base-url))
|
|
(key-env (getf config :key-env))
|
|
(url-env (getf config :url-env))
|
|
(default-model (getf config :default-model))
|
|
(api-key (when key-env (uiop:getenv key-env)))
|
|
(model-id (or model default-model))
|
|
(url (if url-env
|
|
(let ((host (uiop:getenv url-env)))
|
|
(if host
|
|
(format nil "http://~a/v1/chat/completions" host)
|
|
(format nil "~a/chat/completions" base-url)))
|
|
(format nil "~a/chat/completions" base-url)))
|
|
(timeout (or (ignore-errors
|
|
(parse-integer (uiop:getenv "LLM_REQUEST_TIMEOUT")))
|
|
30))
|
|
(headers `(("Content-Type" . "application/json")
|
|
,@(when api-key `(("Authorization" . ,(format nil "Bearer ~a" api-key))))
|
|
,@(when (eq provider :openrouter)
|
|
`(("HTTP-Referer" . "https://github.com/amrgharbeia/passepartout")
|
|
("X-Title" . "Passepartout")))))
|
|
(body (let ((base `((model . ,model-id)
|
|
(messages . (( (role . "system") (content . ,system-prompt) )
|
|
( (role . "user") (content . ,prompt) ))))))
|
|
(if tools
|
|
(append base
|
|
`((tools . ,(loop for tool in tools
|
|
collect (list (cons :|type| "function")
|
|
(cons :|function| (loop for (k v) on tool by #'cddr
|
|
collect (cons (intern (string-upcase (string k)) "KEYWORD") v))))))
|
|
(:|tool_choice| . "auto")))
|
|
base)))
|
|
(body-json (cl-json:encode-json-to-string body)))
|
|
(handler-case
|
|
(let* ((response (dex:post url :headers headers :content body-json
|
|
:connect-timeout (min 5 timeout)
|
|
:read-timeout (max 10 (- timeout 5))))
|
|
(json (cl-json:decode-json-from-string response))
|
|
(choices (cdr (assoc :choices json)))
|
|
(first-choice (car choices))
|
|
(message (cdr (assoc :message first-choice)))
|
|
(tool-calls (cdr (assoc :|tool_calls| message)))
|
|
(content (cdr (assoc :content message))))
|
|
(cond
|
|
(tool-calls
|
|
(list :status :success
|
|
:tool-calls
|
|
(loop for tc in tool-calls
|
|
for fun = (cdr (assoc :|function| tc))
|
|
for args-str = (cdr (assoc :|arguments| fun))
|
|
for args = (when args-str (cl-json:decode-json-from-string args-str))
|
|
collect (list :name (cdr (assoc :|name| fun))
|
|
:arguments args))))
|
|
(content
|
|
(list :status :success :content content))
|
|
(t
|
|
(list :status :error :message (format nil "~a: No content" provider)))))
|
|
(error (c)
|
|
(list :status :error :message (format nil "~a Failure: ~a" provider c))))))
|
|
#+end_src
|
|
|
|
** Register all available providers
|
|
#+begin_src lisp
|
|
(defun provider-register-all ()
|
|
"Scans environment variables and registers all available LLM backends."
|
|
(dolist (entry *provider-configs*)
|
|
(let ((provider (car entry)))
|
|
(when (provider-available-p provider)
|
|
(log-message "LLM BACKEND: Registering provider ~a" provider)
|
|
(register-probabilistic-backend provider
|
|
(lambda (prompt system-prompt &key model tools)
|
|
(provider-openai-request prompt system-prompt :model model :provider provider :tools tools)))))))
|
|
#+end_src
|
|
|
|
** Initialize cascade
|
|
#+begin_src lisp
|
|
(defun provider-cascade-initialize ()
|
|
"Reads PROVIDER_CASCADE from env and sets *provider-cascade*."
|
|
(let ((cascade-str (uiop:getenv "PROVIDER_CASCADE")))
|
|
(if cascade-str
|
|
(setf *provider-cascade*
|
|
(mapcar (lambda (s) (intern (string-upcase (string-trim '(#\Space #\" #\') s)) :keyword))
|
|
(uiop:split-string cascade-str :separator '(#\,))))
|
|
(setf *provider-cascade* (mapcar #'car (remove-if (lambda (e)
|
|
(member (car e) '(:local)))
|
|
*provider-configs*))))))
|
|
#+end_src
|
|
|
|
** Provider connection test (for TUI config)
|
|
;; REPL-verified: 2026-05-04
|
|
#+begin_src lisp
|
|
(defun test-provider-connection (provider &optional api-key)
|
|
"Test a provider API key by hitting its models endpoint.
|
|
Returns (:ok) on success, (:fail reason) on failure.
|
|
If API-KEY is nil, reads from environment."
|
|
(let* ((config (provider-config provider))
|
|
(base-url (getf config :base-url))
|
|
(key-env (getf config :key-env))
|
|
(url-env (getf config :url-env))
|
|
(key (or api-key (when key-env (uiop:getenv key-env)))))
|
|
(handler-case
|
|
(let ((url (if url-env
|
|
(let ((host (or (uiop:getenv url-env) "")))
|
|
(format nil "http://~a/api/tags" host))
|
|
(format nil "~a/models" (or base-url "")))))
|
|
(if key-env
|
|
(progn (dex:get url :headers `(("Authorization" . ,(format nil "Bearer ~a" key)))
|
|
:connect-timeout 5 :read-timeout 10)
|
|
'(:ok))
|
|
(if url-env
|
|
(progn (dex:get url :connect-timeout 5 :read-timeout 10) '(:ok))
|
|
'(:fail "No URL source for this provider"))))
|
|
(error (c) `(:fail ,(format nil "~a" c))))))
|
|
#+end_src
|
|
|
|
** Boot registration
|
|
#+begin_src lisp
|
|
(provider-register-all)
|
|
(provider-cascade-initialize)
|
|
#+end_src
|
|
|
|
** Skill registration
|
|
#+begin_src lisp
|
|
(defskill :passepartout-neuro-provider
|
|
:priority 50
|
|
:trigger (lambda (ctx) (declare (ignore ctx)) nil))
|
|
#+end_src
|
|
|
|
* v0.7.1 — Streaming Backend
|
|
:PROPERTIES:
|
|
:ID: id-v071-streaming
|
|
:CREATED: [2026-05-08 Fri]
|
|
:END:
|
|
|
|
** SSE Parser
|
|
|
|
*** RED
|
|
#+begin_example
|
|
test-parse-sse-line-data: 0/2 pass — stub returns nil instead of content
|
|
test-parse-sse-line-done: 0/1 pass — stub returns nil instead of :done
|
|
test-parse-sse-line-nil: 3/3 pass — stub correctly returns nil
|
|
#+end_example
|
|
|
|
*** GREEN
|
|
#+begin_example
|
|
test-parse-sse-line-data: 2/2 pass (100%)
|
|
test-parse-sse-line-done: 1/1 pass (100%)
|
|
test-parse-sse-line-nil: 3/3 pass (100%)
|
|
test-provider-openai-stream-calls-callback: 3/3 pass (100%)
|
|
llm-gateway-suite: 13/13 pass (100%)
|
|
#+end_example
|
|
|
|
** Cascade Stream
|
|
#+begin_src lisp
|
|
(defun cascade-stream (prompt system-prompt callback)
|
|
"Streaming cascade: calls provider-openai-stream on the first available backend.
|
|
Calls CALLBACK with each delta string, then with '' to signal end-of-stream."
|
|
(dolist (backend *provider-cascade*)
|
|
(when (gethash backend *probabilistic-backends*)
|
|
(let ((result (provider-openai-stream prompt system-prompt callback
|
|
:provider backend)))
|
|
(when (eq (getf result :status) :success)
|
|
(return cascade-stream))))))
|
|
#+end_src
|
|
#+begin_src lisp
|
|
(in-package :passepartout)
|
|
|
|
(defun parse-sse-line (line)
|
|
"Parse an SSE line. Returns data string, :done for [DONE], nil otherwise."
|
|
(cond
|
|
((or (null line) (string= line "")) nil)
|
|
((char= (char line 0) #\:) nil)
|
|
((and (>= (length line) 6) (string-equal (subseq line 0 6) "data: "))
|
|
(let ((content (subseq line 6)))
|
|
(if (string= content "[DONE]")
|
|
:done
|
|
content)))
|
|
(t nil)))
|
|
#+end_src
|
|
|
|
** Streaming request
|
|
#+begin_src lisp
|
|
(defvar *stream-cancel* nil
|
|
"When T, the streaming SSE loop exits early.")
|
|
|
|
(defun provider-openai-stream (prompt system-prompt callback &key model (provider :openrouter) tools)
|
|
"Streaming OpenAI-compatible request. Calls CALLBACK with each delta, then ''."
|
|
(let* ((config (provider-config provider))
|
|
(base-url (getf config :base-url))
|
|
(key-env (getf config :key-env))
|
|
(url-env (getf config :url-env))
|
|
(default-model (getf config :default-model))
|
|
(api-key (when key-env (uiop:getenv key-env)))
|
|
(model-id (or model default-model))
|
|
(url (if url-env
|
|
(let ((host (uiop:getenv url-env)))
|
|
(if host
|
|
(format nil "http://~a/v1/chat/completions" host)
|
|
(format nil "~a/chat/completions" base-url)))
|
|
(format nil "~a/chat/completions" base-url)))
|
|
(timeout (or (ignore-errors (parse-integer (uiop:getenv "LLM_REQUEST_TIMEOUT"))) 30))
|
|
(req-headers (list (cons "Content-Type" "application/json")))
|
|
(base `((model . ,model-id)
|
|
(messages . (( (role . "system") (content . ,system-prompt) )
|
|
( (role . "user") (content . ,prompt) )))
|
|
(stream . t))))
|
|
(when api-key
|
|
(push (cons "Authorization" (format nil "Bearer ~a" api-key)) req-headers))
|
|
(when (eq provider :openrouter)
|
|
(setf req-headers
|
|
(append req-headers
|
|
`(("HTTP-Referer" . "https://github.com/amrgharbeia/passepartout")
|
|
("X-Title" . "Passepartout")))))
|
|
(let ((body (if tools
|
|
(append base
|
|
`((tools . ,(loop for tool in tools
|
|
collect (list (cons :|type| "function")
|
|
(cons :|function|
|
|
(loop for (k v) on tool by #'cddr
|
|
collect (cons (intern (string-upcase (string k)) "KEYWORD") v))))))
|
|
(:|tool_choice| . "auto")))
|
|
base)))
|
|
(handler-case
|
|
(let* ((body-json (cl-json:encode-json-to-string body))
|
|
(stall-seconds 30)
|
|
(s (dex:post url :headers req-headers :content body-json
|
|
:connect-timeout (min 5 timeout)
|
|
:read-timeout stall-seconds
|
|
:want-stream t)))
|
|
;; v0.7.1: track stall timer — reset on each successful chunk
|
|
(let ((last-chunk-time (get-universal-time)))
|
|
(loop for raw = (handler-case (read-line s nil nil)
|
|
(error (c)
|
|
(declare (ignore c))
|
|
nil))
|
|
while raw
|
|
do (when *stream-cancel* ; v0.7.1: cancel check
|
|
(setf *stream-cancel* nil)
|
|
(funcall callback " [cancelled]")
|
|
(return))
|
|
(let ((parsed (parse-sse-line raw)))
|
|
(cond
|
|
((null parsed))
|
|
((eq parsed :done) (return))
|
|
(t (handler-case
|
|
(let* ((json (cl-json:decode-json-from-string parsed))
|
|
(choices (cdr (assoc :choices json)))
|
|
(choice (car choices))
|
|
(delta (cdr (assoc :delta choice)))
|
|
(content (cdr (assoc :content delta))))
|
|
(when content
|
|
(funcall callback content)
|
|
(setf last-chunk-time (get-universal-time))))
|
|
(error ())))))
|
|
(when (> (- (get-universal-time) last-chunk-time) stall-seconds)
|
|
(funcall callback "[Response stalled — timed out at 30s]")
|
|
(return))))
|
|
(funcall callback "")
|
|
(close s)
|
|
(list :status :success))
|
|
(error (c)
|
|
(list :status :error :message (format nil "~a Stream Failure: ~a" provider c)))))))
|
|
#+end_src
|
|
|
|
* Test Suite
|
|
#+begin_src lisp
|
|
(eval-when (:compile-toplevel :load-toplevel :execute)
|
|
(ql:quickload :fiveam :silent t))
|
|
|
|
(defpackage :passepartout-llm-gateway-tests
|
|
(:use :cl :passepartout)
|
|
(:export #:llm-gateway-suite))
|
|
|
|
(in-package :passepartout-llm-gateway-tests)
|
|
|
|
(fiveam:def-suite llm-gateway-suite :description "Tests for the LLM provider backend")
|
|
(fiveam:in-suite llm-gateway-suite)
|
|
|
|
(fiveam:test test-provider-rejects-bad-keyword
|
|
"Contract 3: provider-config returns nil for unregistered provider."
|
|
(let ((config (provider-config :not-a-real-provider)))
|
|
(fiveam:is (null config))))
|
|
|
|
(fiveam:test test-provider-config-registered
|
|
"Contract 1: provider-config returns configuration plist for registered provider."
|
|
(let ((config (provider-config :openrouter)))
|
|
(fiveam:is (listp config))
|
|
(fiveam:is (getf config :base-url))))
|
|
|
|
(fiveam:test test-provider-accepts-tools-parameter
|
|
"Contract 4: provider-openai-request accepts :tools parameter without error."
|
|
(let ((result (provider-openai-request "test" "system" :tools (list))))
|
|
(fiveam:is (member (getf result :status) '(:success :error)))))
|
|
|
|
;; ── v0.7.1 Streaming ──
|
|
|
|
(fiveam:test test-parse-sse-line-data
|
|
"Contract 6: parse-sse-line extracts content from data: lines."
|
|
(fiveam:is (string= "hello world" (passepartout::parse-sse-line "data: hello world")))
|
|
(fiveam:is (string= "{\"a\":1}" (passepartout::parse-sse-line "data: {\"a\":1}"))))
|
|
|
|
(fiveam:test test-parse-sse-line-done
|
|
"Contract 6: parse-sse-line returns :done for [DONE]."
|
|
(fiveam:is (eq :done (passepartout::parse-sse-line "data: [DONE]"))))
|
|
|
|
(fiveam:test test-parse-sse-line-nil
|
|
"Contract 6: parse-sse-line returns nil for comment, empty, non-data lines."
|
|
(fiveam:is (null (passepartout::parse-sse-line "")))
|
|
(fiveam:is (null (passepartout::parse-sse-line ":ok")))
|
|
(fiveam:is (null (passepartout::parse-sse-line "event: ping"))))
|
|
|
|
(fiveam:test test-provider-openai-stream-calls-callback
|
|
"Contract 5: provider-openai-stream calls callback with deltas and final empty string."
|
|
(let ((collected '()))
|
|
(flet ((collector (text) (push text collected)))
|
|
(passepartout::provider-openai-stream "hi" "sys" #'collector :provider :openrouter))
|
|
(let* ((reversed (nreverse collected))
|
|
(last (car (last reversed))))
|
|
(fiveam:is (stringp last))
|
|
(fiveam:is (string= "" last))
|
|
(fiveam:is (>= (length reversed) 2)))))
|
|
#+end_src
|