passepartout/org/gateway-llm.org

SKILL: LLM Gateway (org-skill-llm-gateway.org)

• Overview
• Implementation
  • Request Execution (gateway-llm-request)
  • Skill Registration
• Test Suite

Overview

The LLM Gateway dispatches inference requests to the registered probabilistic backends. It receives a prompt and system prompt, looks up the provider's registered function from *probabilistic-backends*, calls it with the given model, and returns the result. This is the thin routing layer that sits between the reason pipeline and the provider-specific implementations in the unified-llm-backend skill.

Implementation

Request Execution (gateway-llm-request)

(defun gateway-llm-request (&key prompt system-prompt (provider :ollama) model)
  "Central dispatcher for LLM requests."
  (let ((backend (gethash provider *probabilistic-backends*)))
    (if backend
        (handler-case
            (funcall backend prompt system-prompt :model model)
          (error (c)
            (list :status :error :message (format nil "~a Failure: ~a" provider c))))
        (list :status :error :message (format nil "Provider ~a not registered" provider)))))

Skill Registration

(defskill :passepartout-gateway-llm
  :priority 100
  :trigger (lambda (ctx) (getf ctx :user-input))
  :deterministic (lambda (action ctx) (declare (ignore ctx)) action))

Test Suite

(eval-when (:compile-toplevel :load-toplevel :execute)
  (ql:quickload :fiveam :silent t))

(defpackage :passepartout-llm-gateway-tests
  (:use :cl :passepartout)
  (:export #:llm-gateway-suite))

(in-package :passepartout-llm-gateway-tests)

(fiveam:def-suite llm-gateway-suite :description "Tests for the LLM Gateway skill")
(fiveam:in-suite llm-gateway-suite)

(fiveam:test test-llm-gateway-timeout
  "Tier 2 Chaos: Verify that LLM Gateway handles connection failures gracefully."
  (let ((old-host (uiop:getenv "OLLAMA_HOST")))
    (unwind-protect
         (progn
           (setf (uiop:getenv "OLLAMA_HOST") "localhost:1")
           (let ((fn (or (find-symbol "GATEWAY-LLM-REQUEST" :passepartout.gateway-llm)
                          (find-symbol "GATEWAY-LLM-REQUEST" :passepartout))))
             (if fn
                 (let ((result (funcall fn :prompt "hello" :provider :ollama)))
                   (fiveam:is (eq (getf result :status) :error))
                   (fiveam:is (uiop:string-prefix-p "Ollama Failure" (getf result :message))))
                 (fiveam:fail "Could not find EXECUTE-LLM-REQUEST symbol"))))
      (if old-host
          (setf (uiop:getenv "OLLAMA_HOST") old-host)
          (sb-posix:unsetenv "OLLAMA_HOST")))))