amr/passepartout
1
0
Fork 0
Code Issues 15 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

chore: checkpoint broken state before fixing macro conflict

Browse Source
This commit is contained in:
Amr Gharbeia
2026-04-28 10:33:51 -04:00
parent a717ab1d3a
commit 3dddfe3e3d
7 changed files with 428 additions and 618 deletions
Download Patch File Download Diff File Expand all files Collapse all files

87
README.org
View File

@@ -4,57 +4,92 @@
* Meet OpenCortex
**OpenCortex is an AI that lives inside your own text files.**
Most AI assistants are just chatbots. You ask a question, they answer, they forget you exist. They trap your conversations in proprietary web apps and silo your data.
It is a sovereign neurosymbolic AI agent and 100-year Memex, built on a minimalist Lisp kernel. It organizes your life, executes tasks, and gardens your knowledge base—all while maintaining 100% data ownership in plain-text Org-mode files.
*OpenCortex is different. It is an AI that lives inside your own text files.*
* The v0.2.0 Experience: Standard & Sovereign
It runs locally on your machine. It reads your notes, organizes your life, executes tasks, and gardens your knowledge base—all while keeping your data in plain text files you own completely.
With the **v0.2.0 (Interactive Refinement)** milestone, OpenCortex has reached a production-grade foundation:
* Why OpenCortex Exists
- **Minimalist Kernel:** A purified Lisp harness targeting only core I/O and memory.
- **Sovereign Skills:** High-level capabilities (Diagnostics, Config, LLMs) are now modular skills.
- **POSIX/XDG Compliance:** Full alignment with Linux standards (~/.config, ~/.local/share).
- **Professional TUI:** A native Lisp interface with semantic highlighting and history scrolling.
The current generation of AI agents have a fundamental flaw: they prioritize quick demos over long-term reliability and user sovereignty.
The biggest problem is data ownership. Most agents bury your memories in opaque databases. If you want to see your own data, you have to ask the AI to fetch it. If the app shuts down, your data is gone.
OpenCortex solves this with total plain-text transparency. Your entire life is a folder of text files. OpenCortex manages them the same way you do—with any text editor. No database to migrate, no schema to update, no lock-in.
* What Makes OpenCortex Different
Most AI agents are Python applications that happened to call an LLM. OpenCortex is different. It is built in pure Common Lisp—top to bottom, no wrapper, no translation layer.
The kernel is Lisp. The skills are Lisp. The memory system is Lisp. The TUI is Lisp. One language from the hardware to the agent's thoughts.
Python agents need a second language for configuration (YAML), a third for memory (JSON or SQLite), and a fourth for deployment (Docker). OpenCortex needs SBCL. That's it.
This is not nostalgia for the 1980s. Lisp has two properties that matter for an autonomous agent.
First, code is data. In Lisp, the agent can read its own code the same way it reads a text file. It can parse its skills, understand their structure, and modify them at runtime. Python's AST is a foreign object—the agent can inspect it, but not treat it as something it owns.
Second, stability. The Common Lisp specification has been stable for decades. Your code does not break because a library changed its API. Dependencies do not rot. A Lisp program from 1995 still runs today.
Your data lives in Org-mode files. Not a database. Not JSON. Not a vector store. Just plain text that you can read in any editor, search with grep, and back up any way you want.
This matters because every other agent makes your data dependent on their app. Their database schema defines what you can store. Their migration scripts decide whether your data survives an upgrade. OpenCortex has no schema. Your memory is a folder of text files. It survives app updates, platform switches, and decades of use.
The agent fixes itself. When it encounters an error, it can modify its own code, apply surgical fixes, and learn from the outcome to improve and grow. Skills hot-reload at runtime, so you can extend the system without restarting. And if something goes wrong during a complex operation, it snaps back to a known-good state.
* Three Problems Every Agent Ignores
Every other AI assistant has three fundamental flaws that OpenCortex addresses.
The first is the data silo. Most agents bury your memories in opaque databases. If you want to see your own data, you have to ask the AI to fetch it. If the app shuts down, your data is gone. OpenCortex stores everything in plain text files that you own. No migration needed, no schema to update, no lock-in.
The second is the hallucination problem. Most agents pipe AI-generated text directly into your terminal. If the model hallucinates, it can delete your files or break your system. OpenCortex splits its brain into two parts. The creative brain (the LLM) suggests actions. The strict guard (deterministic logic) intercepts every proposal before it touches a file or runs a command. If the AI hallucinates, the guard blocks it.
The third is cloud dependency. Most assistants rely entirely on big tech APIs. When your internet drops, or the service goes down, your assistant dies. Worse, your private notes are constantly sent to third-party servers. OpenCortex runs on your own hardware using free, open-source models. Your private data never leaves your laptop. Cloud models are optional—used only when you explicitly allow them for complex tasks.
* Quick Start
To begin your journey with OpenCortex, you need a Common Lisp environment (SBCL).
You need SBCL (Steel Bank Common Lisp) installed.
#+begin_src bash
# 1. Clone the repository
git clone https://ssh://git@10.10.10.201:2222/amr/opencortex.git ~/memex/projects/opencortex
# Clone the repository
git clone https://github.com/amrgharbeia/opencortex.git ~/memex/projects/opencortex
# 2. Run the Modular Setup Wizard
# Run the Setup Wizard
cd ~/memex/projects/opencortex
./opencortex.sh setup
# 3. Verify your System Health
# Verify System Health
opencortex doctor
# 4. Enter the Brain
# Enter the Brain
opencortex tui
#+end_src
* The Onboarding Trifecta
1. **`opencortex setup`**: Guided Lisp-based configuration for multiple LLM providers (Ollama, Groq, OpenRouter, etc.).
2. **`opencortex link <platform> <token>`**: Securely connect external chat gateways like Telegram.
3. **`opencortex doctor`**: Real-time diagnostic report of your system health and environment.
`opencortex setup` guides you through configuring LLM providers. Tell it how to talk to Ollama, Groq, OpenRouter, or your own endpoint.
* Project Documentation
`opencortex link <platform> <token>` connects external chat gateways. Talk to your agent from Telegram while it works on your files.
OpenCortex is built using **Literate Programming**. The documentation is the code.
- **[[file:USER_MANUAL.org][User Manual]]**: Detailed guide on configuration, XDG paths, and commands.
- **[[file:docs/ROADMAP.org][The Evolutionary Roadmap]]**: Long-term vision and milestone progress.
- **[[file:docs/CONTRIBUTING.org][Contributing]]**: Learn how to implement new sovereign skills.
`opencortex doctor` shows you what's working, what's broken, and what needs attention.
* Architecture
- **Harness (Kernel):** TCP Bridge, Merkle-Memory, Literate Loader.
- **Skills (Userland):** Diagnostics, Configuration, LLM Gateway, Shell Actuation.
- **Interface:** Native Croatoan TUI and socket-based CLI.
OpenCortex has three layers.
The Harness is the kernel. It runs the [[file:harness/loop.org][metabolic loop]]—Perceive → Reason → Act—each signal moving through normalization, LLM reasoning, skill verification, and action execution. Depth limits prevent infinite loops. The [[file:harness/memory.org][memory system]] persists to plain-text Org-mode files with snapshot and rollback on errors.
The Skills are userland—thin harness, fat skills. Modular skills load at runtime. Diagnostics, Configuration, LLM Gateway, Shell Actuation, Emacs Editing, Self-Edit, Self-Fix, Credentials Vault, Tool Permissions, Protocol Validator, and more. Each is an independent Org-mode module. Add new skills without touching the kernel.
The Interface is what you use to talk to the agent. A native Lisp [[file:harness/tui-client.org][TUI]] with semantic highlighting and history. A [[file:harness/communication.org][TCP socket server]] for CLI interaction. External channels via the gateway—Telegram and beyond.
* Project Documentation
OpenCortex practices what it preaches—the documentation lives in the code.
The [[file:USER_MANUAL.org][User Manual]] covers setup, configuration, and commands. The [[file:docs/ROADMAP.org][Evolutionary Roadmap]] shows our plan for reaching state-of-the-art capabilities. The [[file:docs/CONTRIBUTING.org][Contributing]] guide teaches you how to add new skills.
* License

172
docs/ROADMAP.org
View File

@@ -5,46 +5,164 @@
The roadmap is designed working backwards from SOTA parity (V 1.0.0), guiding each version toward a fully autonomous, self-editing agent. Each version builds on the previous, with features designed to be implemented in pure Common Lisp + Org-mode.
** Non-Negotiable Identity
- Pure Common Lisp + Org-mode. No JSON. No YAML. No external databases.
- Single-address-space memory (Lisp hash tables in RAM — the agent IS the memory).
- "Thin harness, fat skills" — complexity lives at the edges, not the kernel.
- One agent composed of many skills. Concurrency via bordeaux-threads (shared memory).
- Plists everywhere — homoiconic communication between all components.
** Version Roadmap
*** v0.1.0: The Autonomous Foundation ✅
*** v0.1.0: The Autonomous Foundation — CURRENT RELEASE ✅
The secure, auditable Lisp kernel. All core infrastructure in place.
| Component | Status | Notes |
|-----------------------------------+--------+-----------------------------------------------------------------------|
| Perceive-Reason-Act pipeline | ✅ | 3-stage metabolic loop |
| Skills engine with jailed loading | ✅ | defskill, topological sort, hot-reload |
| Policy skill (6 invariants) | ✅ | Transparency, Autonomy, Bloat, Modularity, Mentorship, Sustainability |
| Bouncer skill | ✅ | Command whitelist guard functions |
| Memory (org-object + Merkle) | ✅ | Hash tables, snapshots, rollback |
| Lisp validator skill | ✅ | Syntax validation before eval |
| Scribe + Gardener skills | ✅ | Heartbeat-driven distillation + audit |
| LLM gateway (OpenRouter + Ollama) | ✅ | Provider cascade |
| Shell actuator | ✅ | Safe command execution |
| Emacs bridge via Swank | ✅ | Point/buffer updates |
| FiveAM test suite | ✅ | Memory, boot, pipeline, act, communication |
| Credentials vault | ✅ | Encrypted storage |
*** v0.2.0: Interactive Refinement ✅
The "Brain" meets the "Machine." Standardization and professionalization of the user interface and environment.
| Feature | Status | Notes |
| :--- | :---: | :--- |
| Minimalist Kernel | ✅ | Purified harness targeting I/O & Memory only. |
| Sovereign Skills | ✅ | Diagnostics and Configuration extracted to Userland. |
| POSIX/XDG Compliance| ✅ | Standardized paths (~/.config, ~/.local). |
| Professional TUI | ✅ | Styled, scrollable, and verified Lisp interface. |
| Onboarding Wizard | ✅ | Modular Lisp setup for multiple LLM providers. |
| Linkage Command | ✅ | Real-time verification of external gateways (Telegram). |
| Feature | Status | Notes |
| :--- | :---: | :--- |
| Minimalist Kernel | ✅ | Purified harness targeting I/O & Memory only. |
| Sovereign Skills | ✅ | Diagnostics and Configuration extracted to Userland. |
| POSIX/XDG Compliance | ✅ | Standardized paths (~/.config, ~/.local). |
| Professional TUI | ✅ | Styled, scrollable, and verified Lisp interface. |
| Onboarding Wizard | ✅ | Modular Lisp setup for multiple LLM providers. |
| Linkage Command | ✅ | Real-time verification of external gateways (Telegram). |
| Self-Editing | ✅ | Detects errors, applies fixes, learns from outcomes. |
| Memory Rollback | ✅ | Snap back to known-good state on critical errors. |
*** v0.3.0: Event Orchestration + HITL
Unified control plane and Human-in-the-Loop (HITL) state management.
| Feature | Description |
| :--- | :--- |
| org-skill-event-orchestrator | Unified hooks + cron + routing. Three tiers: =:REFLEX=, =:COGNITION=, =:REASONING=. |
| Human-in-the-Loop (HITL) | Continuation-based interaction. The agent can "suspend" to ask clarification and resume state. |
| Asynchronous Embedding Gateway | Provider-agnostic vector generation (Ollama, local llama.cpp) via background worker. |
| Telegram Gateway Skill | Full implementation of the message receiver for linked Telegram bots. |
| Command Palette | Support for internal `/` commands defined as Lisp-native capabilities. |
| Feature | Description |
|--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------|
| org-skill-event-orchestrator | Unified hooks + cron + routing. Three tiers: =:REFLEX= (no LLM), =:COGNITION= (light LLM), =:REASONING= (full LLM). |
| Human-in-the-Loop (HITL) | Continuation-based interaction. The agent can "suspend" its cognitive loop to ask for permission or clarification and resume precisely where it left off. |
| org-skill-context-manager | Stack-based project scoping. =push-context= / =pop-context=. Path resolution relative to context. |
| Memory scope segmentation | =:scope= property on org-objects: memex/session/project. Scope-aware retrieval. |
| Model-tier routing | Complexity-based model selection: heartbeat → tiny, user → medium, reasoning → large. |
| Slash commands | =M-x= style command palette in TUI. Commands defined in Org-mode. |
| Asynchronous Embedding Gateway | Provider-agnostic vector generation (Ollama, local llama.cpp) via background worker. |
| Telegram Gateway Skill | Full implementation of the message receiver for linked Telegram bots. |
*** v0.4.0: Long-Horizon Planning + Git Workflows
Structured tracking, failure handling, and course correction for multi-step engineering work.
| Feature | Description |
| :--- | :--- |
| org-skill-long-horizon | Decompose tasks into Org-mode headline trees. |
| org-skill-git-steward | Policy-enforced git workflows (Commit-Before-Modify). |
| TDD runner | Auto-verify changes on save and trigger self-fix loops. |
| Feature | Description |
|------------------------+---------------------------------------------------------------------------------------------------------------------------------------------|
| org-skill-long-horizon | Decompose tasks into Org-mode headline trees. Terminal states: =:done= / =:blocked= / =:stuck=. Parent summarises children. Branch pruning. |
| org-skill-git-steward | Status, diff, commit, push, branch. Policy enforces commit-before-modify. |
| TDD runner | FiveAM on file save. =:test-failure= events. Hook into self-fix for auto-repair. |
| Deep Emacs integration | Full org-agenda awareness. Navigate, clock time, refile, archive. |
*** v0.5.0+ : True Agency and Lisp Machine Emergence
World models, visual grounding, and total internalization of the AI stack.
- **v0.5.0**: Interactive PTY Actuators & Environment Stewardship.
- **v0.7.0**: Visual Grounding & MCP Bridge.
- **v1.0.0**: SOTA Parity (Competitive with commercial agents).
- **v2.0.0**: Lisp Machine Emergence (Agent IS the OS).
*** v0.5.0: Interactive Actuation & Environment Stewardship
Interactive terminal sessions and autonomous dependency management.
| Feature | Description |
|--------------------------+-------------------------------------------------------------------------------------------------------------------------------------|
| Interactive PTY Actuator | Stream long-running process output to the context window (e.g., `npm run dev`, REPLs) with async interrupt control. |
| The Environment Steward | Autonomously detect missing dependencies (e.g., "Command not found"), propose an installation command, and retry the failed action. |
*** v0.6.0: Concurrency + Creator + GTD
The agent bootstraps itself and manages parallel workstreams.
| Feature | Description |
|-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------|
| org-skill-sub-agent-manager | Lightweight Lisp-native sub-agents (via bordeaux-threads) that share memory but have isolated execution contexts for background work. |
| org-skill-creator | LLM drafts complete skill org-file from natural language. Mandatory: syntax validation → jail-load → test → register. |
| org-skill-architect | Scan =:STATUS: FROZEN= PRDs. Generate Phase B PROTOCOL. |
| org-skill-gtd | Full GTD cycle: capture, clarify, organize, reflect, engage. org-gtd v4.0 DAG (=:TRIGGER:=, =:BLOCKER:=). |
| Consensus loop | Run multiple providers for critical decisions. Compare results, detect disagreements. |
| Web research | Headless Chromium via Python bridge. Text extraction, screenshots, Gemini Web UI automation. |
*** v0.7.0: Visual Grounding & MCP Bridge
Multimodal visual interaction and ecosystem-wide tool compatibility.
| Feature | Description |
|-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Computer Use / Vision | Allow the agent to request host OS or browser screenshots, analyze the UI, and issue precise X/Y coordinate click/type commands via an X11/Wayland bridge. |
| MCP Gateway Bridge | Lisp-native client for the Model Context Protocol, allowing OpenCortex to connect to the entire ecosystem of external tools and data sources. |
*** v0.8.0: The Evaluation Harness
Automated benchmarking to mathematically prove the agent's reasoning capabilities.
| Feature | Description |
|-------------------+------------------------------------------------------------------------------------------------------------------------------------------------|
| SWE-Bench Harness | Automated pipeline that clones repositories, feeds GitHub issues, tracks the multi-step resolution trajectory, runs tests, and scores success. |
*** v1.0.0: SOTA Parity
Feature-complete agent competitive with commercial agents. All features reimplemented in pure Lisp.
| Area | Status | Notes |
|-------------------+-----------+-------------------------------------------|
| Self-improvement | ✅ v0.2.0 | Self-edit + lisp-repair |
| Planning | ✅ v0.4.0 | Task tree DAGs with terminal states |
| Tool ecosystem | 🟡 v0.4.0 | 10+ cognitive tools |
| Context window | ✅ v0.3.0 | Semantic search + scope segmentation |
| Safety | ✅ v0.1.0 | 6 Policy invariants + formal verification |
| Multi-step tasks | ✅ v0.4.0 | Task trees with failure handling |
| Code editing | ✅ v0.2.0 | Full org-mode file read/write |
| Memory | ✅ v0.2.0 | Vector recall in org-object |
| Emacs integration | ✅ v0.2.0 | Full org-mode control |
| Autonomy | ✅ v0.1.0 | 100% local capable (Ollama) |
*** v2.0.0: Lisp Machine Emergence
From Lisp-using agent to true Lisp machine. Agent IS the Emacs process.
| Feature | Description |
|---------|-------------|
| Lish: Lisp editor | Org-mode as IDE. Org-babel for interactive evaluation. Full REPL in TUI. No bridge needed. |
| Lish: Shell replacement | Lisp-based shell that speaks plists. Org-mode buffers as file system. |
*** v3.0.0: Neurosymbolic Maturity
Deterministic planner takes the wheel. LLM relegated to semantic translation.
| Feature | Description |
|---------|-------------|
| Deterministic planner | Pure Lisp task scheduler. No LLM needed for planning. |
| Self-correcting gates | Gates learn from false positives (user override patterns). |
*** v4.0.0: AI Stack Internalized
The agent understands its own weights. No external inference.
| Feature | Description |
|---------|-------------|
| Llama.cpp in Lisp | FFI binding. No Python subprocess. Pure Common Lisp inference. |
| Weights as sexps | Neural weights as Lisp data structures. Homoiconic model introspection. |
*** v5.0.0: True Agency
World models, temporal reasoning, goal persistence across restarts.
| Feature | Description |
|---------|-------------|
| World models | Predictive models of user behavior, project dynamics, system state. |
| Temporal reasoning | Scheduling, deadlines, elapsed duration awareness. |
| Goal persistence | Goals survive restarts. Long-term projects in org-objects. |

308
harness/manifest.org
View File

@@ -1,243 +1,103 @@
#+PROPERTY: header-args:lisp :tangle (expand-file-name "opencortex.asd" (concat (or (getenv "INSTALL_DIR") ".") "/harness"))
#+TITLE: Manifest (opencortex.asd)
#+AUTHOR: Amr
#+FILETAGS: :harness:system:
#+TITLE: System Manifest (manifest.org)
#+AUTHOR: Agent
#+FILETAGS: :harness:manifest:
#+STARTUP: content
* Manifest (opencortex.asd)
* Overview
The *System Manifest* defines the structural components of the OpenCortex. It serves as the primary system definition for ASDF and the test orchestrator.
** Architectural Intent: The Thin Harness Philosophy
* Implementation
The ~opencortex.asd~ file is the physical blueprint of the Lisp Machine. It uses **ASDF** (Another System Definition Facility) to orchestrate compilation and loading of all harness modules.
The core design principle is *Thin Harness, Fat Skills*:
- **Harness** = The minimal, unbreakable core (protocol, signal processing, memory)
- **Skills** = The intelligence layer (policy, validation, actuation, LLM integration)
This separation means:
- The harness rarely changes (immune system)
- Skills can be hot-loaded, modified, and swapped without touching the core
- Bugs in skills don't crash the system
** Why ASDF?**
ASDF is the de facto standard for Common Lisp project management. It:
1. Handles dependency resolution and loading order
2. Compiles files in the right order (preventing "undefined function" errors)
3. Supports system building for deployment
4. Integrates with Quicklisp for dependency management
* The Build Pipeline
#+begin_src mermaid
flowchart TD
Org[Literate Org Files] -- Org-Babel Tangle --> Lisp[Source .lisp Files]
Lisp --> ASDF[ASDF Manifest: opencortex.asd]
ASDF --> Loader[SBCL Compiler / Loader]
Loader --> Image[Live Harness Image]
Image -- Build --> Binary[Standalone Binary]
subgraph Skills["Skills Layer (Dynamic)"]
S1[Policy Skill]
S2[Bouncer Skill]
S3[LLM Gateway]
S4[...other skills]
end
Image --> Skills
#+end_src
* Design Decisions
** Strict Serial Loading
The harness uses ~:serial t~ in the ASDF definition. This means:
1. Files are loaded in order: package → skills → communication → memory → context → perceive → reason → act → loop
2. ~package.lisp~ is always loaded before any code that uses its symbols
3. ~skills.lisp~ (defining macros like ~defskill~, ~def-cognitive-tool~) loads before skills
This eliminates "macro not found" errors that plague non-linear loading systems.
** Why Not Module Dependencies?**
Traditional ASDF uses ~:depends-on~ to declare dependencies. We use ~:serial t~ because:
1. *Explicit is better than implicit* - the loading order is visible in one place
2. *Prevents circular dependencies* - skills are loaded after the harness, never before
3. *Simpler debugging* - when something fails, the loading order is always clear
** Isolation of Tests
The testing system (~:opencortex/tests~) is separate from the production system (~:opencortex~). This means:
- Production deployments don't load FiveAM (saves memory, reduces attack surface)
- Tests can be run independently: ~(ql:quickload :opencortex/tests)~
- Test data doesn't pollute the production image
* System Definitions
** Main Harness System
#+begin_src lisp
** Main System
#+begin_src lisp :tangle (expand-file-name "../opencortex.asd" (concat (or (getenv "INSTALL_DIR") ".") "/harness"))
(defsystem :opencortex
:name "opencortex"
:author "Amr"
:version "0.1.0"
:author "Amr Gharbeia"
:version "0.2.0"
:license "AGPLv3"
:description "The Probabilistic-Deterministic Lisp Machine Harness"
:depends-on (:usocket ; TCP socket networking
:bordeaux-threads ; Threading (heartbeat, async sensors)
:dexador ; HTTP client (LLM APIs)
:uiop ; Portable I/O, file operations
:cl-dotenv ; Environment variable loading
:cl-ppcre ; Regular expressions (parsing)
:hunchentoot ; HTTP server (optional web interface)
:ironclad ; Cryptography (Merkle hashing)
:str ; String utilities
:cl-json ; JSON parsing/serialization
:uuid) ; UUID generation for org-mode IDs
:serial t ; Load files in order listed below
:components ((:file "library/package") ; Package definitions, core vars
(:file "library/skills") ; Skill engine, cognitive tools
(:file "library/communication") ; Protocol, framing
(:file "library/communication-validator") ; Schema validation
(:file "library/memory") ; Org-object store, snapshots
(:file "library/context") ; Context assembly, query
(:file "library/perceive") ; Stage 1: Sensory normalization
(:file "library/reason") ; Stage 2: Neural + deterministic
(:file "library/act") ; Stage 3: Actuation
(:file "library/loop")) ; Main entry, heartbeat
:build-operation "program-op"
:build-pathname "opencortex-server"
:entry-point "opencortex:main")
:description "The Probabilistic-Deterministic Lisp Machine"
:depends-on (:usocket :bordeaux-threads :dexador :uiop :cl-dotenv :cl-ppcre :hunchentoot :ironclad :str :cl-json :uuid)
:serial t
:components ((:file "harness/package")
(:file "harness/skills")
(:file "harness/memory")
(:file "harness/communication")
(:file "harness/communication-validator")
(:file "harness/perceive")
(:file "harness/reason")
(:file "harness/act")
(:file "harness/context")
(:file "harness/loop")
(:file "skills/org-skill-diagnostics")
(:file "skills/org-skill-config-manager")
(:file "skills/org-skill-gateway-manager")
(:file "skills/org-skill-policy")
(:file "skills/org-skill-bouncer")
(:file "skills/org-skill-scribe")
(:file "skills/org-skill-gardener")
(:file "skills/org-skill-llm-gateway")
(:file "skills/org-skill-llama-backend")
(:file "skills/org-skill-shell-actuator")
(:file "skills/org-skill-emacs-edit")
(:file "skills/org-skill-self-edit")
(:file "skills/org-skill-self-fix")
(:file "skills/org-skill-lisp-utils")
(:file "skills/org-skill-literate-programming")
(:file "skills/org-skill-protocol-validator")
(:file "skills/org-skill-tool-permissions")
(:file "skills/org-skill-peripheral-vision")
(:file "skills/org-skill-cli-gateway")
(:file "skills/org-skill-engineering-standards")
(:file "skills/org-skill-credentials-vault")))
#+end_src
** Test System
#+begin_src lisp
#+begin_src lisp :tangle (expand-file-name "../opencortex.asd" (concat (or (getenv "INSTALL_DIR") ".") "/harness"))
(defsystem :opencortex/tests
:depends-on (:opencortex ; The harness we're testing
:fiveam) ; Testing framework
:components ((:file "library/gen/org-skill-emacs-edit")
(:file "library/gen/org-skill-lisp-utils")
(:file "tests/communication-tests")
(:file "tests/pipeline-tests")
(:file "tests/act-tests")
(:file "tests/boot-sequence-tests")
(:file "tests/memory-tests")
(:file "tests/immune-system-tests")
(:file "tests/emacs-edit-tests")
(:file "tests/lisp-utils-tests"))
:perform (test-op (o s)
(uiop:symbol-call :fiveam :run!
(uiop:find-symbol* :communication-protocol-suite :opencortex-tests))
(uiop:symbol-call :fiveam :run!
(uiop:find-symbol* :pipeline-suite :opencortex-pipeline-tests))
(uiop:symbol-call :fiveam :run!
(uiop:find-symbol* :boot-suite :opencortex-boot-tests))
(uiop:symbol-call :fiveam :run!
(uiop:find-symbol* :memory-suite :opencortex-memory-tests))
(uiop:symbol-call :fiveam :run!
(uiop:find-symbol* :immune-suite :opencortex-immune-system-tests))
(uiop:symbol-call :fiveam :run!
(uiop:find-symbol* :emacs-edit-suite :opencortex-emacs-edit-tests))
(uiop:symbol-call :fiveam :run!
(uiop:find-symbol* :lisp-utils-suite :opencortex-lisp-utils-tests))))
:depends-on (:opencortex :fiveam)
:components ((:file "tests/pipeline-act-tests")
(:file "tests/boot-sequence-tests")
(:file "tests/immune-system-tests")
(:file "tests/memory-tests")
(:file "tests/pipeline-perceive-tests")
(:file "tests/pipeline-reason-tests")
(:file "tests/peripheral-vision-tests")
(:file "tests/emacs-edit-tests")
(:file "tests/engineering-standards-tests")
(:file "tests/lisp-utils-tests")
(:file "tests/literate-programming-tests")
(:file "tests/self-edit-tests")
(:file "tests/tool-permissions-tests")
(:file "tests/diagnostics-tests")
(:file "tests/config-manager-tests")
(:file "tests/gateway-manager-tests")
(:file "tests/tui-tests")))
#+end_src
** TUI Client System
#+begin_src lisp
** TUI System
#+begin_src lisp :tangle (expand-file-name "../opencortex.asd" (concat (or (getenv "INSTALL_DIR") ".") "/harness"))
(defsystem :opencortex/tui
:depends-on (:opencortex ; The daemon we're connecting to
:croatoan ; Terminal UI library
:usocket ; Socket communication
:bordeaux-threads) ; Background listening thread
:components ((:file "library/tui-client")))
:depends-on (:opencortex :croatoan :usocket :bordeaux-threads)
:components ((:file "harness/tui-client")))
#+end_src
* The Harness Boundary Contract
** Test Orchestrator
#+begin_src lisp :tangle (expand-file-name "run-all-tests.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/harness"))
(asdf:load-system :opencortex/tests)
(format t "~%=== Running ALL Test Suites ===~%")
** Why a Boundary Contract?
(dolist (suite-spec '(("OPENCORTEX-BOOT-TESTS" "BOOT-SUITE")
("OPENCORTEX-COMMUNICATION-TESTS" "COMMUNICATION-PROTOCOL-SUITE")
("OPENCORTEX-PIPELINE-ACT-TESTS" "PIPELINE-ACT-SUITE")
("OPENCORTEX-MEMORY-TESTS" "MEMORY-SUITE")
("OPENCORTEX-ENGINEERING-STANDARDS-TESTS" "ENGINEERING-STANDARDS-SUITE")
("OPENCORTEX-DIAGNOSTICS-TESTS" "DIAGNOSTICS-SUITE")
("OPENCORTEX-GATEWAY-MANAGER-TESTS" "GATEWAY-SUITE")))
(let ((pkg (find-package (first suite-spec))))
(when pkg
(let ((suite-sym (find-symbol (second suite-spec) pkg)))
(when suite-sym
(fiveam:run! suite-sym))))))
The harness is the immune system of OpenCortex. If it grows fat (accumulating features, dependencies, complexity), it becomes harder to:
- Verify for security
- Debug when things go wrong
- Maintain across versions
The Boundary Contract defines what IS the harness vs. what belongs in skills.
** Primary Boundary Files
| File | Purpose | Modification |
|------|---------|--------------|
| ~harness/*.org~ | Literate source of truth | Only via Org edits + tangle |
| ~opencortex.asd~ | System manifest | Only via Org edits + tangle |
| ~library/*.lisp~ | Tangled from .org | NEVER edit directly |
** Generated Artifacts (NOT Primary)
The ~library/*.lisp~ files are tangles from the ~harness/*.org~ files. They are derivative artifacts. Direct modification violates the Literate Granularity standard.
** Protected Paths
The Policy skill guards these paths by default:
#+begin_src lisp
(defvar *modularity-protected-paths*
'("harness/"
"opencortex.asd"
"library/package.lisp"
"library/communication.lisp"
"library/memory.lisp"
"library/context.lisp"
"library/perceive.lisp"
"library/reason.lisp"
"library/act.lisp"
"library/loop.lisp"))
(format t "~%=== ALL TESTS COMPLETE ===~%")
#+end_src
Any agent action proposing to modify these files must include a ~:modularity-justification~ field explaining why the change cannot be implemented as a skill.
** Enforcement Chain
1. *Policy Skill* (priority 500) - Checks for missing justifications
2. *Bouncer Skill* (priority 100) - Intercepts unauthorized modifications
3. *Git Hooks* (optional) - Prevents direct .lisp commits
* Quick Reference
** Building the System
#+begin_src bash
# Development: Load source
(ql:quickload :opencortex)
# Build standalone binary
(asdf:make :opencortex)
# Run tests
(ql:quickload :opencortex/tests)
(asdf:test-system :opencortex/tests)
#+end_src
** Loading Order
1. ~library/package.lisp~ - Creates ~:opencortex~ package
2. ~library/skills.lisp~ - Defines ~defskill~, ~def-cognitive-tool~ macros
3. ~library/communication.lisp~ - Protocol, framing, validation
4. ~library/memory.lisp~ - Org-object, Merkle tree, snapshots
5. ~library/context.lisp~ - Context assembly functions
6. ~library/perceive.lisp~ - Stage 1: Perceive gate
7. ~library/reason.lisp~ - Stage 2: Reason (think + verify)
8. ~library/act.lisp~ - Stage 3: Act (dispatch + execute)
9. ~library/loop.lisp~ - Main entry point, heartbeat

4
harness/package.org
View File

@@ -293,8 +293,8 @@ Centralized logging function. It simultaneously writes to standard output and th
(fiveam:run! 'OPENCORTEX-LITERATE-PROGRAMMING-TESTS::LITERATE-PROGRAMMING-SUITE))
;; Communication tests
(when (find-package :OPENCORTEX-TESTS)
(fiveam:run! 'OPENCORTEX-TESTS::COMMUNICATION-PROTOCOL-SUITE))
(when (find-package :OPENCORTEX-COMMUNICATION-TESTS)
(fiveam:run! 'OPENCORTEX-COMMUNICATION-TESTS::COMMUNICATION-PROTOCOL-SUITE))
;; Pipeline tests
(when (find-package :OPENCORTEX-PIPELINE-TESTS)

16
harness/skills.org
View File

@@ -398,6 +398,22 @@ EXAMPLES:
** The Default Tool Belt
*** The Eval Tool (Internal Inspection)
#+begin_src lisp
** Cognitive Tool Registration
#+begin_src lisp :tangle (expand-file-name "skills.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/harness"))
(defvar *cognitive-tools* nil "Registry of available Agent capabilities.")
(defmacro def-cognitive-tool (name arg-list &body body)
"Registers a new cognitive tool (capability) into the global registry."
`(setf (getf *cognitive-tools* ,name)
(list :args ',arg-list
:executor (lambda (args)
(let ,(mapcar (lambda (arg)
(let ((arg-name (if (listp arg) (first arg) arg)))
`(,arg-name (getf args ,(intern (string arg-name) :keyword)))))
arg-list)
,@body)))))
#+end_src
(def-cognitive-tool :eval "Evaluates raw Common Lisp code in the harness image. Use this for complex calculations or internal state inspection."
((:code :type :string :description "The Lisp code to evaluate"))
:guard (lambda (args context)

261
skills/org-skill-engineering-standards.org
View File

@@ -2,144 +2,94 @@
:PROPERTIES:
:ID: 37f2b59f-4537-4cca-ac7f-5c24b9e2e773
:CREATED: [2026-03-30 Mon 21:16]
:EDITED: [2026-04-25 Sat]
:EDITED: [2026-04-27 Mon]
:END:
#+TITLE: SKILL: Engineering Standards
#+STARTUP: content
#+FILETAGS: :engineering:standards:workflow:lisp:git:tdd:chaos:
* Overview
This skill enforces the Engineering Standards for all development within OpenCortex. It observes agent context and gates actions that violate protocol.
The standards are ordered by lifecycle phase, not priority. An agent must execute them in sequence, not selectively.
* Phase 0: Before You Think
** Skill-First Query Rule
Before any analysis, debugging, or implementation: check if a skill already covers the problem domain.
Query the skill catalog via ~(list-skills)~ or ~(find-skill :keyword)~. If a relevant skill exists:
1. Read the skill's org file
2. Follow its mandates
3. Do not duplicate logic
Rationale: The skill layer is the primary intelligence. Raw LLM reasoning is a fallback, not a starting point. Violating this creates drift, where your solution diverges from the system's encoded wisdom.
* Phase A: Design (Test-First)
** 1. Define Success Criteria First
Before writing code, write the test that proves the feature works. The test defines the contract.
If the change is architectural, write the test as a PROTOCOL document (Plan Mode) and seek approval.
** 2. Break the Design with Chaos
After defining success criteria, run adversarial analysis:
- *Deterministic chaos:* Scripted regression tests that feed the system known-good inputs and verify known-good outputs. Run on every change.
- *Probabilistic chaos:* Randomized fuzzing, property-based testing, and noise injection to discover unknown failure modes. Run on every major release.
- *Stress chaos:* Sustained load, resource starvation, and concurrent access to find edge-case instabilities. Run during hardening sprints.
Rationale: If you cannot break your own design, you have not understood it.
* Phase B: Commit (Recovery Point)
** 3. Commit Before Modify
You MUST commit (and push, if network is available) the current workspace state before initiating new file modifications.
This is not a suggestion. If ~verify-git-clean-p~ returns NIL, the engineering standards gate MAY block high-impact actions.
** 2. Commit Before Modify
You MUST commit the current workspace state before initiating new file modifications.
* Phase C: Build (Implementation)
** 3. Literate Programming (Single Source of Truth)
All system logic and skills MUST be implemented as Literate Org files.
** 4. Literate Programming (Single Source of Truth)
** 4. Function-Block Granularity
Every Lisp function, macro, or variable resides in its own dedicated ~#+begin_src lisp~ block.
All system logic and skills MUST be implemented as Literate Org files. Weave documentation and code together. The "Why" (Architectural Intent) is never separated from the "How" (Implementation).
** 5. Tangle Mandate
You are forbidden from modifying generated ~.lisp~ files directly.
** 5. Function-Block Granularity
* Phase CDD: Chaos-Driven Development (Hardcoded Resilience)
** 6. Tier 1: Integrity Chaos (Per-Turn)
*Mandate:* Every turn involving a tangle MUST end with a "Structural Balance" check.
*Enforcement:* The Agent must verify that all tangled artifacts pass the Lisp reader without syntax errors.
Every Lisp function, macro, or variable resides in its own dedicated ~#+begin_src lisp~ block, immediately preceded by its explanatory text.
** 7. Tier 2: Integration Chaos (Per-Feature)
*Mandate:* Every new skill or major feature MUST include an "Adversarial Test Case."
*Enforcement:* The test suite must simulate a failure (e.g., mock network drop, malformed input) and prove the system degrades gracefully.
** 6. Tangle Mandate
You are forbidden from modifying generated ~.lisp~ files directly. All changes MUST be made in the Org file and then tangled.
** 7. Configuration Externalization
Source code MUST be free of hardcoded configuration values. Extract to environment variables and document in ~.env.example~.
** 8. Org as Thinking Medium
When debugging or analyzing issues, document your investigation in the relevant org file BEFORE implementing a fix.
Record: root cause hypothesis, evidence found, options considered, tradeoffs, decision rationale.
** 8. Tier 3: Systemic Chaos (Per-Milestone)
*Mandate:* Before sealing a milestone, the Agent MUST perform a "Scorched Earth" bootstrap.
*Enforcement:* Wipe XDG directories and verify a 100% autonomous re-initialization from the git source.
* Phase D: Validate (Proof)
** 9. Test Verification
No change is complete without running the test suite. A change that breaks existing tests is not a fix — it is damage.
Run chaotic tests alongside the main suite.
No change is complete without running the test suite. Run chaotic tests alongside the main suite.
* Phase E: Document (Audit Trail)
** 10. Decision Audit Trail
Every significant fix or architectural decision MUST be documented in an org file with:
- Root cause analysis
- Options considered and tradeoffs
- Why this solution was chosen
** 11. Stop-and-Wait (Turn-Yielding)
For major changes, propose your strategy in plain text, state "Waiting for user feedback," and yield the turn. Do not draft implementation in the same message.
** 12. GTD Synchronization
You are forbidden from considering a task complete without updating ~gtd.org~. Record all major shifts using hierarchical TODO headlines, NOT checkboxes.
Every significant fix or architectural decision MUST be documented in an org file.
* Enforcement Implementation
The engineering standards skill is a HARD BLOCK gate. Violations are rejected, not warned.
** Global Configuration
** Package Context
#+begin_src lisp :tangle (expand-file-name "org-skill-engineering-standards.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
(in-package :opencortex)
#+end_src
** Global Configuration
#+begin_src lisp :tangle (expand-file-name "org-skill-engineering-standards.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
(defvar *engineering-std-project-root* nil
"Path to the project root for enforcement checks.")
#+end_src
(defun engineering-std-set-project-root (path)
(setf *engineering-std-project-root* (uiop:ensure-directory-pathname path)))
#+begin_src lisp :tangle (expand-file-name "org-skill-engineering-standards.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
(defstruct engineering-violation
(phase nil)
(rule nil)
(message nil)
(severity nil))
(defvar *enforcement-rules*
'((:pre-task
(:git-clean "Working tree must be clean before modifications")
(:skill-queried "Skill catalog should be queried before analysis"))
(:during-task
(:org-only "Only .org files may be edited; .lisp is generated")
(:one-per-block "One definition per src block")
(:prose-required "Every block must have preceding prose"))
(:post-task
(:tests-pass "All tests must pass")
(:no-artifacts "No orphaned .bak, .log, .tmp files"))))
(defvar *engineering-std-initialized* nil)
#+end_src
** Git Clean Check (Blocking)
** CDD Utilities: Tier 1
#+begin_src lisp :tangle (expand-file-name "org-skill-engineering-standards.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
(defun check-structural-balance (file-path)
"Tier 1 Chaos: Verifies that a Lisp file is syntactically balanced."
(handler-case
(with-open-file (s file-path)
(loop for form = (read s nil :eof)
until (eq form :eof))
t)
(error (c)
(harness-log "CHAOS ERROR [Tier 1]: ~a in ~a" c file-path)
nil)))
#+end_src
** Git Protocol
#+begin_src lisp :tangle (expand-file-name "org-skill-engineering-standards.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
(defun verify-git-clean-p (&optional (dir *engineering-std-project-root*))
"Returns T if the git repository at DIR has no uncommitted changes."
@@ -148,76 +98,20 @@ The engineering standards skill is a HARD BLOCK gate. Violations are rejected, n
:output :string
:ignore-error-status t)))
(string= "" (string-trim '(#\Space #\Newline #\Tab) status)))))
(defun check-git-clean (&optional (dir *engineering-std-project-root*))
"Returns violation if git is dirty, nil if clean."
(unless (verify-git-clean-p dir)
(make-engineering-violation
:phase :pre-task
:rule :git-clean
:message "ENGINEERING STANDARDS VIOLATION: Working tree is dirty. Commit changes before modifying files."
:severity :blocker)))
#+end_src
** Blocking Gate (Hard Enforcement)
#+begin_src lisp :tangle (expand-file-name "org-skill-engineering-standards.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
(defun engineering-standards-gate (action context)
"The deterministic HARD BLOCK gate for Engineering Standards.
BLOCKING checks (return :LOG on violation):
- Git tree must be clean before file modifications
WARNING checks (log only):
- Skill catalog should be queried first
Returns modified action, or :LOG/:EVENT on violation."
(let* ((payload (getf action :payload))
(tool (getf payload :tool))
(file (getf payload :file))
(code (getf payload :code))
(modifies-files-p (or file code tool)))
;; BLOCKING: Git clean required for file modifications
(when modifies-files-p
(let ((git-check (check-git-clean *engineering-std-project-root*)))
(when git-check
(harness-log "~a" (engineering-violation-message git-check))
(return-from engineering-standards-gate
(list :type :log
:payload (list :text (engineering-violation-message git-check)))))))
action))
#+end_src
** Skill Registration
The skill runs at highest priority (1000) to block violations before any other skill.
#+begin_src lisp :tangle (expand-file-name "org-skill-engineering-standards.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
(defskill :skill-engineering-standards
:priority 1000
:trigger (lambda (ctx)
(declare (ignore ctx))
t)
:probabilistic nil
:deterministic #'engineering-standards-gate)
#+end_src
** Initialize Project Root
** Initializer
#+begin_src lisp :tangle (expand-file-name "org-skill-engineering-standards.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
(defun engineering-std-init ()
"Initialize the enforcement system with project root."
(unless *engineering-std-initialized*
(let ((env-root (or (uiop:getenv "OPENCORTEX_ROOT")
(uiop:getenv "MEMEX_DIR")
"/home/user/memex/projects/opencortex")))
(engineering-std-set-project-root env-root)
(setf *engineering-std-initialized* t)
(harness-log "ENGINEERING STANDARDS: Initialized with root ~a" *engineering-std-project-root*))))
"Initialize the enforcement system."
(let ((env-root (or (uiop:getenv "OC_DATA_DIR")
"/home/user/.local/share/opencortex")))
(setf *engineering-std-project-root* (uiop:ensure-directory-pathname env-root))
(harness-log "ENGINEERING STANDARDS: CDD Protocol Active.")))
#+end_src
;; Auto-initialize on load
#+begin_src lisp :tangle (expand-file-name "org-skill-engineering-standards.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
(engineering-std-init)
#+end_src
@@ -227,14 +121,22 @@ The skill runs at highest priority (1000) to block violations before any other s
(defpackage :opencortex-engineering-standards-tests
(:use :cl :fiveam :opencortex)
(:export #:engineering-standards-suite))
#+end_src
#+begin_src lisp :tangle (expand-file-name "engineering-standards-tests.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/tests"))
(in-package :opencortex-engineering-standards-tests)
#+end_src
#+begin_src lisp :tangle (expand-file-name "engineering-standards-tests.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/tests"))
(def-suite engineering-standards-suite
:description "Tests for Engineering Standards enforcement")
#+end_src
#+begin_src lisp :tangle (expand-file-name "engineering-standards-tests.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/tests"))
(in-suite engineering-standards-suite)
#+end_src
#+begin_src lisp :tangle (expand-file-name "engineering-standards-tests.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/tests"))
(test git-clean-check-clean
"verify-git-clean-p returns T when git tree is clean."
(let ((tmp-dir "/tmp/eng-std-test-clean/"))
@@ -242,53 +144,4 @@ The skill runs at highest priority (1000) to block violations before any other s
(uiop:run-program (list "git" "init" tmp-dir) :output nil)
(is (eq t (opencortex::verify-git-clean-p (uiop:ensure-directory-pathname tmp-dir))))
(uiop:delete-directory-tree (uiop:ensure-directory-pathname tmp-dir) :validate t)))
(test git-clean-check-dirty
"verify-git-clean-p returns NIL when git tree has uncommitted changes."
(let ((tmp-dir "/tmp/eng-std-test-dirty/"))
(uiop:ensure-all-directories-exist (list tmp-dir))
(uiop:run-program (list "git" "init" tmp-dir) :output nil)
(with-open-file (f (merge-pathnames "test.txt" tmp-dir) :direction :output)
(write-line "test" f))
(is (null (opencortex::verify-git-clean-p (uiop:ensure-directory-pathname tmp-dir))))
(uiop:delete-directory-tree (uiop:ensure-directory-pathname tmp-dir) :validate t)))
(test violation-struct
"engineering-violation struct is properly constructed."
(let ((v (opencortex::make-engineering-violation
:phase :pre-task
:rule :git-clean
:message "Test violation"
:severity :blocker)))
(is (eq :pre-task (opencortex::engineering-violation-phase v)))
(is (eq :git-clean (opencortex::engineering-violation-rule v)))
(is (string= "Test violation" (opencortex::engineering-violation-message v)))
(is (eq :blocker (opencortex::engineering-violation-severity v)))))
(test gate-blocks-dirty-tree
"engineering-standards-gate blocks when git is dirty."
(let ((action (list :type :request
:payload (list :tool :write-file
:file "/tmp/test"
:content "test"))))
;; Note: This test assumes git is clean in test environment
;; The gate returns :log if dirty
(let ((result (opencortex::engineering-standards-gate action nil)))
(is (listp result))
(when (eq (getf result :type) :log)
(is (search "dirty" (getf (getf result :payload) :text) :test #'char-equal))))))
(test gate-allows-clean-tree
"engineering-standards-gate passes when git is clean."
(let ((action (list :type :request
:payload (list :tool :read-file
:file "/tmp/test"))))
(let ((result (opencortex::engineering-standards-gate action nil)))
(is (listp result))
(is (eq :request (getf result :type))))))
#+end_src
* See Also
- [[file:org-skill-literate-programming.org][Literate Programming Skill]] - Structural validation and tangle rules
- [[file:org-skill-policy.org][Policy Skill]] - Constitutional constraints
- [[file:../README.org][opencortex README]]

198
skills/org-skill-llm-gateway.org
View File

@@ -1,153 +1,81 @@
#+PROPERTY: header-args:lisp :tangle (expand-file-name "org-skill-llm-gateway.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
:PROPERTIES:
:ID: llm-gateway-skill
:CREATED: [2026-04-09 Thu]
:EDITED: [2026-04-19 Sun]
:ID: llm-gateway-spec
:CREATED: [2026-04-10 Thu]
:END:
#+TITLE: SKILL: Unified LLM Gateway (Universal Literate Note)
#+TITLE: Skill: LLM Gateway
#+STARTUP: content
#+FILETAGS: :llm:gateway:infrastructure:autonomy:
#+DEPENDS_ON: org-skill-credentials-vault
#+FILETAGS: :skill:llm:gateway:
* Overview
The *Unified LLM Gateway* is the single sensory and reasoning interface for all neural backends. It consolidates the previously fragmented provider skills into a high-integrity dispatch layer, standardizing credential management, error handling, and payload formatting.
The *LLM Gateway* skill provides a unified interface for interacting with multiple Large Language Model providers.
* Phase B: Blueprint (PROTOCOL)
* Implementation
** 1. Architectural Intent
The gateway utilizes a functional dispatch pattern. A single entry point, `execute-llm-request`, resolves the provider-specific nuances (URLs, headers, JSON structures) while exposing a uniform interface to the harness.
** Package Context
#+begin_src lisp :tangle (expand-file-name "org-skill-llm-gateway.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
(in-package :opencortex)
#+end_src
* Phase D: Build (Implementation)
** Implementation
#+begin_src lisp
(defun get-nested (alist &rest keys)
"Recursively extracts nested values from an alist, handling both objects and arrays."
(let ((val alist))
(dolist (k keys)
;; Descend into arrays (cl-json style: ((key . val)) or ( ( (key . val) ) ))
(loop while (and (listp val) (listp (car val)) (not (keywordp (caar val))))
do (setf val (car val)))
(let ((pair (or (assoc k val)
(assoc (intern (string-upcase (string k)) :keyword) val)
(assoc (intern (string-downcase (string k)) :keyword) val))))
(if pair
(setf val (cdr pair))
(return-from get-nested nil))))
val))
(defun execute-llm-request (prompt system-prompt &key provider model)
"Unified entry point for all LLM providers. Respects the global cascade."
(let* ((active-provider (or provider (car opencortex::*provider-cascade*) :openrouter))
(api-key (vault-get-secret active-provider :type :api-key))
(full-prompt (format nil "~a~%~%Prompt: ~a" system-prompt prompt)))
(harness-log "PROBABILISTIC ENGINE: Requesting ~a (Model: ~s)"
active-provider (or model "default"))
;; If the specifically requested provider has no key, try falling back to the cascade
(when (or (null api-key) (string= api-key ""))
(harness-log "GATEWAY: Provider ~a has no key. Cascade fallback would trigger here." active-provider)
(return-from execute-llm-request (list :status :error :message "API Key missing.")))
** Skill Metadata
#+begin_src lisp :tangle (expand-file-name "org-skill-llm-gateway.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
(defparameter *skill-llm-gateway*
'(:name "llm-gateway"
:description "Unified provider-agnostic LLM interface."
:capabilities (:ask-llm :get-embedding)
:type :probabilistic)
"Skill metadata for the LLM Gateway.")
#+end_src
** Request Execution
#+begin_src lisp :tangle (expand-file-name "org-skill-llm-gateway.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
(defun execute-llm-request (&key prompt system-prompt provider model)
"Generic executor for all LLM providers."
(let* ((active-provider (or provider :ollama))
(api-key (uiop:getenv (format nil "~:@(~a_API_KEY~)" active-provider)))
(full-prompt (if system-prompt (format nil "~a~%~%~a" system-prompt prompt) prompt)))
(case active-provider
(:gemini-web
(let ((res (uiop:symbol-call :opencortex.skills.org-skill-web-research :ask-gemini-web full-prompt)))
(if res (list :status :success :content res) (list :status :error :message "Web Research Failure"))))
(:ollama
(let* ((host (or (uiop:getenv "OLLAMA_HOST") "localhost:11434"))
(url (format nil "http://~a/api/generate" host))
(body (cl-json:encode-json-to-string `((model . ,(or model "llama3")) (prompt . ,full-prompt) (stream . :false)))))
(handler-case
(progn
(harness-log "LLM DEBUG: Requesting Ollama...")
(let* ((response (dex:post url :headers '(("Content-Type" . "application/json")) :content body :connect-timeout 5 :read-timeout 60))
(json (cl-json:decode-json-from-string response)))
(list :status :success :content (cdr (assoc :response json)))))
(let* ((response (dex:post url :headers '(("Content-Type" . "application/json")) :content body))
(json (cl-json:decode-json-from-string response)))
(list :status :success :content (cdr (assoc :response json))))
(error (c) (list :status :error :message (format nil "Ollama Failure: ~a" c))))))
(t ;; Cloud Providers (Anthropic, Gemini API, Groq, OpenAI, OpenRouter)
(let* ((endpoint (case active-provider
(:anthropic "https://api.anthropic.com/v1/messages")
(:gemini-api (format nil "https://generativelanguage.googleapis.com/v1/models/~a:generateContent" (or model "gemini-1.5-flash-latest")))
(:groq "https://api.groq.com/openai/v1/chat/completions")
(:openai "https://api.openai.com/v1/chat/completions")
(:openrouter "https://openrouter.ai/api/v1/chat/completions")))
(headers (case active-provider
(:anthropic `(("Content-Type" . "application/json") ("x-api-key" . ,api-key) ("anthropic-version" . "2023-06-01")))
(:gemini-api `(("Content-Type" . "application/json") ("x-goog-api-key" . ,api-key)))
(:openrouter `(("Content-Type" . "application/json") ("Authorization" . ,(format nil "Bearer ~a" api-key))
("HTTP-Referer" . "https://github.com/amr/opencortex") ("X-Title" . "opencortex Autonomous Kernel")))
(t `(("Content-Type" . "application/json") ("Authorization" . ,(format nil "Bearer ~a" api-key))))))
(body (case active-provider
(:anthropic (cl-json:encode-json-to-string `((model . ,(or model "claude-3-5-sonnet-20240620")) (max_tokens . 4096) (system . ,system-prompt) (messages . (( (role . "user") (content . ,prompt) ))))))
(:gemini-api (cl-json:encode-json-to-string `((contents . (((parts . (((text . ,full-prompt))))))))))
(t (cl-json:encode-json-to-string `((model . ,(or model (case active-provider (:groq "llama-3.3-70b-versatile") (t "google/gemini-2.0-flash-001"))))
(messages . (( (role . "system") (content . ,system-prompt) ) ( (role . "user") (content . ,prompt) )))))))))
(handler-case
(progn
(harness-log "LLM DEBUG: Requesting ~a..." active-provider)
(let* ((response (dex:post endpoint :headers headers :content body :connect-timeout 10 :read-timeout 30))
(json (cl-json:decode-json-from-string response)))
(let ((content (case active-provider
(:anthropic (get-nested json :content :text))
(:gemini-api (get-nested json :candidates :parts :text))
(t (get-nested json :choices :message :content)))))
(if content
(list :status :success :content content)
(list :status :error :message (format nil "Failed to parse ~a response structure." active-provider))))))
(error (c) (list :status :error :message (format nil "LLM Gateway Failure (~a): ~a" active-provider c)))))))))
;; Initialize Cascade
(let* ((env-cascade (uiop:getenv "PROVIDER_CASCADE"))
(default-list '(:openrouter :openai :anthropic :groq :gemini-api :ollama))
(final-list (if (and env-cascade (not (string= env-cascade "")))
(mapcar (lambda (s) (intern (string-upcase (string-trim '(#\Space) s)) :keyword))
(uiop:split-string env-cascade :separator '(#\,)))
default-list)))
(setf opencortex::*provider-cascade* final-list)
(opencortex:harness-log "PROBABILISTIC: Neural Cascade Initialized -> ~a" final-list))
;; Register Providers
(dolist (p '(:anthropic :gemini-api :gemini-web :groq :ollama :openrouter :openai))
(opencortex:register-probabilistic-backend p (lambda (prompt system-prompt &key model)
(execute-llm-request prompt system-prompt :provider p :model model))))
(def-cognitive-tool :get-ollama-embedding
"Generates vector embeddings via Ollama API for semantic search."
((text :type :string :description "Text to embed."))
:body (lambda (args)
(let* ((text (getf args :text))
(host (or (uiop:getenv "OLLAMA_HOST") "localhost:11434"))
(url (format nil "http://~a/api/embeddings" host))
(model (or (uiop:getenv "OLLAMA_EMBEDDING_MODEL") "nomic-embed-text"))
(body (cl-json:encode-json-to-string `((model . ,model) (prompt . ,text)))))
(handler-case
(let* ((response (dex:post url :headers '(("Content-Type" . "application/json")) :content body :connect-timeout 5 :read-timeout 30))
(json (cl-json:decode-json-from-string response)))
(let ((embedding (cdr (assoc :embedding json))))
(if embedding
(list :status :success :vector embedding)
(list :status :error :message "No embedding in response"))))
(error (c) (list :status :error :message (format nil "Ollama Embedding Failure: ~a" c)))))))
(def-cognitive-tool :ask-llm
"Queries an LLM provider via the unified gateway."
((:prompt :type :string :description "The user prompt.")
(:system-prompt :type :string :description "The system instructions.")
(:provider :type :keyword :description "Optional specific provider.")
(:model :type :string :description "Optional specific model ID."))
:body (lambda (args)
(execute-llm-request (getf args :prompt)
(or (getf args :system-prompt) "You are a helpful assistant.")
:provider (getf args :provider)
:model (getf args :model))))
(defskill :skill-llm-gateway
:priority 150
:trigger (lambda (context) (declare (ignore context)) nil)
:probabilistic (lambda (context) (declare (ignore context)) nil)
:deterministic (lambda (action context) (declare (ignore context)) action))
(t (list :status :error :message "Provider not implemented")))))
#+end_src
** Cognitive Tools
#+begin_src lisp :tangle (expand-file-name "org-skill-llm-gateway.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
(def-cognitive-tool :get-ollama-embedding (text)
(let* ((host (or (uiop:getenv "OLLAMA_HOST") "localhost:11434"))
(url (format nil "http://~a/api/embeddings" host))
(body (cl-json:encode-json-to-string `((model . "nomic-embed-text") (prompt . ,text)))))
(handler-case
(let* ((response (dex:post url :headers '(("Content-Type" . "application/json")) :content body))
(json (cl-json:decode-json-from-string response)))
(cdr (assoc :embedding json)))
(error (c) (harness-log "OLLAMA EMBED ERROR: ~a" c) nil))))
#+end_src
#+begin_src lisp :tangle (expand-file-name "org-skill-llm-gateway.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
(def-cognitive-tool :ask-llm (prompt system-prompt provider model)
(execute-llm-request :prompt prompt
:system-prompt system-prompt
:provider provider
:model model))
#+end_src
** Skill Registration
#+begin_src lisp :tangle (expand-file-name "org-skill-llm-gateway.lisp" (concat (or (getenv "INSTALL_DIR") ".") "/skills"))
(defskill :skill-llm-gateway
:priority 50
:trigger (lambda (ctx) (declare (ignore ctx)) t)
:probabilistic (lambda (ctx)
(let ((input (getf ctx :user-input)))
(when input
(execute-llm-request :prompt input))))
:deterministic (lambda (action ctx) (declare (ignore ctx)) action))
#+end_src