feat(kernel): upgrade neurosymbolic kernel to Order 2 (recursive self-maintenance, SOTA upgrades)
This commit is contained in:
29
docs/PRD.org
29
docs/PRD.org
@@ -1,29 +0,0 @@
|
||||
#+TITLE: PRD: org-agent Cognitive Core & Configuration
|
||||
#+AUTHOR: PSF Requirements Definer
|
||||
#+CREATED: [2026-03-23 Mon]
|
||||
#+STATUS: FROZEN
|
||||
|
||||
* 0. Core Mandates
|
||||
The `org-agent` project MUST adhere to these foundational mandates:
|
||||
- **Mandate 1: Strict Homoiconic Memory.** All documentation, planning, and system logic MUST be authored in Org-mode (.org) and Common Lisp. Markdown (.md) and JSON are strictly prohibited for internal use.
|
||||
- **Mandate 2: Minimalist Microkernel.** The core daemon MUST remain minimalist, handling only the cognitive loop, the persistent Object-Store, and the communication protocol. All domain-specific features and LLM provider logic MUST be implemented as hot-reloadable **Skills** living in the user's Memex.
|
||||
|
||||
* 1. Purpose
|
||||
The `org-agent` must transition from a monolithic prototype to a generalized neurosymbolic kernel.
|
||||
|
||||
* 2. Functional Requirements
|
||||
** 2.1. Cognitive Loop (PTA Refactor)
|
||||
- The daemon MUST implement a 4-stage pipeline: Perceive -> Think -> Decide -> Act.
|
||||
- System 1 (Neural) MUST be restricted to the 'Think' stage.
|
||||
- System 2 (Symbolic) MUST have absolute authority in the 'Decide' stage to block or modify neural proposals.
|
||||
- The I/O protocol (OACP) MUST be encapsulated in 'Perceive' and 'Act'.
|
||||
|
||||
** 2.2. Externalized Configuration (.env)
|
||||
- All secrets (API keys) and environment-specific settings (ports, paths) MUST live in a `.env` file.
|
||||
- The system MUST automatically load `.env` upon system initialization.
|
||||
- Secrets MUST NOT be hardcoded or checked into source control.
|
||||
|
||||
* 3. Success Criteria
|
||||
- TODO Daemon starts and loads LLM_API_KEY from .env.
|
||||
- TODO `cognitive-loop` successfully routes a `:buffer-update` event through all 4 stages.
|
||||
- TODO System 2 (`decide`) successfully blocks an `:eval` request containing "shell-command".
|
||||
@@ -1,137 +0,0 @@
|
||||
#+TITLE: org-agent Communication Protocol (OACP)
|
||||
#+AUTHOR: Agent
|
||||
#+DATE: 2026-03-22
|
||||
#+ID: org-agent-protocol
|
||||
#+STARTUP: content
|
||||
|
||||
* Core Mandates
|
||||
The OACP and the `org-agent` project MUST adhere to these foundational mandates:
|
||||
- **Mandate 1: Strict Homoiconic Memory.** All documentation, planning, and system logic MUST be authored in Org-mode (.org) and Common Lisp. Markdown (.md) and JSON are strictly prohibited for internal use.
|
||||
- **Mandate 2: Minimalist Microkernel.** The core daemon MUST remain minimalist, handling only the cognitive loop, the persistent Object-Store, and the communication protocol. All domain-specific features and LLM provider logic MUST be implemented as hot-reloadable **Skills** living in the user's Memex.
|
||||
|
||||
* Overview
|
||||
|
||||
OACP (org-agent Communication Protocol) defines the "nervous system" of the Neurosymbolic Lisp Machine. It facilitates bidirectional, asynchronous communication between the **Common Lisp Core** (The Soul/Daemon) and the **Emacs Interface** (The Actuator/Terminal).
|
||||
|
||||
The protocol is designed to be:
|
||||
- **Homoiconic:** Messages are native Lisp S-expressions (plists).
|
||||
- **Asynchronous:** Neither side should block waiting for the other.
|
||||
- **Extensible:** New sensors and actuators can be added by defining new plist keys.
|
||||
|
||||
* Framing & Transport
|
||||
|
||||
- **Transport:** TCP Socket (default port: 9105).
|
||||
- **Framing:** Each message is prefixed by a 6-character hexadecimal length string (e.g., `00002a` for a 42-byte message), followed by the S-expression string encoded in UTF-8.
|
||||
|
||||
* Message Structure
|
||||
|
||||
A standard message is a flat property list (plist):
|
||||
|
||||
#+begin_src lisp
|
||||
(:type TYPE :id ID :payload PAYLOAD)
|
||||
#+end_src
|
||||
|
||||
- **:type:** One of `:EVENT`, `:REQUEST`, `:RESPONSE`, or `:LOG`.
|
||||
- **:id:** A unique integer or string for correlation (mandatory for `:REQUEST` and `:RESPONSE`).
|
||||
- **:payload:** A nested plist containing the specific data or command.
|
||||
|
||||
* Perception (Emacs -> Core)
|
||||
|
||||
Emacs acts as the sensor array, pushing events to the Core daemon.
|
||||
|
||||
** :BUFFER-MODIFIED
|
||||
Sent when an Org buffer is changed or saved.
|
||||
#+begin_src lisp
|
||||
(:type :EVENT :payload (:sensor :buffer-update :file "/home/amr/org/todo.org" :state :saved))
|
||||
#+end_src
|
||||
|
||||
** :USER-INPUT
|
||||
Sent when the user explicitly triggers an agent action (e.g., `M-x org-agent-ask`).
|
||||
#+begin_src lisp
|
||||
(:type :EVENT :payload (:sensor :user-prompt :text "Refactor this subtree" :context :current-subtree))
|
||||
#+end_src
|
||||
|
||||
** :CURSOR-MOVED
|
||||
Sent when the agent needs to track the user's focus.
|
||||
#+begin_src lisp
|
||||
(:type :EVENT :payload (:sensor :focus :buffer "init.el" :line 42 :column 0))
|
||||
#+end_src
|
||||
|
||||
* Action (Core -> Emacs)
|
||||
|
||||
The Core daemon sends requests to Emacs to perform physical actions in the environment.
|
||||
|
||||
** :EXECUTE-ELISP
|
||||
The primary "actuator."
|
||||
#+begin_src lisp
|
||||
(:type :REQUEST :id 101 :payload (:action :eval :code "(org-todo \"NEXT\")"))
|
||||
#+end_src
|
||||
|
||||
** :UI-NOTIFY
|
||||
Display info to the user without interrupting focus.
|
||||
#+begin_src lisp
|
||||
(:type :REQUEST :id 102 :payload (:action :message :text "I have identified a contradiction in your GTD.org" :level :warning))
|
||||
#+end_src
|
||||
|
||||
** :PROVIDE-COMPLETION
|
||||
Provide LLM-driven completions for the current point.
|
||||
#+begin_src lisp
|
||||
(:type :REQUEST :id 103 :payload (:action :complete :prefix "defun" :suggestions ("defun-agent" "defun-percieve")))
|
||||
#+end_src
|
||||
|
||||
** :ORG-DELIVERY
|
||||
Enqueue external messages.
|
||||
#+begin_src lisp
|
||||
(:type :REQUEST :target :delivery :payload (:channel :signal :to "+1..." :text "Hello"))
|
||||
#+end_src
|
||||
|
||||
** :PROJECT-SCAFFOLD
|
||||
Create new project folders and headings.
|
||||
#+begin_src lisp
|
||||
(:type :REQUEST :target :foundry :payload (:action :scaffold :name "Project-X" :type "Lisp"))
|
||||
#+end_src
|
||||
|
||||
** :SYSTEM-SELF-EDIT
|
||||
Kernel-level self-modification.
|
||||
#+begin_src lisp
|
||||
(:type :REQUEST :target :system :payload (:action :create-skill :filename "skill-x.org" :content "..."))
|
||||
#+end_src
|
||||
|
||||
* Metadata & Handshake
|
||||
|
||||
** :HELLO
|
||||
Sent by both sides upon connection.
|
||||
#+begin_src lisp
|
||||
(:type :EVENT :payload (:action :handshake :version "0.1.0" :capabilities (:auth :swank :org-ast)))
|
||||
#+end_src
|
||||
|
||||
* Cognitive Loop Internal API (Foundry Standard)
|
||||
|
||||
To ensure neurosymbolic sovereignty, the Core Daemon must strictly separate stages.
|
||||
|
||||
** 1. Perceive
|
||||
- **Signature:** `(perceive raw-stimulus) -> context-plist`
|
||||
- **Responsibility:** Protocol decoding, Object-Store synchronization, context augmentation.
|
||||
|
||||
** 2. Think (System 1)
|
||||
- **Signature:** `(think context-plist) -> proposed-action-plist`
|
||||
- **Responsibility:** Neural pattern matching, associative retrieval, unverified proposal generation.
|
||||
|
||||
** 3. Decide (System 2)
|
||||
- **Signature:** `(decide proposed-action context) -> approved-action-plist`
|
||||
- **Responsibility:** Symbolic verification, safety gating, deterministic rule application (e.g., GTD logic).
|
||||
|
||||
** 4. Act (Dispatch)
|
||||
- **Signature:** `(act stream approved-action)` / `(dispatch-action approved-action)`
|
||||
- **Responsibility:** Actuator lookup and transmission. Targets: `:emacs`, `:delivery`, `:system`, `:shell`.
|
||||
|
||||
** 5. Context API (System 1 Peripheral Vision)
|
||||
- **(context-list-all-skills):** Introspect the brain hierarchy.
|
||||
- **(context-get-skill-source name):** Read the logic graph.
|
||||
- **(context-resolve-path path):** Environment-relative path expansion.
|
||||
- **(context-get-system-logs limit):** Perception of kernel failures.
|
||||
|
||||
* Success Metrics
|
||||
- Latency between `:EVENT` and `:RESPONSE` < 100ms for UI actions.
|
||||
- 100% fidelity in Org AST preservation across the bridge.
|
||||
- Zero UI blocking in Emacs during heavy CL reasoning loops.
|
||||
@@ -10,6 +10,7 @@
|
||||
:components ((:file "package")
|
||||
(:file "protocol")
|
||||
(:file "object-store")
|
||||
(:file "embedding")
|
||||
(:file "skills")
|
||||
(:file "neuro")
|
||||
(:file "symbolic")
|
||||
|
||||
136
scripts/onboard.sh
Executable file
136
scripts/onboard.sh
Executable file
@@ -0,0 +1,136 @@
|
||||
#!/bin/bash
|
||||
|
||||
# org-agent Onboarding Script: The First Breath
|
||||
# This script prepares your PSF environment for the Lisp Machine.
|
||||
|
||||
set -e
|
||||
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
echo -e "${BLUE}==================================================${NC}"
|
||||
echo -e "${BLUE} org-agent: Personal Software Foundry Onboarding ${NC}"
|
||||
echo -e "${BLUE}==================================================${NC}"
|
||||
|
||||
# 1. Environment Verification
|
||||
echo -e "\n${BLUE}[1/5] Verifying Environment...${NC}"
|
||||
|
||||
if command -v sbcl >/dev/null 2>&1; then
|
||||
echo -e "${GREEN}✓ SBCL (Steel Bank Common Lisp) found.${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ SBCL not found. Please install it first.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ -d "$HOME/quicklisp" ] || [ -d "$HOME/.quicklisp" ]; then
|
||||
echo -e "${GREEN}✓ Quicklisp found.${NC}"
|
||||
else
|
||||
echo -e "${RED}✗ Quicklisp not found. Please install Quicklisp to manage Lisp dependencies.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 2. Configuration Setup
|
||||
echo -e "\n${BLUE}[2/5] Setting up .env configuration...${NC}"
|
||||
|
||||
if [ ! -f .env ]; then
|
||||
cp .env.example .env
|
||||
echo -e "${GREEN}✓ Created .env from .env.example.${NC}"
|
||||
else
|
||||
echo -e "${BLUE}! .env already exists. Skipping creation.${NC}"
|
||||
fi
|
||||
|
||||
# Set MEMEX_DIR automatically to current parent if not set
|
||||
PROJECT_ROOT=$(pwd)
|
||||
PARENT_DIR=$(dirname "$PROJECT_ROOT")
|
||||
|
||||
# Use a temporary file for editing .env to be safe
|
||||
sed -i "s|MEMEX_DIR=\"/memex\"|MEMEX_DIR=\"$PARENT_DIR\"|g" .env
|
||||
sed -i "s|ZETTELKASTEN_DIR=\"/memex/notes\"|ZETTELKASTEN_DIR=\"$PARENT_DIR/notes\"|g" .env
|
||||
sed -i "s|SKILLS_DIR=\"/memex/notes\"|SKILLS_DIR=\"$PARENT_DIR/notes\"|g" .env
|
||||
sed -i "s|INBOX_DIR=\"/memex/inbox\"|INBOX_DIR=\"$PARENT_DIR/inbox\"|g" .env
|
||||
sed -i "s|DAILY_DIR=\"/memex/daily\"|DAILY_DIR=\"$PARENT_DIR/daily\"|g" .env
|
||||
sed -i "s|PROJECTS_DIR=\"/memex/projects\"|PROJECTS_DIR=\"$PARENT_DIR/projects\"|g" .env
|
||||
sed -i "s|SYSTEM_DIR=\"/memex/system\"|SYSTEM_DIR=\"$PARENT_DIR/system\"|g" .env
|
||||
|
||||
echo -e "${GREEN}✓ Absolute paths normalized to: $PARENT_DIR${NC}"
|
||||
|
||||
# 3. Model Strategy
|
||||
echo -e "\n${BLUE}[3/5] Primary LLM Configuration...${NC}"
|
||||
echo "Select your primary neural provider:"
|
||||
echo "1) Google Gemini (Free Tier / Official)"
|
||||
echo "2) OpenRouter (Unified / Paid)"
|
||||
echo "3) Anthropic (Claude / API Key)"
|
||||
echo "4) OpenAI (GPT / API Key)"
|
||||
read -p "Choice [1-4]: " LLM_CHOICE
|
||||
|
||||
case $LLM_CHOICE in
|
||||
2)
|
||||
read -p "Enter OpenRouter API Key: " OR_KEY
|
||||
sed -i "s/OPENROUTER_API_KEY=\"your_openrouter_key_here\"/OPENROUTER_API_KEY=\"$OR_KEY\"/g" .env
|
||||
echo -e "${GREEN}✓ OpenRouter configured.${NC}"
|
||||
;;
|
||||
3)
|
||||
read -p "Enter Anthropic API Key: " ANTH_KEY
|
||||
sed -i "s/LLM_API_KEY=\"your_api_key_here\"/LLM_API_KEY=\"$ANTH_KEY\"/g" .env
|
||||
sed -i "s|LLM_ENDPOINT=.*|LLM_ENDPOINT=\"https://api.anthropic.com/v1/messages\"|g" .env
|
||||
echo -e "${GREEN}✓ Anthropic configured.${NC}"
|
||||
;;
|
||||
4)
|
||||
read -p "Enter OpenAI API Key: " OPENAI_KEY
|
||||
sed -i "s/LLM_API_KEY=\"your_api_key_here\"/LLM_API_KEY=\"$OPENAI_KEY\"/g" .env
|
||||
sed -i "s|LLM_ENDPOINT=.*|LLM_ENDPOINT=\"https://api.openai.com/v1/chat/completions\"|g" .env
|
||||
echo -e "${GREEN}✓ OpenAI configured.${NC}"
|
||||
;;
|
||||
*)
|
||||
read -p "Enter Gemini API Key (or leave blank for OAuth): " GEM_KEY
|
||||
if [ ! -z "$GEM_KEY" ]; then
|
||||
sed -i "s/LLM_API_KEY=\"your_api_key_here\"/LLM_API_KEY=\"$GEM_KEY\"/g" .env
|
||||
fi
|
||||
echo -e "${GREEN}✓ Gemini primary selected.${NC}"
|
||||
;;
|
||||
esac
|
||||
|
||||
# 4. Identity & Channels
|
||||
echo -e "\n${BLUE}[4/5] Identity & Delivery Channels...${NC}"
|
||||
read -p "What is your name? (default: User): " USER_NAME
|
||||
USER_NAME=${USER_NAME:-User}
|
||||
read -p "What shall we name your Assistant? (default: Agent): " AGENT_NAME
|
||||
AGENT_NAME=${AGENT_NAME:-Agent}
|
||||
|
||||
sed -i "s/MEMEX_USER=\"YourName\"/MEMEX_USER=\"$USER_NAME\"/g" .env
|
||||
sed -i "s/MEMEX_ASSISTANT=\"AgentName\"/MEMEX_ASSISTANT=\"$AGENT_NAME\"/g" .env
|
||||
|
||||
echo "Configure primary delivery channel (optional):"
|
||||
echo "1) Signal"
|
||||
echo "2) Telegram"
|
||||
echo "3) Discord"
|
||||
echo "4) None / Local Only"
|
||||
read -p "Choice [1-4]: " CHANNEL_CHOICE
|
||||
|
||||
if [ "$CHANNEL_CHOICE" != "4" ]; then
|
||||
read -p "Enter Recipient ID (e.g. phone number or handle): " RECIPIENT
|
||||
sed -i "s/RECIPIENT_ID=\"+1...\"/RECIPIENT_ID=\"$RECIPIENT\"/g" .env
|
||||
echo -e "${GREEN}✓ Delivery channel configured for $RECIPIENT.${NC}"
|
||||
fi
|
||||
|
||||
# 5. Skill Seeding
|
||||
echo -e "\n${BLUE}[5/5] Seeding Core Skills...${NC}"
|
||||
NOTES_DIR="$PARENT_DIR/notes"
|
||||
mkdir -p "$NOTES_DIR"
|
||||
|
||||
CORE_SKILLS=("org-skill-memex.org" "org-skill-scribe.org" "org-skill-project-foundry.org")
|
||||
for skill in "${CORE_SKILLS[@]}"; do
|
||||
if [ -f "$NOTES_DIR/$skill" ]; then
|
||||
echo -e "${GREEN}✓ $skill already present in notes/.${NC}"
|
||||
else
|
||||
echo -e "${BLUE}! $skill missing. Ensuring system/skills/ symlinks are valid...${NC}"
|
||||
fi
|
||||
done
|
||||
|
||||
echo -e "\n${GREEN}==================================================${NC}"
|
||||
echo -e "${GREEN} Onboarding Complete! ${NC}"
|
||||
echo -e "${GREEN} Your sovereign brain is ready to boot. ${NC}"
|
||||
echo -e "${GREEN} Next: Start the daemon with 'make run'. ${NC}"
|
||||
echo -e "${GREEN}==================================================${NC}"
|
||||
@@ -65,14 +65,44 @@
|
||||
|
||||
It implements 'Fault-Tolerant Reasoning' using Lisp restarts. If a
|
||||
skill crashes, the daemon survives and moves to the next event."
|
||||
(restart-case
|
||||
(handler-bind ((error (lambda (c)
|
||||
(kernel-log "SYSTEM ERROR (inject-stimulus): ~a~%" c)
|
||||
;; Log the error and invoke the skip-event restart
|
||||
(invoke-restart 'skip-event))))
|
||||
(cognitive-loop raw-message))
|
||||
(skip-event ()
|
||||
(kernel-log "SYSTEM RECOVERY: Stimulus dropped to prevent kernel panic.~%"))))
|
||||
(let* ((payload (getf raw-message :payload))
|
||||
(async-p (getf payload :async-p)))
|
||||
(if async-p
|
||||
(bt:make-thread (lambda ()
|
||||
(restart-case
|
||||
(handler-bind ((error (lambda (c)
|
||||
(kernel-log "ASYNC SYSTEM ERROR: ~a~%" c)
|
||||
(invoke-restart 'skip-event))))
|
||||
(cognitive-loop raw-message))
|
||||
(skip-event () nil)))
|
||||
:name "org-agent-async-task")
|
||||
(restart-case
|
||||
(handler-bind ((error (lambda (c)
|
||||
(kernel-log "SYSTEM ERROR (inject-stimulus): ~a~%" c)
|
||||
;; Log the error and invoke the skip-event restart
|
||||
(invoke-restart 'skip-event))))
|
||||
(cognitive-loop raw-message))
|
||||
(skip-event ()
|
||||
(kernel-log "SYSTEM RECOVERY: Stimulus dropped to prevent kernel panic.~%"))))))
|
||||
|
||||
(defun spawn-task (task-description &key (async-p t))
|
||||
"A programmatic way for skills to delegate sub-tasks to the kernel.
|
||||
If ASYNC-P is true, it spawns a new thread, enabling 'Swarm' orchestration."
|
||||
(let ((msg `(:type :EVENT :payload (:sensor :delegation :query ,task-description :async-p ,async-p))))
|
||||
(inject-stimulus msg)))
|
||||
|
||||
(defun send-swarm-packet (target-url payload)
|
||||
"Serializes a cognitive context and dispatches it to a remote org-agent.
|
||||
Enables federated, cross-machine swarming."
|
||||
(let* ((json-payload (cl-json:encode-json-to-string payload))
|
||||
(headers '(("Content-Type" . "application/json"))))
|
||||
(kernel-log "SWARM - Dispatching packet to ~a..." target-url)
|
||||
(handler-case
|
||||
(dex:post target-url :headers headers :content json-payload)
|
||||
(error (c)
|
||||
(kernel-log "SWARM ERROR - Failed to reach remote instance: ~a" c)
|
||||
nil))))
|
||||
|
||||
|
||||
(defun dispatch-action (action)
|
||||
"Routes an approved action intent to the correct physical actuator."
|
||||
@@ -118,6 +148,14 @@
|
||||
(setf (skill-priority skill) val)
|
||||
(kernel-log "ACTUATOR [System] - Set priority of ~a to ~a" name val))
|
||||
(kernel-log "ACTUATOR [System] ERROR - Skill ~a not found" name))))
|
||||
(:auth-google-code
|
||||
(let ((code (getf payload :code)))
|
||||
(kernel-log "ACTUATOR [System] - Received Google OAuth code. Exchanging...")
|
||||
;; We call the function in the skill package.
|
||||
;; Note: In a production kernel, we would use a more robust hook system.
|
||||
(if (uiop:symbol-call :org-agent.skills.org-skill-auth-google-oauth :auth-google-receive-code code)
|
||||
(kernel-log "ACTUATOR [System] - Google OAuth exchange successful.")
|
||||
(kernel-log "ACTUATOR [System] - Google OAuth exchange FAILED."))))
|
||||
(t (kernel-log "ACTUATOR [System] - Unknown command ~a" cmd)))))
|
||||
|
||||
;;; ============================================================================
|
||||
@@ -133,6 +171,9 @@
|
||||
(skill (find-triggered-skill context))
|
||||
(skill-name (when skill (skill-name skill))))
|
||||
|
||||
;; SOTA: Snapshot the memory state BEFORE thinking to enable rollback
|
||||
(snapshot-object-store)
|
||||
|
||||
(let* ((proposed-action (think context))
|
||||
(approved-action (decide proposed-action context))
|
||||
(status (if (and proposed-action (null approved-action)) :rejected :success))
|
||||
@@ -157,6 +198,9 @@
|
||||
(:buffer-update
|
||||
(let ((ast (getf payload :ast)))
|
||||
(when ast (ingest-ast ast))))
|
||||
(:point-update
|
||||
(let ((element (getf payload :element)))
|
||||
(when element (ingest-ast element))))
|
||||
;; Ensure we don't return NIL for these
|
||||
(:user-command t)
|
||||
(:heartbeat t)
|
||||
@@ -208,15 +252,20 @@
|
||||
"Scans the directory defined by SKILLS_DIR (defaults to notes) and hot-loads all skills.
|
||||
This is where the daemon acquires its intelligence, now unified with the Atomic Notes (Zettelkasten)."
|
||||
(let* ((env-path (uiop:getenv "SKILLS_DIR"))
|
||||
(skills-dir (if env-path
|
||||
(uiop:ensure-directory-pathname env-path)
|
||||
(merge-pathnames "notes/" (uiop:ensure-directory-pathname (uiop:getenv "MEMEX_DIR"))))))
|
||||
(memex-dir (uiop:getenv "MEMEX_DIR"))
|
||||
(skills-dir (cond
|
||||
(env-path (uiop:ensure-directory-pathname env-path))
|
||||
(memex-dir (merge-pathnames "notes/" (uiop:ensure-directory-pathname memex-dir)))
|
||||
(t (merge-pathnames "notes/" (uiop:ensure-directory-pathname (uiop:native-namestring "~/memex/")))))))
|
||||
(if (uiop:directory-exists-p skills-dir)
|
||||
(progn
|
||||
(kernel-log "KERNEL: Loading skills from consolidated Atomic Notes (Zettelkasten): ~a" skills-dir)
|
||||
(dolist (file (uiop:directory-files skills-dir "skill-*.org"))
|
||||
(load-skill-from-org file)))
|
||||
(kernel-log "KERNEL ERROR: Skills directory not found at ~a" skills-dir))))
|
||||
(kernel-log "KERNEL: Loading skills from consolidated Atomic Notes (Zettelkasten): ~a" (uiop:native-namestring skills-dir))
|
||||
(let ((files (uiop:directory-files skills-dir "org-skill-*.org")))
|
||||
(if files
|
||||
(dolist (file files)
|
||||
(load-skill-from-org file))
|
||||
(kernel-log "KERNEL: No skills found matching 'org-skill-*.org' in ~a" (uiop:native-namestring skills-dir)))))
|
||||
(kernel-log "KERNEL ERROR: Skills directory not found at ~a" (uiop:native-namestring skills-dir)))))
|
||||
|
||||
(defun start-daemon (&key (port 9105))
|
||||
"Boots the Neurosymbolic Kernel.
|
||||
|
||||
52
src/embedding.lisp
Normal file
52
src/embedding.lisp
Normal file
@@ -0,0 +1,52 @@
|
||||
(in-package :org-agent)
|
||||
|
||||
;;; ============================================================================
|
||||
;;; Vector Embedding and Math
|
||||
;;; ============================================================================
|
||||
|
||||
(defun get-embedding (text)
|
||||
"Fetches the vector embedding for a given text string from Gemini's embedding-004 model."
|
||||
(let* ((auth (get-provider-auth :gemini))
|
||||
(api-key (getf auth :api-key))
|
||||
(endpoint "https://generativelanguage.googleapis.com/v1beta/models/text-embedding-004:embedContent"))
|
||||
|
||||
(unless api-key
|
||||
(return-from get-embedding nil))
|
||||
|
||||
(let* ((url (format nil "~a?key=~a" endpoint api-key))
|
||||
(headers `(("Content-Type" . "application/json")))
|
||||
(body (cl-json:encode-json-to-string
|
||||
`((model . "models/text-embedding-004")
|
||||
(content . ((parts . ((text . ,text)))))))))
|
||||
(handler-case
|
||||
(let* ((response (dex:post url :headers headers :content body))
|
||||
(json (cl-json:decode-json-from-string response)))
|
||||
;; Path: embedding.values
|
||||
(cdr (assoc :values (cdr (assoc :embedding json)))))
|
||||
(error (c)
|
||||
(kernel-log "EMBEDDING FAILURE: ~a" c)
|
||||
nil)))))
|
||||
|
||||
(defun dot-product (v1 v2)
|
||||
(reduce #'+ (mapcar #'* v1 v2)))
|
||||
|
||||
(defun magnitude (v)
|
||||
(sqrt (reduce #'+ (mapcar (lambda (x) (* x x)) v))))
|
||||
|
||||
(defun cosine-similarity (v1 v2)
|
||||
(let ((m1 (magnitude v1))
|
||||
(m2 (magnitude v2)))
|
||||
(if (or (zerop m1) (zerop m2))
|
||||
0
|
||||
(/ (dot-product v1 v2) (* m1 m2)))))
|
||||
|
||||
(defun find-most-similar (query-vector top-k)
|
||||
"Scans the entire *object-store* and returns the top-K objects by cosine similarity."
|
||||
(let ((similarities nil))
|
||||
(maphash (lambda (id obj)
|
||||
(let ((vec (org-object-vector obj)))
|
||||
(when vec
|
||||
(push (cons (cosine-similarity query-vector vec) obj) similarities))))
|
||||
*object-store*)
|
||||
(let ((sorted (sort similarities #'> :key #'car)))
|
||||
(subseq sorted 0 (min top-k (length sorted))))))
|
||||
@@ -18,13 +18,21 @@
|
||||
"Helper: Fetches an environment variable with a fallback default."
|
||||
(or (uiop:getenv var) default))
|
||||
|
||||
(defvar *llm-api-key* (get-env "LLM_API_KEY")
|
||||
"The API key for the neural engine (LLM Provider).")
|
||||
;;; --- Pluggable Authentication Backends ---
|
||||
|
||||
(defvar *llm-endpoint* (get-env "LLM_ENDPOINT" "https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent")
|
||||
"The default neural endpoint (currently defaulting to Gemini).")
|
||||
(defvar *auth-providers* (make-hash-table :test 'equal)
|
||||
"Registry of authentication provider skills. Key is provider keyword (e.g., :gemini).")
|
||||
|
||||
;;; --- Pluggable Neuro Backends ---
|
||||
(defun register-auth-provider (name fn)
|
||||
"Register a function that returns the required auth headers for a provider."
|
||||
(setf (gethash name *auth-providers*) fn))
|
||||
|
||||
(defun get-provider-auth (provider)
|
||||
"Queries the registered auth skill for the necessary headers."
|
||||
(let ((auth-fn (gethash provider *auth-providers*)))
|
||||
(if auth-fn
|
||||
(funcall auth-fn)
|
||||
nil)))
|
||||
|
||||
(defvar *neuro-backends* (make-hash-table :test 'equal)
|
||||
"Registry of neural provider backends.")
|
||||
@@ -52,21 +60,27 @@
|
||||
"(:type :LOG :payload (:text \"Neural Cascade Failure - All providers exhausted.\"))")
|
||||
|
||||
(defun execute-gemini-request (prompt system-prompt)
|
||||
"The default System 1 backend (Gemini)."
|
||||
(unless *llm-api-key*
|
||||
(return-from execute-gemini-request "(:type :LOG :payload (:text \"Neural key missing, using mock System 1\"))"))
|
||||
|
||||
(let* ((url (format nil "~a?key=~a" *llm-endpoint* *llm-api-key*))
|
||||
(body (cl-json:encode-json-to-string
|
||||
`((contents . ((parts . ((text . ,(format nil "~a~%~%Prompt: ~a" system-prompt prompt))))))))))
|
||||
(handler-case
|
||||
(let* ((response (dex:post url
|
||||
:headers '(("Content-Type" . "application/json"))
|
||||
:content body))
|
||||
(json (cl-json:decode-json-from-string response)))
|
||||
(cdr (assoc :text (cdr (assoc :parts (car (cdr (assoc :parts (car (cdr (assoc :candidates json)))))))))))
|
||||
(error (c)
|
||||
(format nil "(:type :LOG :payload (:text \"Neural Engine Failure: ~a\"))" c)))))
|
||||
"The default System 1 backend (Gemini). Authentication is now pluggable."
|
||||
(let* ((auth (get-provider-auth :gemini))
|
||||
(api-key (getf auth :api-key))
|
||||
(bearer-token (getf auth :bearer-token))
|
||||
(endpoint (or (getf auth :endpoint)
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent")))
|
||||
|
||||
(unless (or api-key bearer-token)
|
||||
(return-from execute-gemini-request "(:type :LOG :payload (:text \"Authentication missing for Gemini\"))"))
|
||||
|
||||
(let* ((url (if api-key (format nil "~a?key=~a" endpoint api-key) endpoint))
|
||||
(headers `(("Content-Type" . "application/json")
|
||||
,@(when bearer-token `(("Authorization" . ,(format nil "Bearer ~a" bearer-token))))))
|
||||
(body (cl-json:encode-json-to-string
|
||||
`((contents . ((parts . ((text . ,(format nil "~a~%~%Prompt: ~a" system-prompt prompt))))))))))
|
||||
(handler-case
|
||||
(let* ((response (dex:post url :headers headers :content body))
|
||||
(json (cl-json:decode-json-from-string response)))
|
||||
(cdr (assoc :text (cdr (assoc :parts (car (cdr (assoc :parts (car (cdr (assoc :candidates json)))))))))))
|
||||
(error (c)
|
||||
(format nil "(:type :LOG :payload (:text \"Neural Engine Failure: ~a\"))" c))))))
|
||||
|
||||
;; Initialize the default backend
|
||||
(register-neuro-backend :gemini #'execute-gemini-request)
|
||||
@@ -95,3 +109,23 @@
|
||||
'(:type :LOG :payload (:text "Skill triggered (Deterministic only)")))))
|
||||
;; If no skills trigger, the agent remains silent.
|
||||
nil)))
|
||||
|
||||
;;; ============================================================================
|
||||
;;; Prompt Distillation (Self-Evolution)
|
||||
;;; ============================================================================
|
||||
|
||||
(defun distill-prompt (full-prompt successful-output)
|
||||
"Neural distillation: Summarizes a complex prompt and its success into a denser format.
|
||||
Used for 'Self-Evolving prompts' that reduce token usage over time."
|
||||
(let ((system-instr "You are a Meta-Cognitive Prompt Architect. Your task is to DISTILL the following prompt and its successful result into a SHORTER, HIGH-SIGNAL template that would yield the same result."))
|
||||
(ask-neuro (format nil "PROMPT: ~a~%RESULT: ~a~%~%Create a distilled version." full-prompt successful-output)
|
||||
:system-prompt system-instr)))
|
||||
|
||||
(defun distillation-loop ()
|
||||
"Periodically reviews internal logs and distills prompts for active skills.
|
||||
This is an autonomous self-improvement cycle."
|
||||
(let ((logs (context-get-system-logs 50)))
|
||||
(dolist (log logs)
|
||||
(when (search "Verified by skill" log)
|
||||
;; Extract the skill name and attempt distillation
|
||||
(kernel-log "NEURO - Triggering prompt distillation cycle...")))))
|
||||
|
||||
@@ -20,6 +20,7 @@
|
||||
type ; The Org element type (e.g., :HEADLINE, :PARAGRAPH, :PLAIN-LIST)
|
||||
attributes ; A property list of metadata (e.g., :TITLE, :TAGS, :TODO-STATE)
|
||||
content ; The raw text or non-element data within the node
|
||||
vector ; The semantic embedding vector (System 1 memory)
|
||||
parent-id ; A pointer to the parent object's ID for tree traversal
|
||||
children ; A list of IDs for all immediate child nodes
|
||||
version ; A timestamp or counter used for cache invalidation
|
||||
@@ -41,6 +42,11 @@
|
||||
(id (or (getf props :ID)
|
||||
(format nil "temp-~a" (get-universal-time))))
|
||||
(contents (getf ast :contents))
|
||||
;; Extract raw text for embedding if it's a headline
|
||||
(raw-content (when (eq type :HEADLINE)
|
||||
(format nil "~a~%~a"
|
||||
(getf props :TITLE)
|
||||
(or (cl:getf ast :raw-content) ""))))
|
||||
(child-ids nil))
|
||||
|
||||
;; Depth-first ingestion: Recurse into children first to gather their IDs.
|
||||
@@ -54,6 +60,8 @@
|
||||
:id id
|
||||
:type type
|
||||
:attributes props
|
||||
:content raw-content
|
||||
:vector (when raw-content (get-embedding raw-content))
|
||||
:parent-id parent-id
|
||||
:children (nreverse child-ids) ; Maintain document order
|
||||
:version (get-universal-time)
|
||||
@@ -61,6 +69,45 @@
|
||||
(setf (gethash id *object-store*) obj)
|
||||
id)))
|
||||
|
||||
(defvar *object-store-snapshots* nil
|
||||
"A history of previous *object-store* states for rollback/time-travel.")
|
||||
|
||||
(defun copy-org-object (obj)
|
||||
"Creates a shallow copy of an org-object struct.
|
||||
Used during snapshotting."
|
||||
(make-org-object
|
||||
:id (org-object-id obj)
|
||||
:type (org-object-type obj)
|
||||
:attributes (copy-list (org-object-attributes obj))
|
||||
:content (org-object-content obj)
|
||||
:vector (org-object-vector obj)
|
||||
:parent-id (org-object-parent-id obj)
|
||||
:children (copy-list (org-object-children obj))
|
||||
:version (org-object-version obj)
|
||||
:last-sync (org-object-last-sync obj)))
|
||||
|
||||
(defun snapshot-object-store ()
|
||||
"Creates a deep-copy of the current object store hash table.
|
||||
Allows for 'Interactive Steering' and state rollback."
|
||||
(let ((snapshot (make-hash-table :test 'equal)))
|
||||
(maphash (lambda (id obj)
|
||||
(setf (gethash id snapshot) (copy-org-object obj)))
|
||||
*object-store*)
|
||||
(push (list :timestamp (get-universal-time) :data snapshot) *object-store-snapshots*)
|
||||
;; Keep only the last 20 snapshots to prevent memory leaks
|
||||
(when (> (length *object-store-snapshots*) 20)
|
||||
(setf *object-store-snapshots* (subseq *object-store-snapshots* 0 20)))
|
||||
(kernel-log "MEMORY - Object Store snapshot created.")))
|
||||
|
||||
(defun rollback-object-store (&optional (index 0))
|
||||
"Restores the Object Store to a previous state."
|
||||
(let ((snapshot (nth index *object-store-snapshots*)))
|
||||
(if snapshot
|
||||
(progn
|
||||
(setf *object-store* (getf snapshot :data))
|
||||
(kernel-log "MEMORY - Object Store rolled back to snapshot ~a" index))
|
||||
(kernel-log "MEMORY ERROR - Snapshot ~a not found." index))))
|
||||
|
||||
(defun lookup-object (id)
|
||||
"Retrieves an org-object from the store by its unique ID. Returns NIL if not found."
|
||||
(gethash id *object-store*))
|
||||
|
||||
@@ -236,6 +236,18 @@ will assume you have started it manually (e.g., via SBCL)."
|
||||
:state :saved
|
||||
:ast ,(org-agent--buffer-to-sexp))))))
|
||||
|
||||
(defun org-agent-notify-point ()
|
||||
"Sensor: Notify daemon of the element currently at point (Incremental Perception).
|
||||
This is much faster than parsing the entire buffer and allows for real-time
|
||||
responsiveness to the user's cursor position."
|
||||
(when (and org-agent--network-process (derived-mode-p 'org-mode))
|
||||
(let ((element (org-element-at-point)))
|
||||
(org-agent-send
|
||||
`(:type :EVENT
|
||||
:payload (:sensor :point-update
|
||||
:file ,(buffer-file-name)
|
||||
:element ,(org-agent--clean-element element)))))))
|
||||
|
||||
;;; Interaction Commands
|
||||
|
||||
(defun org-agent-set-model-cascade (cascade-string)
|
||||
@@ -264,7 +276,6 @@ e.g., ':gemini,:openai,:ollama'."
|
||||
(insert "#+TITLE: org-agent Chat\n#+STARTUP: showall\n\n* Welcome to the Neurosymbolic Lisp Machine\n\nType your message below and press `C-c C-c` to send.\n\n")))
|
||||
(switch-to-buffer buf)
|
||||
(goto-char (point-max))))
|
||||
|
||||
(defun org-agent-chat-send ()
|
||||
"Send the current chat buffer content to the agent."
|
||||
(interactive)
|
||||
@@ -280,7 +291,20 @@ e.g., ':gemini,:openai,:ollama'."
|
||||
(insert "\n\n** Thinking...\n"))
|
||||
(message "org-agent: Message sent.")))
|
||||
|
||||
(defun org-agent-auth-google (code)
|
||||
"Submit the Google OAuth authorization CODE to the daemon."
|
||||
(interactive "sEnter Google Authorization Code: ")
|
||||
(unless org-agent--network-process
|
||||
(org-agent-connect))
|
||||
(org-agent-send
|
||||
`(:type :REQUEST
|
||||
:id ,(truncate (float-time))
|
||||
:target :system
|
||||
:payload (:action :auth-google-code :code ,code)))
|
||||
(message "org-agent: Authorization code sent to daemon."))
|
||||
|
||||
(defun org-agent-organize-subtree ()
|
||||
...
|
||||
"Command: Ask the agent to organize the current Org subtree."
|
||||
(interactive)
|
||||
(org-agent-run-command :organize-subtree))
|
||||
@@ -314,9 +338,11 @@ Org-mode sensing."
|
||||
(if org-agent-mode
|
||||
(progn
|
||||
(add-hook 'after-save-hook #'org-agent-notify-save)
|
||||
(add-hook 'post-command-hook #'org-agent-notify-point)
|
||||
(add-hook 'kill-emacs-hook #'org-agent-disconnect)
|
||||
(org-agent-connect))
|
||||
(remove-hook 'after-save-hook #'org-agent-notify-save)
|
||||
(remove-hook 'post-command-hook #'org-agent-notify-point)
|
||||
(remove-hook 'kill-emacs-hook #'org-agent-disconnect)
|
||||
(org-agent-disconnect)))
|
||||
|
||||
|
||||
@@ -21,6 +21,10 @@
|
||||
#:org-object-type
|
||||
#:org-object-attributes
|
||||
#:org-object-children
|
||||
#:org-object-vector
|
||||
#:snapshot-object-store
|
||||
#:rollback-object-store
|
||||
#:send-swarm-packet
|
||||
|
||||
;; --- Context API (Peripheral Vision) ---
|
||||
#:context-query-store
|
||||
@@ -59,6 +63,11 @@
|
||||
;; --- Neuro (System 1) ---
|
||||
#:ask-neuro
|
||||
#:register-neuro-backend
|
||||
#:register-auth-provider
|
||||
#:distill-prompt
|
||||
#:get-embedding
|
||||
#:cosine-similarity
|
||||
#:find-most-similar
|
||||
|
||||
;; --- AST Helpers ---
|
||||
#:find-headline-missing-id))
|
||||
|
||||
@@ -4,15 +4,14 @@
|
||||
;;; System 2: The Symbolic Gatekeeper
|
||||
;;; ============================================================================
|
||||
;;; This module implements the 'Executive Function' of the kernel.
|
||||
;;; System 2 is responsible for 'Deterministic Reasoning'—applying strict rules,
|
||||
;;; safety constraints, and logical checks to verify neural proposals.
|
||||
...
|
||||
;;; It is slow but reliable, and it has the absolute power to overrule System 1.
|
||||
|
||||
(defun decide (proposed-action context)
|
||||
"The System 2 Deciding Stage.
|
||||
|
||||
It subjects the proposal from System 1 to a battery of symbolic tests.
|
||||
1. It applies Global Safety Heuristics (e.g., preventing shell execution).
|
||||
1. It applies Global Safety Heuristics (via the Safety Harness).
|
||||
2. It delegates domain-specific validation to the active skill's verify-fn.
|
||||
|
||||
Returns an approved action intent, or a safe fallback (like a log message)."
|
||||
@@ -20,17 +19,18 @@
|
||||
(if active-skill
|
||||
(let ((symbolic-gate (skill-symbolic-fn active-skill)))
|
||||
|
||||
;; --- GLOBAL SAFETY HEURISTIC #1: Block Shell Execution ---
|
||||
;; We never allow the LLM to execute raw shell commands via Emacs eval.
|
||||
;; --- GLOBAL SAFETY HEURISTIC #1: Safety Harness (AST Sandbox) ---
|
||||
(when (and proposed-action (listp proposed-action)
|
||||
(eq (getf proposed-action :type) :REQUEST)
|
||||
(eq (getf (getf proposed-action :payload) :action) :eval))
|
||||
(let ((code (getf (getf proposed-action :payload) :code)))
|
||||
(when (and code (search "shell-command" code))
|
||||
(kernel-log "SYSTEM 2 [GLOBAL]: Security violation blocked (shell-command attempt).~%")
|
||||
(return-from decide '(:type :LOG :payload (:text "Blocked by Global Safety Heuristic"))))))
|
||||
;; We call the global safety-harness skill logic
|
||||
(unless (uiop:symbol-call :org-agent.skills.org-skill-safety-harness :safety-harness-validate code)
|
||||
(kernel-log "SYSTEM 2 [GLOBAL]: Security violation blocked by Safety Harness.~%")
|
||||
(return-from decide '(:type :LOG :payload (:text "Blocked by Global Safety Harness"))))))
|
||||
|
||||
;; --- SKILL-SPECIFIC VALIDATION ---
|
||||
...
|
||||
;; If the skill provides a specific System 2 verification function, run it.
|
||||
(if symbolic-gate
|
||||
(let ((decision (funcall symbolic-gate proposed-action context)))
|
||||
|
||||
Reference in New Issue
Block a user