feat(kernel): upgrade neurosymbolic kernel to Order 2 (recursive self-maintenance, SOTA upgrades)

docs/PRD.org

@@ -1,29 +0,0 @@
-#+TITLE: PRD: org-agent Cognitive Core & Configuration
-#+AUTHOR: PSF Requirements Definer
-#+CREATED: [2026-03-23 Mon]
-#+STATUS: FROZEN
-
-* 0. Core Mandates
-The `org-agent` project MUST adhere to these foundational mandates:
-- **Mandate 1: Strict Homoiconic Memory.** All documentation, planning, and system logic MUST be authored in Org-mode (.org) and Common Lisp. Markdown (.md) and JSON are strictly prohibited for internal use.
-- **Mandate 2: Minimalist Microkernel.** The core daemon MUST remain minimalist, handling only the cognitive loop, the persistent Object-Store, and the communication protocol. All domain-specific features and LLM provider logic MUST be implemented as hot-reloadable **Skills** living in the user's Memex.
-
-* 1. Purpose
-The `org-agent` must transition from a monolithic prototype to a generalized neurosymbolic kernel.
-
-* 2. Functional Requirements
-** 2.1. Cognitive Loop (PTA Refactor)
-- The daemon MUST implement a 4-stage pipeline: Perceive -> Think -> Decide -> Act.
-- System 1 (Neural) MUST be restricted to the 'Think' stage.
-- System 2 (Symbolic) MUST have absolute authority in the 'Decide' stage to block or modify neural proposals.
-- The I/O protocol (OACP) MUST be encapsulated in 'Perceive' and 'Act'.
-
-** 2.2. Externalized Configuration (.env)
-- All secrets (API keys) and environment-specific settings (ports, paths) MUST live in a `.env` file.
-- The system MUST automatically load `.env` upon system initialization.
-- Secrets MUST NOT be hardcoded or checked into source control.
-
-* 3. Success Criteria
- - TODO Daemon starts and loads LLM_API_KEY from .env.
- - TODO `cognitive-loop` successfully routes a `:buffer-update` event through all 4 stages.
- - TODO System 2 (`decide`) successfully blocks an `:eval` request containing "shell-command".

docs/PROTOCOL.org

@@ -1,137 +0,0 @@
-#+TITLE: org-agent Communication Protocol (OACP)
-#+AUTHOR: Agent
-#+DATE: 2026-03-22
-#+ID: org-agent-protocol
-#+STARTUP: content
-
-* Core Mandates
-The OACP and the `org-agent` project MUST adhere to these foundational mandates:
-- **Mandate 1: Strict Homoiconic Memory.** All documentation, planning, and system logic MUST be authored in Org-mode (.org) and Common Lisp. Markdown (.md) and JSON are strictly prohibited for internal use.
-- **Mandate 2: Minimalist Microkernel.** The core daemon MUST remain minimalist, handling only the cognitive loop, the persistent Object-Store, and the communication protocol. All domain-specific features and LLM provider logic MUST be implemented as hot-reloadable **Skills** living in the user's Memex.
-
-* Overview
-
-OACP (org-agent Communication Protocol) defines the "nervous system" of the Neurosymbolic Lisp Machine. It facilitates bidirectional, asynchronous communication between the **Common Lisp Core** (The Soul/Daemon) and the **Emacs Interface** (The Actuator/Terminal).
-
-The protocol is designed to be:
-- **Homoiconic:** Messages are native Lisp S-expressions (plists).
-- **Asynchronous:** Neither side should block waiting for the other.
-- **Extensible:** New sensors and actuators can be added by defining new plist keys.
-
-* Framing & Transport
-
-- **Transport:** TCP Socket (default port: 9105).
-- **Framing:** Each message is prefixed by a 6-character hexadecimal length string (e.g., `00002a` for a 42-byte message), followed by the S-expression string encoded in UTF-8.
-
-* Message Structure
-
-A standard message is a flat property list (plist):
-
-#+begin_src lisp
-(:type TYPE :id ID :payload PAYLOAD)
-#+end_src
-
-- **:type:** One of `:EVENT`, `:REQUEST`, `:RESPONSE`, or `:LOG`.
-- **:id:** A unique integer or string for correlation (mandatory for `:REQUEST` and `:RESPONSE`).
-- **:payload:** A nested plist containing the specific data or command.
-
-* Perception (Emacs -> Core)
-
-Emacs acts as the sensor array, pushing events to the Core daemon.
-
-** :BUFFER-MODIFIED
-Sent when an Org buffer is changed or saved.
-#+begin_src lisp
-(:type :EVENT :payload (:sensor :buffer-update :file "/home/amr/org/todo.org" :state :saved))
-#+end_src
-
-** :USER-INPUT
-Sent when the user explicitly triggers an agent action (e.g., `M-x org-agent-ask`).
-#+begin_src lisp
-(:type :EVENT :payload (:sensor :user-prompt :text "Refactor this subtree" :context :current-subtree))
-#+end_src
-
-** :CURSOR-MOVED
-Sent when the agent needs to track the user's focus.
-#+begin_src lisp
-(:type :EVENT :payload (:sensor :focus :buffer "init.el" :line 42 :column 0))
-#+end_src
-
-* Action (Core -> Emacs)
-
-The Core daemon sends requests to Emacs to perform physical actions in the environment.
-
-** :EXECUTE-ELISP
-The primary "actuator."
-#+begin_src lisp
-(:type :REQUEST :id 101 :payload (:action :eval :code "(org-todo \"NEXT\")"))
-#+end_src
-
-** :UI-NOTIFY
-Display info to the user without interrupting focus.
-#+begin_src lisp
-(:type :REQUEST :id 102 :payload (:action :message :text "I have identified a contradiction in your GTD.org" :level :warning))
-#+end_src
-
-** :PROVIDE-COMPLETION
-Provide LLM-driven completions for the current point.
-#+begin_src lisp
-(:type :REQUEST :id 103 :payload (:action :complete :prefix "defun" :suggestions ("defun-agent" "defun-percieve")))
-#+end_src
-
-** :ORG-DELIVERY
-Enqueue external messages.
-#+begin_src lisp
-(:type :REQUEST :target :delivery :payload (:channel :signal :to "+1..." :text "Hello"))
-#+end_src
-
-** :PROJECT-SCAFFOLD
-Create new project folders and headings.
-#+begin_src lisp
-(:type :REQUEST :target :foundry :payload (:action :scaffold :name "Project-X" :type "Lisp"))
-#+end_src
-
-** :SYSTEM-SELF-EDIT
-Kernel-level self-modification.
-#+begin_src lisp
-(:type :REQUEST :target :system :payload (:action :create-skill :filename "skill-x.org" :content "..."))
-#+end_src
-
-* Metadata & Handshake
-
-** :HELLO
-Sent by both sides upon connection.
-#+begin_src lisp
-(:type :EVENT :payload (:action :handshake :version "0.1.0" :capabilities (:auth :swank :org-ast)))
-#+end_src
-
-* Cognitive Loop Internal API (Foundry Standard)
-
-To ensure neurosymbolic sovereignty, the Core Daemon must strictly separate stages.
-
-** 1. Perceive
-- **Signature:** `(perceive raw-stimulus) -> context-plist`
-- **Responsibility:** Protocol decoding, Object-Store synchronization, context augmentation.
-
-** 2. Think (System 1)
-- **Signature:** `(think context-plist) -> proposed-action-plist`
-- **Responsibility:** Neural pattern matching, associative retrieval, unverified proposal generation.
-
-** 3. Decide (System 2)
-- **Signature:** `(decide proposed-action context) -> approved-action-plist`
-- **Responsibility:** Symbolic verification, safety gating, deterministic rule application (e.g., GTD logic).
-
-** 4. Act (Dispatch)
-- **Signature:** `(act stream approved-action)` / `(dispatch-action approved-action)`
-- **Responsibility:** Actuator lookup and transmission. Targets: `:emacs`, `:delivery`, `:system`, `:shell`.
-
-** 5. Context API (System 1 Peripheral Vision)
-- **(context-list-all-skills):** Introspect the brain hierarchy.
-- **(context-get-skill-source name):** Read the logic graph.
-- **(context-resolve-path path):** Environment-relative path expansion.
-- **(context-get-system-logs limit):** Perception of kernel failures.
-
-* Success Metrics
-- Latency between `:EVENT` and `:RESPONSE` < 100ms for UI actions.
-- 100% fidelity in Org AST preservation across the bridge.
-- Zero UI blocking in Emacs during heavy CL reasoning loops.

org-agent.asd

@@ -10,6 +10,7 @@
                 :components ((:file "package")
                              (:file "protocol")
                              (:file "object-store")
+                             (:file "embedding")
                              (:file "skills")
                              (:file "neuro")
                              (:file "symbolic")

scripts/onboard.sh

@@ -0,0 +1,136 @@
+#!/bin/bash
+
+# org-agent Onboarding Script: The First Breath
+# This script prepares your PSF environment for the Lisp Machine.
+
+set -e
+
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+echo -e "${BLUE}==================================================${NC}"
+echo -e "${BLUE} org-agent: Personal Software Foundry Onboarding  ${NC}"
+echo -e "${BLUE}==================================================${NC}"
+
+# 1. Environment Verification
+echo -e "\n${BLUE}[1/5] Verifying Environment...${NC}"
+
+if command -v sbcl >/dev/null 2>&1; then
+    echo -e "${GREEN}✓ SBCL (Steel Bank Common Lisp) found.${NC}"
+else
+    echo -e "${RED}✗ SBCL not found. Please install it first.${NC}"
+    exit 1
+fi
+
+if [ -d "$HOME/quicklisp" ] || [ -d "$HOME/.quicklisp" ]; then
+    echo -e "${GREEN}✓ Quicklisp found.${NC}"
+else
+    echo -e "${RED}✗ Quicklisp not found. Please install Quicklisp to manage Lisp dependencies.${NC}"
+    exit 1
+fi
+
+# 2. Configuration Setup
+echo -e "\n${BLUE}[2/5] Setting up .env configuration...${NC}"
+
+if [ ! -f .env ]; then
+    cp .env.example .env
+    echo -e "${GREEN}✓ Created .env from .env.example.${NC}"
+else
+    echo -e "${BLUE}! .env already exists. Skipping creation.${NC}"
+fi
+
+# Set MEMEX_DIR automatically to current parent if not set
+PROJECT_ROOT=$(pwd)
+PARENT_DIR=$(dirname "$PROJECT_ROOT")
+
+# Use a temporary file for editing .env to be safe
+sed -i "s|MEMEX_DIR=\"/memex\"|MEMEX_DIR=\"$PARENT_DIR\"|g" .env
+sed -i "s|ZETTELKASTEN_DIR=\"/memex/notes\"|ZETTELKASTEN_DIR=\"$PARENT_DIR/notes\"|g" .env
+sed -i "s|SKILLS_DIR=\"/memex/notes\"|SKILLS_DIR=\"$PARENT_DIR/notes\"|g" .env
+sed -i "s|INBOX_DIR=\"/memex/inbox\"|INBOX_DIR=\"$PARENT_DIR/inbox\"|g" .env
+sed -i "s|DAILY_DIR=\"/memex/daily\"|DAILY_DIR=\"$PARENT_DIR/daily\"|g" .env
+sed -i "s|PROJECTS_DIR=\"/memex/projects\"|PROJECTS_DIR=\"$PARENT_DIR/projects\"|g" .env
+sed -i "s|SYSTEM_DIR=\"/memex/system\"|SYSTEM_DIR=\"$PARENT_DIR/system\"|g" .env
+
+echo -e "${GREEN}✓ Absolute paths normalized to: $PARENT_DIR${NC}"
+
+# 3. Model Strategy
+echo -e "\n${BLUE}[3/5] Primary LLM Configuration...${NC}"
+echo "Select your primary neural provider:"
+echo "1) Google Gemini (Free Tier / Official)"
+echo "2) OpenRouter (Unified / Paid)"
+echo "3) Anthropic (Claude / API Key)"
+echo "4) OpenAI (GPT / API Key)"
+read -p "Choice [1-4]: " LLM_CHOICE
+
+case $LLM_CHOICE in
+    2)
+        read -p "Enter OpenRouter API Key: " OR_KEY
+        sed -i "s/OPENROUTER_API_KEY=\"your_openrouter_key_here\"/OPENROUTER_API_KEY=\"$OR_KEY\"/g" .env
+        echo -e "${GREEN}✓ OpenRouter configured.${NC}"
+        ;;
+    3)
+        read -p "Enter Anthropic API Key: " ANTH_KEY
+        sed -i "s/LLM_API_KEY=\"your_api_key_here\"/LLM_API_KEY=\"$ANTH_KEY\"/g" .env
+        sed -i "s|LLM_ENDPOINT=.*|LLM_ENDPOINT=\"https://api.anthropic.com/v1/messages\"|g" .env
+        echo -e "${GREEN}✓ Anthropic configured.${NC}"
+        ;;
+    4)
+        read -p "Enter OpenAI API Key: " OPENAI_KEY
+        sed -i "s/LLM_API_KEY=\"your_api_key_here\"/LLM_API_KEY=\"$OPENAI_KEY\"/g" .env
+        sed -i "s|LLM_ENDPOINT=.*|LLM_ENDPOINT=\"https://api.openai.com/v1/chat/completions\"|g" .env
+        echo -e "${GREEN}✓ OpenAI configured.${NC}"
+        ;;
+    *)
+        read -p "Enter Gemini API Key (or leave blank for OAuth): " GEM_KEY
+        if [ ! -z "$GEM_KEY" ]; then
+            sed -i "s/LLM_API_KEY=\"your_api_key_here\"/LLM_API_KEY=\"$GEM_KEY\"/g" .env
+        fi
+        echo -e "${GREEN}✓ Gemini primary selected.${NC}"
+        ;;
+esac
+
+# 4. Identity & Channels
+echo -e "\n${BLUE}[4/5] Identity & Delivery Channels...${NC}"
+read -p "What is your name? (default: User): " USER_NAME
+USER_NAME=${USER_NAME:-User}
+read -p "What shall we name your Assistant? (default: Agent): " AGENT_NAME
+AGENT_NAME=${AGENT_NAME:-Agent}
+
+sed -i "s/MEMEX_USER=\"YourName\"/MEMEX_USER=\"$USER_NAME\"/g" .env
+sed -i "s/MEMEX_ASSISTANT=\"AgentName\"/MEMEX_ASSISTANT=\"$AGENT_NAME\"/g" .env
+
+echo "Configure primary delivery channel (optional):"
+echo "1) Signal"
+echo "2) Telegram"
+echo "3) Discord"
+echo "4) None / Local Only"
+read -p "Choice [1-4]: " CHANNEL_CHOICE
+
+if [ "$CHANNEL_CHOICE" != "4" ]; then
+    read -p "Enter Recipient ID (e.g. phone number or handle): " RECIPIENT
+    sed -i "s/RECIPIENT_ID=\"+1...\"/RECIPIENT_ID=\"$RECIPIENT\"/g" .env
+    echo -e "${GREEN}✓ Delivery channel configured for $RECIPIENT.${NC}"
+fi
+
+# 5. Skill Seeding
+echo -e "\n${BLUE}[5/5] Seeding Core Skills...${NC}"
+NOTES_DIR="$PARENT_DIR/notes"
+mkdir -p "$NOTES_DIR"
+
+CORE_SKILLS=("org-skill-memex.org" "org-skill-scribe.org" "org-skill-project-foundry.org")
+for skill in "${CORE_SKILLS[@]}"; do
+    if [ -f "$NOTES_DIR/$skill" ]; then
+        echo -e "${GREEN}✓ $skill already present in notes/.${NC}"
+    else
+        echo -e "${BLUE}! $skill missing. Ensuring system/skills/ symlinks are valid...${NC}"
+    fi
+done
+
+echo -e "\n${GREEN}==================================================${NC}"
+echo -e "${GREEN} Onboarding Complete!                             ${NC}"
+echo -e "${GREEN} Your sovereign brain is ready to boot.           ${NC}"
+echo -e "${GREEN} Next: Start the daemon with 'make run'.          ${NC}"
+echo -e "${GREEN}==================================================${NC}"

src/core.lisp

@@ -65,14 +65,44 @@
    
    It implements 'Fault-Tolerant Reasoning' using Lisp restarts. If a 
    skill crashes, the daemon survives and moves to the next event."
-  (restart-case
+  (let* ((payload (getf raw-message :payload))
-      (handler-bind ((error (lambda (c)
+         (async-p (getf payload :async-p)))
-                              (kernel-log "SYSTEM ERROR (inject-stimulus): ~a~%" c)
+    (if async-p
-                              ;; Log the error and invoke the skip-event restart
+        (bt:make-thread (lambda () 
-                              (invoke-restart 'skip-event))))
+                          (restart-case
-        (cognitive-loop raw-message))
+                              (handler-bind ((error (lambda (c)
-    (skip-event ()
+                                                      (kernel-log "ASYNC SYSTEM ERROR: ~a~%" c)
-      (kernel-log "SYSTEM RECOVERY: Stimulus dropped to prevent kernel panic.~%"))))
+                                                      (invoke-restart 'skip-event))))
+                                (cognitive-loop raw-message))
+                            (skip-event () nil)))
+                        :name "org-agent-async-task")
+        (restart-case
+            (handler-bind ((error (lambda (c)
+                                    (kernel-log "SYSTEM ERROR (inject-stimulus): ~a~%" c)
+                                    ;; Log the error and invoke the skip-event restart
+                                    (invoke-restart 'skip-event))))
+              (cognitive-loop raw-message))
+          (skip-event ()
+            (kernel-log "SYSTEM RECOVERY: Stimulus dropped to prevent kernel panic.~%"))))))
+
+(defun spawn-task (task-description &key (async-p t))
+  "A programmatic way for skills to delegate sub-tasks to the kernel.
+   If ASYNC-P is true, it spawns a new thread, enabling 'Swarm' orchestration."
+  (let ((msg `(:type :EVENT :payload (:sensor :delegation :query ,task-description :async-p ,async-p))))
+    (inject-stimulus msg)))
+
+(defun send-swarm-packet (target-url payload)
+  "Serializes a cognitive context and dispatches it to a remote org-agent.
+   Enables federated, cross-machine swarming."
+  (let* ((json-payload (cl-json:encode-json-to-string payload))
+         (headers '(("Content-Type" . "application/json"))))
+    (kernel-log "SWARM - Dispatching packet to ~a..." target-url)
+    (handler-case
+        (dex:post target-url :headers headers :content json-payload)
+      (error (c)
+        (kernel-log "SWARM ERROR - Failed to reach remote instance: ~a" c)
+        nil))))
+
 
 (defun dispatch-action (action)
   "Routes an approved action intent to the correct physical actuator."
@@ -118,6 +148,14 @@
                (setf (skill-priority skill) val)
                (kernel-log "ACTUATOR [System] - Set priority of ~a to ~a" name val))
              (kernel-log "ACTUATOR [System] ERROR - Skill ~a not found" name))))
+      (:auth-google-code
+       (let ((code (getf payload :code)))
+         (kernel-log "ACTUATOR [System] - Received Google OAuth code. Exchanging...")
+         ;; We call the function in the skill package. 
+         ;; Note: In a production kernel, we would use a more robust hook system.
+         (if (uiop:symbol-call :org-agent.skills.org-skill-auth-google-oauth :auth-google-receive-code code)
+             (kernel-log "ACTUATOR [System] - Google OAuth exchange successful.")
+             (kernel-log "ACTUATOR [System] - Google OAuth exchange FAILED."))))
       (t (kernel-log "ACTUATOR [System] - Unknown command ~a" cmd)))))
 
 ;;; ============================================================================
@@ -133,6 +171,9 @@
          (skill      (find-triggered-skill context))
          (skill-name (when skill (skill-name skill))))
     
+    ;; SOTA: Snapshot the memory state BEFORE thinking to enable rollback
+    (snapshot-object-store)
+    
     (let* ((proposed-action (think context))
            (approved-action (decide proposed-action context))
            (status (if (and proposed-action (null approved-action)) :rejected :success))
@@ -157,6 +198,9 @@
           (:buffer-update
            (let ((ast (getf payload :ast)))
              (when ast (ingest-ast ast))))
+          (:point-update
+           (let ((element (getf payload :element)))
+             (when element (ingest-ast element))))
           ;; Ensure we don't return NIL for these
           (:user-command t)
           (:heartbeat t)
@@ -208,15 +252,20 @@
   "Scans the directory defined by SKILLS_DIR (defaults to notes) and hot-loads all skills.
    This is where the daemon acquires its intelligence, now unified with the Atomic Notes (Zettelkasten)."
   (let* ((env-path (uiop:getenv "SKILLS_DIR"))
-         (skills-dir (if env-path 
+         (memex-dir (uiop:getenv "MEMEX_DIR"))
-                         (uiop:ensure-directory-pathname env-path)
+         (skills-dir (cond
-                         (merge-pathnames "notes/" (uiop:ensure-directory-pathname (uiop:getenv "MEMEX_DIR"))))))
+                       (env-path (uiop:ensure-directory-pathname env-path))
+                       (memex-dir (merge-pathnames "notes/" (uiop:ensure-directory-pathname memex-dir)))
+                       (t (merge-pathnames "notes/" (uiop:ensure-directory-pathname (uiop:native-namestring "~/memex/")))))))
     (if (uiop:directory-exists-p skills-dir)
         (progn
-          (kernel-log "KERNEL: Loading skills from consolidated Atomic Notes (Zettelkasten): ~a" skills-dir)
+          (kernel-log "KERNEL: Loading skills from consolidated Atomic Notes (Zettelkasten): ~a" (uiop:native-namestring skills-dir))
-          (dolist (file (uiop:directory-files skills-dir "skill-*.org"))
+          (let ((files (uiop:directory-files skills-dir "org-skill-*.org")))
-            (load-skill-from-org file)))
+            (if files
-        (kernel-log "KERNEL ERROR: Skills directory not found at ~a" skills-dir))))
+                (dolist (file files)
+                  (load-skill-from-org file))
+                (kernel-log "KERNEL: No skills found matching 'org-skill-*.org' in ~a" (uiop:native-namestring skills-dir)))))
+        (kernel-log "KERNEL ERROR: Skills directory not found at ~a" (uiop:native-namestring skills-dir)))))
 
 (defun start-daemon (&key (port 9105))
   "Boots the Neurosymbolic Kernel.

src/embedding.lisp

@@ -0,0 +1,52 @@
+(in-package :org-agent)
+
+;;; ============================================================================
+;;; Vector Embedding and Math
+;;; ============================================================================
+
+(defun get-embedding (text)
+  "Fetches the vector embedding for a given text string from Gemini's embedding-004 model."
+  (let* ((auth (get-provider-auth :gemini))
+         (api-key (getf auth :api-key))
+         (endpoint "https://generativelanguage.googleapis.com/v1beta/models/text-embedding-004:embedContent"))
+    
+    (unless api-key
+      (return-from get-embedding nil))
+    
+    (let* ((url (format nil "~a?key=~a" endpoint api-key))
+           (headers `(("Content-Type" . "application/json")))
+           (body (cl-json:encode-json-to-string
+                  `((model . "models/text-embedding-004")
+                    (content . ((parts . ((text . ,text)))))))))
+      (handler-case
+          (let* ((response (dex:post url :headers headers :content body))
+                 (json (cl-json:decode-json-from-string response)))
+            ;; Path: embedding.values
+            (cdr (assoc :values (cdr (assoc :embedding json)))))
+        (error (c)
+          (kernel-log "EMBEDDING FAILURE: ~a" c)
+          nil)))))
+
+(defun dot-product (v1 v2)
+  (reduce #'+ (mapcar #'* v1 v2)))
+
+(defun magnitude (v)
+  (sqrt (reduce #'+ (mapcar (lambda (x) (* x x)) v))))
+
+(defun cosine-similarity (v1 v2)
+  (let ((m1 (magnitude v1))
+        (m2 (magnitude v2)))
+    (if (or (zerop m1) (zerop m2))
+        0
+        (/ (dot-product v1 v2) (* m1 m2)))))
+
+(defun find-most-similar (query-vector top-k)
+  "Scans the entire *object-store* and returns the top-K objects by cosine similarity."
+  (let ((similarities nil))
+    (maphash (lambda (id obj)
+               (let ((vec (org-object-vector obj)))
+                 (when vec
+                   (push (cons (cosine-similarity query-vector vec) obj) similarities))))
+             *object-store*)
+    (let ((sorted (sort similarities #'> :key #'car)))
+      (subseq sorted 0 (min top-k (length sorted))))))

src/neuro.lisp

@@ -18,13 +18,21 @@
   "Helper: Fetches an environment variable with a fallback default."
   (or (uiop:getenv var) default))
 
-(defvar *llm-api-key* (get-env "LLM_API_KEY")
+;;; --- Pluggable Authentication Backends ---
-  "The API key for the neural engine (LLM Provider).")
 
-(defvar *llm-endpoint* (get-env "LLM_ENDPOINT" "https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent")
+(defvar *auth-providers* (make-hash-table :test 'equal)
-  "The default neural endpoint (currently defaulting to Gemini).")
+  "Registry of authentication provider skills. Key is provider keyword (e.g., :gemini).")
 
-;;; --- Pluggable Neuro Backends ---
+(defun register-auth-provider (name fn)
+  "Register a function that returns the required auth headers for a provider."
+  (setf (gethash name *auth-providers*) fn))
+
+(defun get-provider-auth (provider)
+  "Queries the registered auth skill for the necessary headers."
+  (let ((auth-fn (gethash provider *auth-providers*)))
+    (if auth-fn
+        (funcall auth-fn)
+        nil)))
 
 (defvar *neuro-backends* (make-hash-table :test 'equal)
   "Registry of neural provider backends.")
@@ -52,21 +60,27 @@
     "(:type :LOG :payload (:text \"Neural Cascade Failure - All providers exhausted.\"))")
 
 (defun execute-gemini-request (prompt system-prompt)
-  "The default System 1 backend (Gemini)."
+  "The default System 1 backend (Gemini). Authentication is now pluggable."
-  (unless *llm-api-key*
+  (let* ((auth (get-provider-auth :gemini))
-    (return-from execute-gemini-request "(:type :LOG :payload (:text \"Neural key missing, using mock System 1\"))"))
+         (api-key (getf auth :api-key))
-  
+         (bearer-token (getf auth :bearer-token))
-  (let* ((url (format nil "~a?key=~a" *llm-endpoint* *llm-api-key*))
+         (endpoint (or (getf auth :endpoint) 
-         (body (cl-json:encode-json-to-string
+                       "https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent")))
-                `((contents . ((parts . ((text . ,(format nil "~a~%~%Prompt: ~a" system-prompt prompt))))))))))
+    
-    (handler-case
+    (unless (or api-key bearer-token)
-        (let* ((response (dex:post url
+      (return-from execute-gemini-request "(:type :LOG :payload (:text \"Authentication missing for Gemini\"))"))
-                                   :headers '(("Content-Type" . "application/json"))
+    
-                                   :content body))
+    (let* ((url (if api-key (format nil "~a?key=~a" endpoint api-key) endpoint))
-               (json (cl-json:decode-json-from-string response)))
+           (headers `(("Content-Type" . "application/json")
-          (cdr (assoc :text (cdr (assoc :parts (car (cdr (assoc :parts (car (cdr (assoc :candidates json)))))))))))
+                      ,@(when bearer-token `(("Authorization" . ,(format nil "Bearer ~a" bearer-token))))))
-      (error (c)
+           (body (cl-json:encode-json-to-string
-        (format nil "(:type :LOG :payload (:text \"Neural Engine Failure: ~a\"))" c)))))
+                  `((contents . ((parts . ((text . ,(format nil "~a~%~%Prompt: ~a" system-prompt prompt))))))))))
+      (handler-case
+          (let* ((response (dex:post url :headers headers :content body))
+                 (json (cl-json:decode-json-from-string response)))
+            (cdr (assoc :text (cdr (assoc :parts (car (cdr (assoc :parts (car (cdr (assoc :candidates json)))))))))))
+        (error (c)
+          (format nil "(:type :LOG :payload (:text \"Neural Engine Failure: ~a\"))" c))))))
 
 ;; Initialize the default backend
 (register-neuro-backend :gemini #'execute-gemini-request)
@@ -95,3 +109,23 @@
                 '(:type :LOG :payload (:text "Skill triggered (Deterministic only)")))))
         ;; If no skills trigger, the agent remains silent.
         nil)))
+
+;;; ============================================================================
+;;; Prompt Distillation (Self-Evolution)
+;;; ============================================================================
+
+(defun distill-prompt (full-prompt successful-output)
+  "Neural distillation: Summarizes a complex prompt and its success into a denser format.
+   Used for 'Self-Evolving prompts' that reduce token usage over time."
+  (let ((system-instr "You are a Meta-Cognitive Prompt Architect. Your task is to DISTILL the following prompt and its successful result into a SHORTER, HIGH-SIGNAL template that would yield the same result."))
+    (ask-neuro (format nil "PROMPT: ~a~%RESULT: ~a~%~%Create a distilled version." full-prompt successful-output)
+               :system-prompt system-instr)))
+
+(defun distillation-loop ()
+  "Periodically reviews internal logs and distills prompts for active skills.
+   This is an autonomous self-improvement cycle."
+  (let ((logs (context-get-system-logs 50)))
+    (dolist (log logs)
+      (when (search "Verified by skill" log)
+        ;; Extract the skill name and attempt distillation
+        (kernel-log "NEURO - Triggering prompt distillation cycle...")))))

src/object-store.lisp

@@ -20,6 +20,7 @@
   type       ; The Org element type (e.g., :HEADLINE, :PARAGRAPH, :PLAIN-LIST)
   attributes ; A property list of metadata (e.g., :TITLE, :TAGS, :TODO-STATE)
   content    ; The raw text or non-element data within the node
+  vector     ; The semantic embedding vector (System 1 memory)
   parent-id  ; A pointer to the parent object's ID for tree traversal
   children   ; A list of IDs for all immediate child nodes
   version    ; A timestamp or counter used for cache invalidation
@@ -41,6 +42,11 @@
          (id (or (getf props :ID) 
                  (format nil "temp-~a" (get-universal-time))))
          (contents (getf ast :contents))
+         ;; Extract raw text for embedding if it's a headline
+         (raw-content (when (eq type :HEADLINE)
+                        (format nil "~a~%~a" 
+                                (getf props :TITLE)
+                                (or (cl:getf ast :raw-content) ""))))
          (child-ids nil))
     
     ;; Depth-first ingestion: Recurse into children first to gather their IDs.
@@ -54,6 +60,8 @@
                 :id id
                 :type type
                 :attributes props
+                :content raw-content
+                :vector (when raw-content (get-embedding raw-content))
                 :parent-id parent-id
                 :children (nreverse child-ids) ; Maintain document order
                 :version (get-universal-time)
@@ -61,6 +69,45 @@
       (setf (gethash id *object-store*) obj)
       id)))
 
+(defvar *object-store-snapshots* nil
+  "A history of previous *object-store* states for rollback/time-travel.")
+
+(defun copy-org-object (obj)
+  "Creates a shallow copy of an org-object struct.
+   Used during snapshotting."
+  (make-org-object 
+   :id (org-object-id obj)
+   :type (org-object-type obj)
+   :attributes (copy-list (org-object-attributes obj))
+   :content (org-object-content obj)
+   :vector (org-object-vector obj)
+   :parent-id (org-object-parent-id obj)
+   :children (copy-list (org-object-children obj))
+   :version (org-object-version obj)
+   :last-sync (org-object-last-sync obj)))
+
+(defun snapshot-object-store ()
+  "Creates a deep-copy of the current object store hash table.
+   Allows for 'Interactive Steering' and state rollback."
+  (let ((snapshot (make-hash-table :test 'equal)))
+    (maphash (lambda (id obj)
+               (setf (gethash id snapshot) (copy-org-object obj)))
+             *object-store*)
+    (push (list :timestamp (get-universal-time) :data snapshot) *object-store-snapshots*)
+    ;; Keep only the last 20 snapshots to prevent memory leaks
+    (when (> (length *object-store-snapshots*) 20)
+      (setf *object-store-snapshots* (subseq *object-store-snapshots* 0 20)))
+    (kernel-log "MEMORY - Object Store snapshot created.")))
+
+(defun rollback-object-store (&optional (index 0))
+  "Restores the Object Store to a previous state."
+  (let ((snapshot (nth index *object-store-snapshots*)))
+    (if snapshot
+        (progn
+          (setf *object-store* (getf snapshot :data))
+          (kernel-log "MEMORY - Object Store rolled back to snapshot ~a" index))
+        (kernel-log "MEMORY ERROR - Snapshot ~a not found." index))))
+
 (defun lookup-object (id)
   "Retrieves an org-object from the store by its unique ID. Returns NIL if not found."
   (gethash id *object-store*))

src/org-agent.el

@@ -236,6 +236,18 @@ will assume you have started it manually (e.g., via SBCL)."
                  :state :saved
                  :ast ,(org-agent--buffer-to-sexp))))))
 
+(defun org-agent-notify-point ()
+  "Sensor: Notify daemon of the element currently at point (Incremental Perception).
+This is much faster than parsing the entire buffer and allows for real-time
+responsiveness to the user's cursor position."
+  (when (and org-agent--network-process (derived-mode-p 'org-mode))
+    (let ((element (org-element-at-point)))
+      (org-agent-send
+       `(:type :EVENT
+         :payload (:sensor :point-update
+                   :file ,(buffer-file-name)
+                   :element ,(org-agent--clean-element element)))))))
+
 ;;; Interaction Commands
 
 (defun org-agent-set-model-cascade (cascade-string)
@@ -264,7 +276,6 @@ e.g., ':gemini,:openai,:ollama'."
         (insert "#+TITLE: org-agent Chat\n#+STARTUP: showall\n\n* Welcome to the Neurosymbolic Lisp Machine\n\nType your message below and press `C-c C-c` to send.\n\n")))
     (switch-to-buffer buf)
     (goto-char (point-max))))
-
 (defun org-agent-chat-send ()
   "Send the current chat buffer content to the agent."
   (interactive)
@@ -280,7 +291,20 @@ e.g., ':gemini,:openai,:ollama'."
       (insert "\n\n** Thinking...\n"))
     (message "org-agent: Message sent.")))
 
+(defun org-agent-auth-google (code)
+  "Submit the Google OAuth authorization CODE to the daemon."
+  (interactive "sEnter Google Authorization Code: ")
+  (unless org-agent--network-process
+    (org-agent-connect))
+  (org-agent-send 
+   `(:type :REQUEST 
+     :id ,(truncate (float-time))
+     :target :system
+     :payload (:action :auth-google-code :code ,code)))
+  (message "org-agent: Authorization code sent to daemon."))
+
 (defun org-agent-organize-subtree ()
+...
   "Command: Ask the agent to organize the current Org subtree."
   (interactive)
   (org-agent-run-command :organize-subtree))
@@ -314,9 +338,11 @@ Org-mode sensing."
   (if org-agent-mode
       (progn
         (add-hook 'after-save-hook #'org-agent-notify-save)
+        (add-hook 'post-command-hook #'org-agent-notify-point)
         (add-hook 'kill-emacs-hook #'org-agent-disconnect)
         (org-agent-connect))
     (remove-hook 'after-save-hook #'org-agent-notify-save)
+    (remove-hook 'post-command-hook #'org-agent-notify-point)
     (remove-hook 'kill-emacs-hook #'org-agent-disconnect)
     (org-agent-disconnect)))
 

src/package.lisp

@@ -21,6 +21,10 @@
    #:org-object-type
    #:org-object-attributes
    #:org-object-children
+   #:org-object-vector
+   #:snapshot-object-store
+   #:rollback-object-store
+   #:send-swarm-packet
    
    ;; --- Context API (Peripheral Vision) ---
    #:context-query-store
@@ -59,6 +63,11 @@
    ;; --- Neuro (System 1) ---
    #:ask-neuro
    #:register-neuro-backend
+   #:register-auth-provider
+   #:distill-prompt
+   #:get-embedding
+   #:cosine-similarity
+   #:find-most-similar
    
    ;; --- AST Helpers ---
    #:find-headline-missing-id))

src/symbolic.lisp

@@ -4,15 +4,14 @@
 ;;; System 2: The Symbolic Gatekeeper
 ;;; ============================================================================
 ;;; This module implements the 'Executive Function' of the kernel. 
-;;; System 2 is responsible for 'Deterministic Reasoning'—applying strict rules,
+...
-;;; safety constraints, and logical checks to verify neural proposals. 
 ;;; It is slow but reliable, and it has the absolute power to overrule System 1.
 
 (defun decide (proposed-action context)
   "The System 2 Deciding Stage.
    
    It subjects the proposal from System 1 to a battery of symbolic tests.
-   1. It applies Global Safety Heuristics (e.g., preventing shell execution).
+   1. It applies Global Safety Heuristics (via the Safety Harness).
    2. It delegates domain-specific validation to the active skill's verify-fn.
    
    Returns an approved action intent, or a safe fallback (like a log message)."
@@ -20,17 +19,18 @@
     (if active-skill
         (let ((symbolic-gate (skill-symbolic-fn active-skill)))
           
-          ;; --- GLOBAL SAFETY HEURISTIC #1: Block Shell Execution ---
+          ;; --- GLOBAL SAFETY HEURISTIC #1: Safety Harness (AST Sandbox) ---
-          ;; We never allow the LLM to execute raw shell commands via Emacs eval.
           (when (and proposed-action (listp proposed-action)
                      (eq (getf proposed-action :type) :REQUEST)
                      (eq (getf (getf proposed-action :payload) :action) :eval))
             (let ((code (getf (getf proposed-action :payload) :code)))
-              (when (and code (search "shell-command" code))
+              ;; We call the global safety-harness skill logic
-                (kernel-log "SYSTEM 2 [GLOBAL]: Security violation blocked (shell-command attempt).~%")
+              (unless (uiop:symbol-call :org-agent.skills.org-skill-safety-harness :safety-harness-validate code)
-                (return-from decide '(:type :LOG :payload (:text "Blocked by Global Safety Heuristic"))))))
+                (kernel-log "SYSTEM 2 [GLOBAL]: Security violation blocked by Safety Harness.~%")
+                (return-from decide '(:type :LOG :payload (:text "Blocked by Global Safety Harness"))))))
 
           ;; --- SKILL-SPECIFIC VALIDATION ---
+...
           ;; If the skill provides a specific System 2 verification function, run it.
           (if symbolic-gate
               (let ((decision (funcall symbolic-gate proposed-action context)))