REFACTOR: Explanatory Core Architecture & Terminology Alignment

This commit is contained in:
2026-04-13 09:03:42 -04:00
parent 10a500c480
commit 3e68cc11af
52 changed files with 446 additions and 1756 deletions

View File

@@ -11,7 +11,7 @@ A common failure mode for Large Language Models (LLMs) is the "Lost in the Middl
The ~org-agent~ harness implements a deterministic, tree-aware solution: the **Foveal-Peripheral Hybrid Model**.
*** 1. The Foveal Focus (High Resolution)
When the harness prepares a prompt for the Associative Engine, it identifies a "Foveal Focus"—typically the specific Org headline or task the user is currently interacting with. This node, along with its immediate children and semantically relevant neighbors, is rendered at "High Resolution," meaning its full body text, properties, and metadata are included in the prompt.
When the harness prepares a prompt for the Probabilistic Engine, it identifies a "Foveal Focus"—typically the specific Org headline or task the user is currently interacting with. This node, along with its immediate children and semantically relevant neighbors, is rendered at "High Resolution," meaning its full body text, properties, and metadata are included in the prompt.
*** 2. The Peripheral Vision (Low Resolution)
To maintain global awareness without bloating the context window, the rest of the Memex is rendered at "Low Resolution." The harness recursively walks the Object Store and generates a skeletal outline consisting only of titles and IDs. This gives the LLM a "mental map" of the entire system, allowing it to reference other projects or skills without needing to see their full content until they are explicitly brought into focus.
@@ -28,7 +28,7 @@ flowchart TD
Identification --> Peripheral[Render Outline: Titles Only]
Foveal --> Assembly[Assemble Global Awareness String]
Peripheral --> Assembly
Assembly --> LLM[Associative Engine Proposal]
Assembly --> LLM[Probabilistic Engine Proposal]
#+end_src
* Context Assembly (context.lisp)
@@ -105,7 +105,7 @@ Reads the raw literate Org source of a specific skill. This is a foundational ca
#+end_src
** Harness Logs (context-get-system-logs)
Retrieves the most recent entries from the harness's internal circular log buffer. This allows the Associative Engine to see recent errors or successful dispatches, enabling it to course-correct or explain failures to the user.
Retrieves the most recent entries from the harness's internal circular log buffer. This allows the Probabilistic Engine to see recent errors or successful dispatches, enabling it to course-correct or explain failures to the user.
#+begin_src lisp :tangle ../src/context.lisp
(defun context-get-system-logs (&optional (limit 20))

View File

@@ -10,7 +10,7 @@ The core of the ~org-agent~ harness is a functional transformation pipeline. In
We have evolved the harness into a **Reactive Signal Pipeline**. Every event—whether it is a user keystroke, a heartbeat timer pulse, or a suggested action from an LLM—is treated as a discrete **Signal**.
Signals move through a series of formal **Gates**. Each gate transforms or validates the signal until it is either physically dispatched to an actuator or safely rejected by the Deliberate Engine.
Signals move through a series of formal **Gates**. Each gate transforms or validates the signal until it is either physically dispatched to an actuator or safely rejected by the Deterministic Engine.
*** Advantages of the Pipeline Model:
- **Consensus Ready:** By treating reasoning as a signal moving through a pipe, we can "split" the pipe to query multiple LLM backends simultaneously. A Consensus Gate later in the pipe compares these proposals.
@@ -22,7 +22,7 @@ Signals move through a series of formal **Gates**. Each gate transforms or valid
flowchart TD
S1[Signal: External Stimulus] --> P[Perceive Gate]
S2[Signal: Heartbeat Pulse] --> P
P --> N[Associative Gate]
P --> N[Probabilistic Gate]
N --> C[Consensus Gate]
C --> V[Validation Gate]
V --> D[Dispatch Gate]
@@ -138,15 +138,15 @@ The Perceive Gate is responsible for data normalization and sensory intake. It t
signal))
#+end_src
*** Associative Gate
The Associative Gate invokes the neural reasoning engine. It takes the current context and generates a list of "intuitions" or proposed actions.
*** Probabilistic Gate
The Probabilistic Gate invokes the neural reasoning engine. It takes the current context and generates a list of "intuitions" or proposed actions.
#+begin_src lisp :tangle ../src/loop.lisp
(defun neuro-gate (signal)
"Associative: Neural intuition and proposed actions."
"Probabilistic: Neural intuition and proposed actions."
(unless (eq (getf signal :type) :EVENT)
(return-from neuro-gate signal))
(harness-log "GATE [Associative]: Consulting LLM...")
(harness-log "GATE [Probabilistic]: Consulting LLM...")
(let ((thoughts (think signal)))
(setf (getf signal :proposals) (if (and (listp thoughts) (listp (car thoughts)))
thoughts
@@ -182,11 +182,11 @@ When multiple LLM backends provide diverging thoughts, the Consensus Gate resolv
#+end_src
*** Decide Gate
The Decide Gate is the final deterministic safety net. It runs the candidate action through all loaded skill safety gates (The Deliberate Engine) before allowing it to proceed.
The Decide Gate is the final deterministic safety net. It runs the candidate action through all loaded skill safety gates (The Deterministic Engine) before allowing it to proceed.
#+begin_src lisp :tangle ../src/loop.lisp
(defun decide-gate (signal)
"Deliberate: Deterministic safety and validation."
"Deterministic: Deterministic safety and validation."
(let ((candidate (getf signal :candidate)))
(if candidate
(let* ((normalized-candidate (if (listp candidate) candidate (list :type :RESPONSE :payload (list :text candidate))))
@@ -440,7 +440,7 @@ The Reactive Signal Pipeline must be empirically verified through automated test
:priority 200
:trigger (lambda (ctx) t)
:neuro (lambda (ctx) (list :type :REQUEST :payload (list :action :eval :code "(error \"BOOM\")")))
:symbolic (lambda (action ctx) (error "CRASH IN SYSTEM 2")))
:symbolic (lambda (action ctx) (error "CRASH IN DETERMINISTIC ENGINE")))
(process-signal (list :type :EVENT :payload (list :sensor :test)))
;; Verify that we are still in State A
(let ((obj (lookup-object "node-1")))

View File

@@ -6,21 +6,21 @@
* The Neurosymbolic Bridge (neuro.lisp & symbolic.lisp)
*** The Neurosymbolic Loop
In our loop, the Associative Engine never speaks to the world directly. It only proposes "thoughts" to the Deliberate Engine. the Deliberate Engine, the Lisp harness, evaluates these thoughts against a chain of symbolic safety gates (Skills) before any action is actually dispatched to an actuator (Emacs, Shell, etc.).
In our loop, the Probabilistic Engine never speaks to the world directly. It only proposes "thoughts" to the Deterministic Engine. the Deterministic Engine, the Lisp harness, evaluates these thoughts against a chain of symbolic safety gates (Skills) before any action is actually dispatched to an actuator (Emacs, Shell, etc.).
#+begin_src mermaid
flowchart TD
Stimulus[External Stimulus/Signal] --> Perceive[Perceive: Skill Trigger]
Perceive --> Associative[Associative Engine: LLM]
Associative --> Proposal[Lisp Action Proposal]
Proposal --> Deliberate[Deliberate Engine: Lisp Gates]
Deliberate --> Gate1[Safety Gate: Skill A]
Perceive --> Probabilistic[Probabilistic Engine: LLM]
Probabilistic --> Proposal[Lisp Action Proposal]
Proposal --> Deterministic[Deterministic Engine: Lisp Gates]
Deterministic --> Gate1[Safety Gate: Skill A]
Gate1 --> Gate2[Safety Gate: Skill B]
Gate2 --> Verified[Verified Action]
Verified --> Dispatch[Dispatch: Actuator]
style Associative fill:#f9f,stroke:#333,stroke-width:2px
style Deliberate fill:#bbf,stroke:#333,stroke-width:2px
style Probabilistic fill:#f9f,stroke:#333,stroke-width:2px
style Deterministic fill:#bbf,stroke:#333,stroke-width:2px
#+end_src
*** Sovereign Decoupling (The Thin Harness)
@@ -29,15 +29,15 @@ The harness files ~neuro.lisp~ and ~symbolic.lisp~ are intentionally "Thin Harne
By moving the "Fat" logic (vendor APIs, security rules) into **Skills**, we achieve total sovereign decoupling. You can swap your LLM provider or your security policy without ever touching the harness.
* Associative Engine (neuro.lisp)
The Associative engine handles the interface with LLM providers, providing a unified associative space regardless of the underlying model.
* Probabilistic Engine (neuro.lisp)
The Probabilistic engine handles the interface with LLM providers, providing a unified probabilistic space regardless of the underlying model.
** Package Context
#+begin_src lisp :tangle ../src/neuro.lisp
(in-package :org-agent)
#+end_src
** Associative Backends Registry
** Probabilistic Backends Registry
The harness maintains a neutral registry of backends. Skills (like the LLM Gateway) register themselves here to provide actual neural reasoning capabilities.
#+begin_src lisp :tangle ../src/neuro.lisp
@@ -51,7 +51,7 @@ The ordered list of backends to attempt for neural reasoning. This list is ~nil~
(defvar *provider-cascade* nil)
#+end_src
** Register Associative Backend
** Register Probabilistic Backend
A simple mapping from a keyword identifier to a backend implementation function.
#+begin_src lisp :tangle ../src/neuro.lisp
@@ -65,8 +65,8 @@ A hook for dynamic model selection. A skill might look at the current context an
(defvar *model-selector-fn* nil "A function called with (provider context) to return a model ID.")
#+end_src
** Associative Dispatch (ask-neuro)
This is the primary entrance to the Associative engine. It implements two modes of operation:
** Probabilistic Dispatch (ask-neuro)
This is the primary entrance to the Probabilistic engine. It implements two modes of operation:
1. **Sequential Cascade:** Attempt backends one by one until success.
2. **Parallel Consensus:** Query multiple backends simultaneously to resolve hallucinations or select the best "thought."
@@ -85,7 +85,7 @@ sequenceDiagram
#+begin_src lisp :tangle ../src/neuro.lisp
(defvar *consensus-enabled-p* nil "If T, ask-neuro queries all backends in parallel.")
(defun ask-neuro (prompt &key (system-prompt "You are the Associative engine of a Neurosymbolic Lisp Machine.") (cascade nil) (context nil))
(defun ask-neuro (prompt &key (system-prompt "You are the Probabilistic engine of a Neurosymbolic Lisp Machine.") (cascade nil) (context nil))
"Dispatches a neural request through the provider cascade or parallel consensus."
(let ((backends (cond
((and cascade (listp cascade)) cascade)
@@ -101,7 +101,7 @@ sequenceDiagram
(when backend-fn
(push (bt:make-thread
(lambda ()
(harness-log "ASSOCIATIVE [Consensus]: Querying backend ~a..." backend)
(harness-log "PROBABILISTIC [Consensus]: Querying backend ~a..." backend)
(let* ((model (when *model-selector-fn* (funcall *model-selector-fn* backend context)))
(result (ignore-errors
(if model
@@ -125,7 +125,7 @@ sequenceDiagram
(or (dolist (backend backends)
(let ((backend-fn (gethash backend *neuro-backends*)))
(when backend-fn
(harness-log "ASSOCIATIVE: Attempting backend ~a..." backend)
(harness-log "PROBABILISTIC: Attempting backend ~a..." backend)
(let* ((model (when *model-selector-fn* (funcall *model-selector-fn* backend context)))
(result (if model
(funcall backend-fn prompt system-prompt :model model)
@@ -136,20 +136,20 @@ sequenceDiagram
"(:type :LOG :payload (:text \"Neural Cascade Failure\"))"))))
#+end_src
** Associative Reasoning (think)
** Probabilistic Reasoning (think)
The ~think~ function is where the "Neuro" meets the "Symbolic." It gathers the global awareness context (Peripheral Vision), the tool definitions (The Tool Belt), and any skill-specific triggers to form the final prompt.
Crucially, it mandates that the output be a Common Lisp property list, forcing the LLM to "think in Lisp."
#+begin_src lisp :tangle ../src/neuro.lisp
(defun think (context)
"Invokes the neural Associative engine to propose a Lisp action based on context."
"Invokes the neural Probabilistic engine to propose a Lisp action based on context."
(let ((active-skill (find-triggered-skill context))
(tool-belt (generate-tool-belt-prompt))
(global-context (context-assemble-global-awareness)))
(if active-skill
(progn
(harness-log "ASSOCIATIVE: Engaging skill '~a'~%" (skill-name active-skill))
(harness-log "PROBABILISTIC: Engaging skill '~a'~%" (skill-name active-skill))
(let* ((prompt-generator (skill-neuro-prompt active-skill))
(raw-prompt (when prompt-generator (funcall prompt-generator context)))
(full-system-prompt (concatenate 'string
@@ -177,7 +177,7 @@ To call a tool, you MUST use:
(raw-thoughts (cl-ppcre:split (cl-ppcre:quote-meta-chars "|CONSENSUS-SEP|") thought))
(suggestions nil))
(dolist (raw-thought raw-thoughts)
(harness-log "ASSOCIATIVE RAW: ~a~%" raw-thought)
(harness-log "PROBABILISTIC RAW: ~a~%" raw-thought)
(let* ((cleaned-thought
(let ((match (cl-ppcre:scan-to-strings "(?s)```(?:lisp)?\\n?(.*?)\\n?```" raw-thought)))
(if match
@@ -191,7 +191,7 @@ To call a tool, you MUST use:
(list :sensor :syntax-error
:code cleaned-thought
:error (format nil "~a" c)))))))
(harness-log "ASSOCIATIVE Suggestion: ~a~%" cleaned-thought)
(harness-log "PROBABILISTIC Suggestion: ~a~%" cleaned-thought)
(when (and suggestion (listp suggestion))
(push suggestion suggestions))))
(if (and *consensus-enabled-p* suggestions)
@@ -202,7 +202,7 @@ To call a tool, you MUST use:
#+end_src
** Prompt Meta-Cognition (distill-prompt)
Even the Associative engine can benefit from introspection. This function allows the agent to observe its own prompts and successful results to distill them into reusable templates.
Even the Probabilistic engine can benefit from introspection. This function allows the agent to observe its own prompts and successful results to distill them into reusable templates.
#+begin_src lisp :tangle ../src/neuro.lisp
(defun distill-prompt (full-prompt successful-output)
@@ -211,10 +211,10 @@ Even the Associative engine can benefit from introspection. This function allows
#+end_src
* Deliberate Engine (symbolic.lisp)
The Deliberate engine is the deterministic gatekeeper that ensures all proposed actions—whether from the user or from the neural engine—are safe and logically valid.
* Deterministic Engine (symbolic.lisp)
The Deterministic engine is the deterministic gatekeeper that ensures all proposed actions—whether from the user or from the neural engine—are safe and logically valid.
As a "Thin Harness," the Deliberate engine does not contain specific security rules or task integrity checks. Instead, it provides a priority-based dispatcher that iterates through all loaded skills to validate or transform proposed actions.
As a "Thin Harness," the Deterministic engine does not contain specific security rules or task integrity checks. Instead, it provides a priority-based dispatcher that iterates through all loaded skills to validate or transform proposed actions.
** Package Context
#+begin_src lisp :tangle ../src/symbolic.lisp
@@ -241,7 +241,7 @@ flowchart LR
#+begin_src lisp :tangle ../src/symbolic.lisp
(defun decide (proposed-action context)
"The Deliberate Safety Gate: iterates through all skill symbolic-gates sorted by priority."
"The Deterministic Safety Gate: iterates through all skill symbolic-gates sorted by priority."
(let ((current-action proposed-action)
(skills nil))
;; 1. Collect all skills with symbolic gates
@@ -261,7 +261,7 @@ flowchart LR
;; If any gate returns a LOG or EVENT (blocking/intercepting), stop and return it.
(when (and (listp current-action)
(member (getf current-action :type) '(:LOG :EVENT :log :event)))
(harness-log "DELIBERATE: Intercepted by skill '~a'~%" (skill-name skill))
(harness-log "DETERMINISTIC: Intercepted by skill '~a'~%" (skill-name skill))
(return-from decide current-action))))
current-action))

View File

@@ -7,7 +7,7 @@
The ~package.lisp~ file defines the public API of the ~org-agent~ harness. It serves as the primary membrane between the deterministic core modules and the dynamic world of skills and actuators.
** Architectural Intent: The Package Membrane
By strictly defining the public interface, we ensure that skills remain decoupled from the harness implementation details. This allows for sovereign replacement of any component (e.g., swapping the Object Store or the Associative Engine) without breaking existing skills.
By strictly defining the public interface, we ensure that skills remain decoupled from the harness implementation details. This allows for sovereign replacement of any component (e.g., swapping the Object Store or the Probabilistic Engine) without breaking existing skills.
#+begin_src mermaid
flowchart TD
@@ -82,7 +82,7 @@ flowchart TD
#:load-skill-with-timeout
#:topological-sort-skills
#:validate-lisp-syntax
#:safety-harness-validate
#:lisp-validator-validate
#:defskill
#:*skills-registry*
#:skill
@@ -109,7 +109,7 @@ flowchart TD
#:register-emacs-client
#:unregister-emacs-client
;; --- Associative Engine ---
;; --- Probabilistic Engine ---
#:ask-neuro
#:register-neuro-backend
#:distill-prompt

View File

@@ -19,7 +19,7 @@ Skills often depend on one another. The harness implements a deterministic topol
** Skill Architecture
#+begin_src mermaid
flowchart TD
Registry[Skills Registry] --> S1[Skill: System Invariants]
Registry[Skills Registry] --> S1[Skill: System Policy]
Registry --> S2[Skill: LLM Gateway]
Registry --> S3[Skill: Token Accountant]
S2 -- Depends On --> S1
@@ -65,7 +65,7 @@ The harness maintains a stateful tracking table for all skill files discovered i
#+end_src
** Skill Selection (find-triggered-skill)
The primary dispatcher for the Associative Engine. It iterates through the registry to find the highest-priority skill whose trigger function matches the current cognitive context.
The primary dispatcher for the Probabilistic Engine. It iterates through the registry to find the highest-priority skill whose trigger function matches the current cognitive context.
#+begin_src lisp :tangle ../src/skills.lisp
(defun find-triggered-skill (context)
@@ -303,7 +303,7 @@ The unified orchestrator for the system boot sequence.
(return-from initialize-all-skills nil))
(let ((sorted-files (topological-sort-skills skills-dir)))
;; MANDATE: The System Invariants must be present for a safe boot
;; MANDATE: The System Policy must be present for a safe boot
(unless (member "org-skill-system-invariants" sorted-files :key #'pathname-name :test #'string-equal)
(error "BOOT FAILURE: org-skill-system-invariants.org not found in skills directory."))
@@ -376,9 +376,9 @@ The harness provides a baseline set of cognitive tools that enable core system i
:guard (lambda (args context)
(declare (ignore context))
(let ((code (getf args :code)))
(let ((harness-pkg (find-package :org-agent.skills.org-skill-safety-harness)))
(let ((harness-pkg (find-package :org-agent.skills.org-skill-lisp-validator)))
(if harness-pkg
(uiop:symbol-call :org-agent.skills.org-skill-safety-harness :safety-harness-validate code)
(uiop:symbol-call :org-agent.skills.org-skill-lisp-validator :lisp-validator-validate code)
t))))
:body (lambda (args)
(let ((code (getf args :code)))

View File

@@ -53,7 +53,7 @@ This system defines the core "Thin Harness." It includes the protocol, the objec
(:file "src/credentials-vault")
(:file "src/llm-gateway")
(:file "src/symbolic")
(:file "src/safety-harness")
(:file "src/lisp-validator")
(:file "src/self-fix")
(:file "src/lisp-repair")
(:file "src/bouncer")
@@ -77,7 +77,7 @@ This system contains the empirical tests required by the Engineering Standards.
:components ((:file "tests/protocol-tests")
(:file "tests/pipeline-tests")
(:file "tests/peripheral-vision-tests")
(:file "tests/safety-harness-tests")
(:file "tests/lisp-validator-tests")
(:file "tests/boot-sequence-tests")
(:file "tests/object-store-tests")
(:file "tests/immune-system-tests")