284 lines
14 KiB
Org Mode
284 lines
14 KiB
Org Mode
#+TITLE: The Neurosymbolic Bridge (neuro.lisp & symbolic.lisp)
|
|
#+AUTHOR: Amr
|
|
#+FILETAGS: :kernel:neuro:symbolic:
|
|
#+STARTUP: content
|
|
|
|
* The Neurosymbolic Bridge (neuro.lisp & symbolic.lisp)
|
|
*** The Neurosymbolic Loop
|
|
|
|
In our loop, the Associative Engine never speaks to the world directly. It only proposes "thoughts" to the Deliberate Engine. the Deliberate Engine, the Lisp harness, evaluates these thoughts against a chain of symbolic safety gates (Skills) before any action is actually dispatched to an actuator (Emacs, Shell, etc.).
|
|
|
|
#+begin_src mermaid
|
|
flowchart TD
|
|
Stimulus[External Stimulus/Signal] --> Perceive[Perceive: Skill Trigger]
|
|
Perceive --> Associative[Associative Engine: LLM]
|
|
Associative --> Proposal[Lisp Action Proposal]
|
|
Proposal --> Deliberate[Deliberate Engine: Lisp Gates]
|
|
Deliberate --> Gate1[Safety Gate: Skill A]
|
|
Gate1 --> Gate2[Safety Gate: Skill B]
|
|
Gate2 --> Verified[Verified Action]
|
|
Verified --> Dispatch[Dispatch: Actuator]
|
|
|
|
style Associative fill:#f9f,stroke:#333,stroke-width:2px
|
|
style Deliberate fill:#bbf,stroke:#333,stroke-width:2px
|
|
#+end_src
|
|
|
|
*** Sovereign Decoupling (The Thin Harness)
|
|
|
|
The harness files ~neuro.lisp~ and ~symbolic.lisp~ are intentionally "Thin Harnesses." They do not know how to talk to Google Gemini or Anthropic Claude; they do not know what a "Bouncer" or a "Token Accountant" is. Instead, they provide the **protocol** for these behaviors.
|
|
|
|
By moving the "Fat" logic (vendor APIs, security rules) into **Skills**, we achieve total sovereign decoupling. You can swap your LLM provider or your security policy without ever touching the harness.
|
|
|
|
* Associative Engine (neuro.lisp)
|
|
The Associative engine handles the interface with LLM providers, providing a unified associative space regardless of the underlying model.
|
|
|
|
** Package Context
|
|
#+begin_src lisp :tangle ../src/neuro.lisp
|
|
(in-package :org-agent)
|
|
#+end_src
|
|
|
|
** Associative Backends Registry
|
|
The harness maintains a neutral registry of backends. Skills (like the LLM Gateway) register themselves here to provide actual neural reasoning capabilities.
|
|
|
|
#+begin_src lisp :tangle ../src/neuro.lisp
|
|
(defvar *neuro-backends* (make-hash-table :test 'equal))
|
|
#+end_src
|
|
|
|
** Provider Cascade
|
|
Intelligence is often a matter of availability. The cascade defines the order in which backends are attempted. Skills can dynamically override this list to optimize for cost, speed, or privacy.
|
|
|
|
#+begin_src lisp :tangle ../src/neuro.lisp
|
|
(defvar *provider-cascade* '(:openrouter :gemini-api))
|
|
#+end_src
|
|
|
|
** Register Associative Backend
|
|
A simple mapping from a keyword identifier to a backend implementation function.
|
|
|
|
#+begin_src lisp :tangle ../src/neuro.lisp
|
|
(defun register-neuro-backend (name fn) (setf (gethash name *neuro-backends*) fn))
|
|
#+end_src
|
|
|
|
** Model Selector Function
|
|
A hook for dynamic model selection. A skill might look at the current context and decide to use a "cheap" model for summaries and an "expensive" model for coding tasks.
|
|
|
|
#+begin_src lisp :tangle ../src/neuro.lisp
|
|
(defvar *model-selector-fn* nil "A function called with (provider context) to return a model ID.")
|
|
#+end_src
|
|
|
|
** Associative Dispatch (ask-neuro)
|
|
This is the primary entrance to the Associative engine. It implements two modes of operation:
|
|
1. **Sequential Cascade:** Attempt backends one by one until success.
|
|
2. **Parallel Consensus:** Query multiple backends simultaneously to resolve hallucinations or select the best "thought."
|
|
|
|
#+begin_src mermaid
|
|
sequenceDiagram
|
|
participant Harness
|
|
participant ProviderA as LLM 1
|
|
participant ProviderB as LLM 2
|
|
Harness->>ProviderA: Parallel Query
|
|
Harness->>ProviderB: Parallel Query
|
|
ProviderA-->>Harness: Suggestion A
|
|
ProviderB-->>Harness: Suggestion B
|
|
Harness->>Harness: Resolve Consensus
|
|
#+end_src
|
|
|
|
#+begin_src lisp :tangle ../src/neuro.lisp
|
|
(defvar *consensus-enabled-p* nil "If T, ask-neuro queries all backends in parallel.")
|
|
|
|
(defun ask-neuro (prompt &key (system-prompt "You are the Associative engine of a Neurosymbolic Lisp Machine.") (cascade nil) (context nil))
|
|
"Dispatches a neural request through the provider cascade or parallel consensus."
|
|
(let ((backends (cond
|
|
((and cascade (listp cascade)) cascade)
|
|
((functionp cascade) (funcall cascade context))
|
|
(t *provider-cascade*))))
|
|
(if *consensus-enabled-p*
|
|
;; PARALLEL CONSENSUS MODE
|
|
(let ((results nil)
|
|
(threads nil)
|
|
(lock (bt:make-lock)))
|
|
(dolist (backend backends)
|
|
(let ((backend-fn (gethash backend *neuro-backends*)))
|
|
(when backend-fn
|
|
(push (bt:make-thread
|
|
(lambda ()
|
|
(kernel-log "ASSOCIATIVE [Consensus]: Querying backend ~a..." backend)
|
|
(let* ((model (when *model-selector-fn* (funcall *model-selector-fn* backend context)))
|
|
(result (ignore-errors
|
|
(if model
|
|
(funcall backend-fn prompt system-prompt :model model)
|
|
(funcall backend-fn prompt system-prompt)))))
|
|
(bt:with-lock-held (lock)
|
|
(push result results)))))
|
|
threads))))
|
|
;; Wait for all threads with a timeout (e.g., 30s)
|
|
(let ((start-time (get-universal-time)))
|
|
(loop while (and (< (length results) (length threads))
|
|
(< (- (get-universal-time) start-time) 30))
|
|
do (sleep 0.1)))
|
|
;; Return the list of raw results (filtering out nils or errors)
|
|
(let ((valid-results (remove-if-not #'stringp results)))
|
|
(if valid-results
|
|
(format nil "~{~a~^|CONSENSUS-SEP|~}" valid-results)
|
|
"(:type :LOG :payload (:text \"Neural Consensus Failure\"))")))
|
|
|
|
;; SEQUENTIAL CASCADE MODE
|
|
(or (dolist (backend backends)
|
|
(let ((backend-fn (gethash backend *neuro-backends*)))
|
|
(when backend-fn
|
|
(kernel-log "ASSOCIATIVE: Attempting backend ~a..." backend)
|
|
(let* ((model (when *model-selector-fn* (funcall *model-selector-fn* backend context)))
|
|
(result (if model
|
|
(funcall backend-fn prompt system-prompt :model model)
|
|
(funcall backend-fn prompt system-prompt))))
|
|
(unless (or (null result)
|
|
(and (stringp result) (search ":LOG" result) (or (search "Failure" result) (search "missing" result))))
|
|
(return result))))))
|
|
"(:type :LOG :payload (:text \"Neural Cascade Failure\"))"))))
|
|
#+end_src
|
|
|
|
** Associative Reasoning (think)
|
|
The ~think~ function is where the "Neuro" meets the "Symbolic." It gathers the global awareness context (Peripheral Vision), the tool definitions (The Tool Belt), and any skill-specific triggers to form the final prompt.
|
|
|
|
Crucially, it mandates that the output be a Common Lisp property list, forcing the LLM to "think in Lisp."
|
|
|
|
#+begin_src lisp :tangle ../src/neuro.lisp
|
|
(defun think (context)
|
|
"Invokes the neural Associative engine to propose a Lisp action based on context."
|
|
(let ((active-skill (find-triggered-skill context))
|
|
(tool-belt (generate-tool-belt-prompt))
|
|
(global-context (context-assemble-global-awareness)))
|
|
(if active-skill
|
|
(progn
|
|
(kernel-log "ASSOCIATIVE: Engaging skill '~a'~%" (skill-name active-skill))
|
|
(let* ((prompt-generator (skill-neuro-prompt active-skill))
|
|
(raw-prompt (when prompt-generator (funcall prompt-generator context)))
|
|
(full-system-prompt (concatenate 'string
|
|
"ACTUATOR IDENTITY: You are the pure Lisp actuator for the org-agent kernel.
|
|
MANDATE: Output EXACTLY ONE Common Lisp property list starting with (:type :REQUEST).
|
|
ZERO CONVERSATION: Do not explain. Do not say 'Okay'. Do not use markdown blocks.
|
|
STRICT RULE: Do not output multiple lists. Do not chain multiple requests.
|
|
DO NOT embed tool calls inside text strings.
|
|
|
|
"
|
|
global-context
|
|
"
|
|
"
|
|
tool-belt
|
|
"
|
|
IMPORTANT: To reply to the user, you MUST use:
|
|
(:type :REQUEST :target :emacs :action :insert-at-end :buffer \"*org-agent-chat*\" :text \"* <Response Text>\")
|
|
|
|
To call a tool, you MUST use:
|
|
(:type :REQUEST :target :tool :action :call :tool \"<name>\" :args (:arg1 \"val\"))
|
|
|
|
")))
|
|
(if (and raw-prompt (> (length raw-prompt) 1))
|
|
(let* ((thought (ask-neuro raw-prompt :system-prompt full-system-prompt :context context))
|
|
(raw-thoughts (cl-ppcre:split (cl-ppcre:quote-meta-chars "|CONSENSUS-SEP|") thought))
|
|
(suggestions nil))
|
|
(dolist (raw-thought raw-thoughts)
|
|
(kernel-log "ASSOCIATIVE RAW: ~a~%" raw-thought)
|
|
(let* ((cleaned-thought
|
|
(let ((match (cl-ppcre:scan-to-strings "(?s)```(?:lisp)?\\n?(.*?)\\n?```" raw-thought)))
|
|
(if match
|
|
(let ((regs (nth-value 1 (cl-ppcre:scan-to-strings "(?s)```(?:lisp)?\\n?(.*?)\\n?```" raw-thought))))
|
|
(if (and regs (> (length regs) 0)) (elt regs 0) raw-thought))
|
|
(string-trim '(#\Space #\Newline #\Tab) raw-thought))))
|
|
(suggestion (handler-case (read-from-string cleaned-thought)
|
|
(error (c)
|
|
;; EMIT ASYNCHRONOUS REPAIR STIMULUS
|
|
(list :type :EVENT :payload
|
|
(list :sensor :syntax-error
|
|
:code cleaned-thought
|
|
:error (format nil "~a" c)))))))
|
|
(kernel-log "ASSOCIATIVE Suggestion: ~a~%" cleaned-thought)
|
|
(when (and suggestion (listp suggestion))
|
|
(push suggestion suggestions))))
|
|
(if (and *consensus-enabled-p* suggestions)
|
|
(nreverse suggestions)
|
|
(first (nreverse suggestions))))
|
|
'(:type :LOG :payload (:text "Skill triggered (Deterministic only)")))))
|
|
nil)))
|
|
#+end_src
|
|
|
|
** Prompt Meta-Cognition (distill-prompt)
|
|
Even the Associative engine can benefit from introspection. This function allows the agent to observe its own prompts and successful results to distill them into reusable templates.
|
|
|
|
#+begin_src lisp :tangle ../src/neuro.lisp
|
|
(defun distill-prompt (full-prompt successful-output)
|
|
(let ((system-instr "You are a Meta-Cognitive Prompt Architect. DISTILL into template."))
|
|
(ask-neuro (format nil "PROMPT: ~a~%RESULT: ~a" full-prompt successful-output) :system-prompt system-instr)))
|
|
#+end_src
|
|
|
|
|
|
* Deliberate Engine (symbolic.lisp)
|
|
The Deliberate engine is the deterministic gatekeeper that ensures all proposed actions—whether from the user or from the neural engine—are safe and logically valid.
|
|
|
|
As a "Thin Harness," the Deliberate engine does not contain specific security rules or task integrity checks. Instead, it provides a priority-based dispatcher that iterates through all loaded skills to validate or transform proposed actions.
|
|
|
|
** Package Context
|
|
#+begin_src lisp :tangle ../src/symbolic.lisp
|
|
(in-package :org-agent)
|
|
#+end_src
|
|
|
|
** Validation Gate (decide)
|
|
This is the **Supervisor**. It intercepts every action proposal and runs it through the symbolic gates of all registered skills, sorted by priority.
|
|
|
|
This sequential chain allows for multi-layered defense:
|
|
1. **Low Priority Gates:** Suggest refinements or logging.
|
|
2. **Medium Priority Gates:** Transform actions (e.g., adding project IDs).
|
|
3. **High Priority Gates:** Absolute security blocks (e.g., the Bouncer blocking shell access).
|
|
|
|
#+begin_src mermaid
|
|
flowchart LR
|
|
Proposal[Proposal] --> SkillA[Skill A: Priority 10]
|
|
SkillA --> SkillB[Skill B: Priority 50]
|
|
SkillB --> SkillC[Skill C: Priority 100]
|
|
SkillC --> Verified[Verified Action]
|
|
|
|
style SkillC fill:#fbb,stroke:#333
|
|
#+end_src
|
|
|
|
#+begin_src lisp :tangle ../src/symbolic.lisp
|
|
(defun decide (proposed-action context)
|
|
"The Deliberate Safety Gate: iterates through all skill symbolic-gates sorted by priority."
|
|
(let ((current-action proposed-action)
|
|
(skills nil))
|
|
;; 1. Collect all skills with symbolic gates
|
|
(maphash (lambda (name skill)
|
|
(declare (ignore name))
|
|
(when (skill-symbolic-fn skill)
|
|
(push skill skills)))
|
|
*skills-registry*)
|
|
|
|
;; 2. Sort skills by priority (highest first)
|
|
(setf skills (sort skills #'> :key #'skill-priority))
|
|
|
|
;; 3. Execute symbolic gates sequentially
|
|
(dolist (skill skills)
|
|
(let ((gate (skill-symbolic-fn skill)))
|
|
(setf current-action (funcall gate current-action context))
|
|
;; If any gate returns a LOG or EVENT (blocking/intercepting), stop and return it.
|
|
(when (and (listp current-action)
|
|
(member (getf current-action :type) '(:LOG :EVENT :log :event)))
|
|
(kernel-log "DELIBERATE: Intercepted by skill '~a'~%" (skill-name skill))
|
|
(return-from decide current-action))))
|
|
|
|
current-action))
|
|
#+end_src
|
|
|
|
** Store Filtering (list-objects-with-attribute)
|
|
A symbolic helper function to find nodes with specific attributes. This is used by skills to perform complex semantic lookups in the Object Store.
|
|
|
|
#+begin_src lisp :tangle ../src/symbolic.lisp
|
|
(defun list-objects-with-attribute (attr-key attr-val)
|
|
"Filters the Object Store for nodes having a specific attribute value."
|
|
(let ((results nil))
|
|
(maphash (lambda (id obj)
|
|
(declare (ignore id))
|
|
(when (equal (getf (org-object-attributes obj) attr-key) attr-val)
|
|
(push obj results)))
|
|
*object-store*)
|
|
results))
|
|
#+end_src
|