#+TITLE: The Neurosymbolic Bridge (neuro.lisp & symbolic.lisp)
#+AUTHOR: Amr
#+FILETAGS: :kernel:neuro:symbolic:
#+STARTUP: content

* The Neurosymbolic Bridge (neuro.lisp & symbolic.lisp)
** Deep Reasoning: Imagination Checked by Physics
Associative (LLM) is creative but hallucination-prone. Deliberate (Lisp) is rigid but 100% accurate.
- **The Safety Gate:** We never allow the LLM to talk to the actuators directly. It must propose a Lisp form. Deliberate intercepts this form and validates it against mathematical rules and PSF invariants. 
- **Sovereign Decoupling:** By moving the physical API logic into skills, the core remains a neutral "Thinking Engine" that doesn't care if the imagination comes from Google, Anthropic, or a local Llama instance.

* Associative Engine (neuro.lisp)
This module handles the interaction with Large Language Models, providing a unified interface for multiple backends. As a "Thin Harness," it does not handle authentication or specific provider logic; these are delegated to skills.

** Package Context
#+begin_src lisp :tangle ../src/neuro.lisp
(in-package :org-agent)
#+end_src

** Associative Backends Registry
Tracks the actual implementation functions for each LLM provider.

#+begin_src lisp :tangle ../src/neuro.lisp
(defvar *neuro-backends* (make-hash-table :test 'equal))
#+end_src

** Provider Cascade
The ordered list of backends to attempt for neural reasoning. This is a default that can be overridden by skills like the Token Accountant.

#+begin_src lisp :tangle ../src/neuro.lisp
(defvar *provider-cascade* '(:openrouter :gemini-api))
#+end_src

** Register Associative Backend
Maps a keyword identifier to a backend implementation function.

#+begin_src lisp :tangle ../src/neuro.lisp
(defun register-neuro-backend (name fn) (setf (gethash name *neuro-backends*) fn))
#+end_src

** Model Selector Function
A hook for dynamic model selection based on the current context.

#+begin_src lisp :tangle ../src/neuro.lisp
(defvar *model-selector-fn* nil "A function called with (provider context) to return a model ID.")
#+end_src

** Associative Dispatch (ask-neuro)
The primary entry point for Associative. It handles the retry logic and backend selection. It supports a parallel consensus mode where multiple backends are queried.

#+begin_src lisp :tangle ../src/neuro.lisp
(defvar *consensus-enabled-p* nil "If T, ask-neuro queries all backends in parallel.")

(defun ask-neuro (prompt &key (system-prompt "You are the Associative engine of a Neurosymbolic Lisp Machine.") (cascade nil) (context nil))
  "Dispatches a neural request through the provider cascade or parallel consensus."
  (let ((backends (cond
                    ((and cascade (listp cascade)) cascade)
                    ((functionp cascade) (funcall cascade context))
                    (t *provider-cascade*))))
    (if *consensus-enabled-p*
        ;; PARALLEL CONSENSUS MODE
        (let ((results nil)
              (threads nil)
              (lock (bt:make-lock)))
          (dolist (backend backends)
            (let ((backend-fn (gethash backend *neuro-backends*)))
              (when backend-fn
                (push (bt:make-thread 
                       (lambda ()
                         (kernel-log "ASSOCIATIVE [Consensus]: Querying backend ~a..." backend)
                         (let* ((model (when *model-selector-fn* (funcall *model-selector-fn* backend context)))
                                (result (ignore-errors 
                                          (if model 
                                              (funcall backend-fn prompt system-prompt :model model)
                                              (funcall backend-fn prompt system-prompt)))))
                           (bt:with-lock-held (lock)
                             (push result results)))))
                      threads))))
          ;; Wait for all threads with a timeout (e.g., 30s)
          (let ((start-time (get-universal-time)))
            (loop while (and (< (length results) (length threads))
                             (< (- (get-universal-time) start-time) 30))
                  do (sleep 0.1)))
          ;; Return the list of raw results (filtering out nils or errors)
          (let ((valid-results (remove-if-not #'stringp results)))
            (if valid-results
                (format nil "~{~a~^|CONSENSUS-SEP|~}" valid-results)
                "(:type :LOG :payload (:text \"Neural Consensus Failure\"))")))
        
        ;; SEQUENTIAL CASCADE MODE
        (or (dolist (backend backends)
              (let ((backend-fn (gethash backend *neuro-backends*)))
                (when backend-fn
                  (kernel-log "ASSOCIATIVE: Attempting backend ~a..." backend)
                  (let* ((model (when *model-selector-fn* (funcall *model-selector-fn* backend context)))
                         (result (if model 
                                     (funcall backend-fn prompt system-prompt :model model)
                                     (funcall backend-fn prompt system-prompt))))
                    (unless (or (null result)
                                (and (stringp result) (search ":LOG" result) (or (search "Failure" result) (search "missing" result))))
                      (return result))))))
            "(:type :LOG :payload (:text \"Neural Cascade Failure\"))"))))
#+end_src

** Associative Reasoning (think)
Invokes the Associative engine to generate a proposed Lisp action based on context. It automatically injects the tool documentation and global context into the prompt.

#+begin_src lisp :tangle ../src/neuro.lisp
(defun think (context)
  "Invokes the neural Associative engine to propose a Lisp action based on context."
  (let ((active-skill (find-triggered-skill context))
        (tool-belt (generate-tool-belt-prompt))
        (global-context (context-assemble-global-awareness)))
    (if active-skill
        (progn
          (kernel-log "ASSOCIATIVE: Engaging skill '~a'~%" (skill-name active-skill))
          (let* ((prompt-generator (skill-neuro-prompt active-skill)) 
                 (raw-prompt (when prompt-generator (funcall prompt-generator context)))
                 (full-system-prompt (concatenate 'string 
                                                 "ACTUATOR IDENTITY: You are the pure Lisp actuator for the org-agent kernel.
MANDATE: Output EXACTLY ONE Common Lisp property list starting with (:type :REQUEST).
ZERO CONVERSATION: Do not explain. Do not say 'Okay'. Do not use markdown blocks.
STRICT RULE: Do not output multiple lists. Do not chain multiple requests. 
DO NOT embed tool calls inside text strings.

"
                                                 global-context
                                                 "
"
                                                 tool-belt
                                                 "
IMPORTANT: To reply to the user, you MUST use:
(:type :REQUEST :target :emacs :action :insert-at-end :buffer \"*org-agent-chat*\" :text \"* <Response Text>\")

To call a tool, you MUST use:
(:type :REQUEST :target :tool :action :call :tool \"<name>\" :args (:arg1 \"val\"))

")))
            (if (and raw-prompt (> (length raw-prompt) 1))
                (let* ((thought (ask-neuro raw-prompt :system-prompt full-system-prompt :context context))
                       (raw-thoughts (cl-ppcre:split (cl-ppcre:quote-meta-chars "|CONSENSUS-SEP|") thought))
                       (suggestions nil))
                  (dolist (raw-thought raw-thoughts)
                    (kernel-log "ASSOCIATIVE RAW: ~a~%" raw-thought)
                    (let* ((cleaned-thought 
                            (let ((match (cl-ppcre:scan-to-strings "(?s)```(?:lisp)?\\n?(.*?)\\n?```" raw-thought)))
                              (if match
                                  (let ((regs (nth-value 1 (cl-ppcre:scan-to-strings "(?s)```(?:lisp)?\\n?(.*?)\\n?```" raw-thought))))
                                    (if (and regs (> (length regs) 0)) (elt regs 0) raw-thought))
                                  (string-trim '(#\Space #\Newline #\Tab) raw-thought))))
                           (suggestion (handler-case (read-from-string cleaned-thought)
                                         (error (c)
                                           ;; EMIT ASYNCHRONOUS REPAIR STIMULUS
                                           (list :type :EVENT :payload 
                                                 (list :sensor :syntax-error 
                                                       :code cleaned-thought 
                                                       :error (format nil "~a" c)))))))
                      (kernel-log "ASSOCIATIVE Suggestion: ~a~%" cleaned-thought)
                      (when (and suggestion (listp suggestion))
                        (push suggestion suggestions))))
                  (if (and *consensus-enabled-p* suggestions)
                      (nreverse suggestions)
                      (first (nreverse suggestions))))
                '(:type :LOG :payload (:text "Skill triggered (Deterministic only)")))))
        nil)))
#+end_src

** Prompt Meta-Cognition (distill-prompt)
Allows the agent to self-optimize its own prompts.

#+begin_src lisp :tangle ../src/neuro.lisp
(defun distill-prompt (full-prompt successful-output)
  (let ((system-instr "You are a Meta-Cognitive Prompt Architect. DISTILL into template."))
    (ask-neuro (format nil "PROMPT: ~a~%RESULT: ~a" full-prompt successful-output) :system-prompt system-instr)))
#+end_src


* Deliberate Engine (symbolic.lisp)
The deterministic gatekeeper that ensures all proposed actions are safe and logically valid. As a "Thin Harness," the Deliberate engine does not contain specific security rules; instead, it provides a priority-based dispatcher that iterates through all loaded skills to validate or transform proposed actions.

** Package Context
#+begin_src lisp :tangle ../src/symbolic.lisp
(in-package :org-agent)
#+end_src

** Validation Gate (decide)
The "Deliberate" supervisor. It intercepts every action proposed by Associative and runs it through the symbolic gates of all registered skills, sorted by priority.

#+begin_src lisp :tangle ../src/symbolic.lisp
(defun decide (proposed-action context)
  "The Deliberate Safety Gate: iterates through all skill symbolic-gates sorted by priority."
  (let ((current-action proposed-action)
        (skills nil))
    ;; 1. Collect all skills with symbolic gates
    (maphash (lambda (name skill)
               (declare (ignore name))
               (when (skill-symbolic-fn skill)
                 (push skill skills)))
             *skills-registry*)
    
    ;; 2. Sort skills by priority (highest first)
    (setf skills (sort skills #'> :key #'skill-priority))
    
    ;; 3. Execute symbolic gates sequentially
    (dolist (skill skills)
      (let ((gate (skill-symbolic-fn skill)))
        (setf current-action (funcall gate current-action context))
        ;; If any gate returns a LOG or EVENT (blocking/intercepting), stop and return it.
        (when (and (listp current-action) 
                   (member (getf current-action :type) '(:LOG :EVENT :log :event)))
          (kernel-log "DELIBERATE: Intercepted by skill '~a'~%" (skill-name skill))
          (return-from decide current-action))))
    
    current-action))
#+end_src

** Store Filtering (list-objects-with-attribute)
A symbolic helper function to find nodes with specific attributes.

#+begin_src lisp :tangle ../src/symbolic.lisp
(defun list-objects-with-attribute (attr-key attr-val)
  "Filters the Object Store for nodes having a specific attribute value."
  (let ((results nil))
    (maphash (lambda (id obj) 
               (declare (ignore id)) 
               (when (equal (getf (org-object-attributes obj) attr-key) attr-val) 
                 (push obj results))) 
             *object-store*)
    results))
#+end_src