amr/passepartout
1
0
Fork 0
Code Issues 15 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

ALIGN: Rename Protocol to Communication and unify terminology

Browse Source
This commit is contained in:
Amr Gharbeia
2026-04-13 14:17:28 -04:00
parent f4e74b732d
commit 5f86bcd8dc
84 changed files with 383 additions and 347 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
literate/protocol.org → literate/communication.org
View File

@@ -1,16 +1,18 @@
#+TITLE: The Harness Protocol (protocol.lisp)
#+TITLE: The Harness Communication (communication.lisp)
#+AUTHOR: Amr
#+FILETAGS: :harness:protocol:
#+STARTUP: content
* The Harness Protocol (protocol.lisp)
** Architectural Intent: Deterministic Framing & Reader Security
* The Harness Communication (communication.lisp)
** Architectural Intent: Secure Inter-Process Communication & Deterministic Framing
The ~org-agent~ harness operates as a perfectly deterministic, highly secure computational engine. When communicating with external actuators—such as an Emacs instance, a web dashboard, or a remote script—the harness cannot rely on unpredictable, "loose" data streams.
Yes, the Harness Communication is fundamentally about **Communication**. It defines the exact physical and semantic boundaries for how the isolated Lisp environment talks to the outside world.
The ~org-agent~ harness operates as a perfectly deterministic, highly secure computational engine. When communicating with external processes or actuators—such as an Emacs client, a web dashboard, or a remote Shell script—the harness cannot rely on unpredictable, "loose" data streams.
Streaming raw Lisp or JSON over a TCP socket is inherently fragile. If a multi-megabyte Org Abstract Syntax Tree (AST) is fragmented by the operating system's network stack during transmission, a standard stream parser might attempt to evaluate an incomplete string, leading to immediate crashes or desynchronization.
To solve this, we implement the **Harness Protocol**, which enforces absolute deterministic boundaries around every message.
To solve this, we implement the **Harness Communication**, which enforces absolute deterministic boundaries around every message.
*** 1. Physical Boundary: Hex-Length Prefixing
Every message crossing the wire is prefixed with a strict 6-character hexadecimal length string (zero-padded). This creates an unbreakable physical boundary. The harness reads exactly the number of bytes specified by the hex length. It will never under-read (crashing on a partial form) and never over-read (consuming bytes meant for the next message).
@@ -19,7 +21,7 @@ Every message crossing the wire is prefixed with a strict 6-character hexadecima
The protocol keeps the Lisp harness completely agnostic of its clients. The harness does not care if the client is written in Emacs Lisp, Python, or Rust. Any environment capable of calculating a byte length and opening a TCP socket can interface with the Lisp Machine.
*** 3. Preventing Reader Macro Injection
Common Lisp's ~read-from-string~ is extremely powerful but dangerous; it allows "reader macros" (like ~#.~) which execute code during the parsing phase. The Harness Protocol mandates that ~*read-eval*~ is explicitly bound to ~nil~ before any network data is parsed, physically preventing arbitrary code execution.
Common Lisp's ~read-from-string~ is extremely powerful but dangerous; it allows "reader macros" (like ~#.~) which execute code during the parsing phase. The Harness Communication mandates that ~*read-eval*~ is explicitly bound to ~nil~ before any network data is parsed, physically preventing arbitrary code execution.
** Message Framing Logic
#+begin_src mermaid
@@ -40,14 +42,14 @@ flowchart LR
** Package Context
We ensure all protocol logic resides within the isolated harness namespace.
#+begin_src lisp :tangle ../src/protocol.lisp
#+begin_src lisp :tangle ../src/communication.lisp
(in-package :org-agent)
#+end_src
** Actuator Registry
The harness maintains a decoupled registry of target actuators. This allows the system to route messages to Emacs, the Shell, or Web Gateways without hardcoding the routing logic into the protocol itself.
#+begin_src lisp :tangle ../src/protocol.lisp
#+begin_src lisp :tangle ../src/communication.lisp
(defvar *actuator-registry* (make-hash-table :test 'equal)
"Global registry mapping target keywords to their physical actuator functions.")
@@ -59,7 +61,7 @@ The harness maintains a decoupled registry of target actuators. This allows the
** Message Framing (frame-message)
The ~frame-message~ function prepares an outgoing Lisp string for transmission. It calculates the byte length, converts it into a 6-character padded hex string, and prefixes it. If ~HARNESS_PROTOCOL_ENFORCE_HMAC~ is enabled in the environment, it also prepends a cryptographic signature to ensure the message hasn't been tampered with.
#+begin_src lisp :tangle ../src/protocol.lisp
#+begin_src lisp :tangle ../src/communication.lisp
(defun frame-message (msg-string)
"Prefixes MSG-STRING with a 6-character hex length.
If security is enabled, prefixes a 64-char HMAC-SHA256 signature."
@@ -80,7 +82,7 @@ The ~frame-message~ function prepares an outgoing Lisp string for transmission.
** Message Parsing (parse-message)
Parsing is the high-security inverse of framing. This function acts as the final perimeter defense. It validates the length, verifies the HMAC integrity, and—most importantly—jails the Lisp reader by disabling ~*read-eval*~.
#+begin_src lisp :tangle ../src/protocol.lisp
#+begin_src lisp :tangle ../src/communication.lisp
(defun parse-message (framed-string)
"Extracts and parses the S-expression from a framed string securely."
(when (< (length framed-string) 6)
@@ -89,7 +91,7 @@ Parsing is the high-security inverse of framing. This function acts as the final
(use-hmac (and enforce-hmac (string-equal enforce-hmac "true")))
(prefix-len (if use-hmac 70 6)))
(when (< (length framed-string) prefix-len)
(error "Framed string too short for Harness Protocol prefix"))
(error "Framed string too short for Harness Communication prefix"))
(let* ((len-str (subseq framed-string 0 6))
(signature (when use-hmac (subseq framed-string 6 70)))
@@ -109,7 +111,7 @@ Parsing is the high-security inverse of framing. This function acts as the final
(ironclad:update-mac hmac payload-bytes)
(let ((expected-signature (ironclad:byte-array-to-hex-string (ironclad:produce-mac hmac))))
(unless (string-equal signature expected-signature)
(error "Harness Protocol Integrity Failure: HMAC mismatch"))))))
(error "Harness Communication Integrity Failure: HMAC mismatch"))))))
;; SECURITY: Disable the reader's ability to execute code during parsing
(let ((*read-eval* nil))
@@ -121,7 +123,7 @@ Parsing is the high-security inverse of framing. This function acts as the final
** Handshaking (make-hello-message)
Every session begins with a standard ~HELLO~ handshake, allowing the harness to announce its capabilities and protocol version to the connecting client.
#+begin_src lisp :tangle ../src/protocol.lisp
#+begin_src lisp :tangle ../src/communication.lisp
(defun make-hello-message (version)
"Constructs the standard HELLO handshake message."
(list :type :EVENT

2
literate/object-store.org
View File

@@ -24,7 +24,7 @@ flowchart TD
H -- Pointers --> S2
end
subgraph IPCSlow[External Layer]
E[Emacs / Actuators] -. Harness Protocol .-> H
E[Emacs / Actuators] -. Harness Communication .-> H
end
#+end_src

16
literate/package.org
View File

@@ -11,7 +11,7 @@ By strictly defining the public interface, we ensure that skills remain decouple
#+begin_src mermaid
flowchart TD
External[Actuators / Clients] -- Harness Protocol --> Package[Package Membrane: API]
External[Actuators / Clients] -- Harness Communication --> Package[Package Membrane: API]
Skills[Dynamic Skills] -- API Calls --> Package
Package --> Internal[Harness Internal Modules]
style Package fill:#f9f,stroke:#333,stroke-width:4px
@@ -22,7 +22,7 @@ flowchart TD
(defpackage :org-agent
(:use :cl)
(:export
;; --- Harness Protocol ---
;; --- Harness Communication ---
#:frame-message
#:parse-message
#:make-hello-message
@@ -68,7 +68,7 @@ flowchart TD
;; --- Reactive Signal Pipeline ---
#:process-signal
#:perceive-gate
#:neuro-gate
#:probabilistic-gate
#:consensus-gate
#:decide-gate
#:dispatch-gate
@@ -90,8 +90,8 @@ flowchart TD
#:skill-priority
#:skill-dependencies
#:skill-trigger-fn
#:skill-neuro-prompt
#:skill-symbolic-fn
#:skill-probabilistic-prompt
#:skill-deterministic-fn
;; --- Tool Registry ---
#:def-cognitive-tool
@@ -110,12 +110,12 @@ flowchart TD
#:unregister-emacs-client
;; --- Probabilistic Engine ---
#:ask-neuro
#:register-neuro-backend
#:ask-probabilistic
#:register-probabilistic-backend
#:distill-prompt
#:*provider-cascade*
;; --- Symbolic Logic ---
;; --- Deterministic Logic ---
#:list-objects-with-attribute
#:decide

10
literate/reason.org
View File

@@ -7,7 +7,7 @@
** Architectural Intent: Unified Cognition
The Reason stage is the cognitive engine of the Org-Agent. It unifies two distinct reasoning modes:
1. **Probabilistic Reasoning:** Consulting neural models to generate action proposals based on context.
2. **Deterministic Reasoning:** Running those proposals through symbolic safety gates (Policy and Validation) to ensure alignment.
2. **Deterministic Reasoning:** Running those proposals through deterministic safety gates (Policy and Validation) to ensure alignment.
#+begin_src lisp :tangle ../src/reason.lisp
(in-package :org-agent)
@@ -43,7 +43,7 @@ The Reason stage is the cognitive engine of the Org-Agent. It unifies two distin
(tool-belt (generate-tool-belt-prompt))
(global-context (context-assemble-global-awareness)))
(if active-skill
(let* ((prompt-generator (skill-neuro-prompt active-skill))
(let* ((prompt-generator (skill-probabilistic-prompt active-skill))
(raw-prompt (when prompt-generator (funcall prompt-generator context)))
(system-prompt (concatenate 'string "IDENTITY: Actuator for org-agent. MANDATE: ONE Lisp plist. " global-context " " tool-belt)))
(if (and raw-prompt (> (length raw-prompt) 1))
@@ -57,13 +57,13 @@ The Reason stage is the cognitive engine of the Org-Agent. It unifies two distin
;; --- 2. Deterministic Mechanisms ---
(defun deterministic-verify (proposed-action context)
"Iterates through all skill symbolic-gates sorted by priority."
"Iterates through all skill deterministic-gates sorted by priority."
(let ((current-action proposed-action)
(skills nil))
(maphash (lambda (name skill) (declare (ignore name)) (when (skill-symbolic-fn skill) (push skill skills))) *skills-registry*)
(maphash (lambda (name skill) (declare (ignore name)) (when (skill-deterministic-fn skill) (push skill skills))) *skills-registry*)
(setf skills (sort skills #'> :key #'skill-priority))
(dolist (skill skills)
(let ((gate (skill-symbolic-fn skill)))
(let ((gate (skill-deterministic-fn skill)))
(setf current-action (funcall gate current-action context))
(when (and (listp current-action) (member (getf current-action :type) '(:LOG :EVENT)))
(harness-log "DETERMINISTIC: Intercepted by skill '~a'" (skill-name skill))

53
literate/skills.org
View File

@@ -44,10 +44,10 @@ We begin by ensuring we are in the correct isolated harness namespace.
#+end_src
** Skill Metadata (defstruct skill)
The core data structure representing an agent capability. It includes the trigger condition, the neural prompt generator, and the symbolic safety gate.
The core data structure representing an agent capability. It includes the trigger condition, the probabilistic prompt generator, and the deterministic safety gate.
#+begin_src lisp :tangle ../src/skills.lisp
(defstruct skill name priority dependencies trigger-fn neuro-prompt symbolic-fn)
(defstruct skill name priority dependencies trigger-fn probabilistic-prompt deterministic-fn)
#+end_src
** Skill Catalog Tracking
@@ -83,15 +83,15 @@ The primary dispatcher for the Probabilistic Engine. It iterates through the reg
The interface used within Org files to register new capabilities. Note that dependencies are explicitly quoted to prevent premature evaluation during the macro expansion phase.
#+begin_src lisp :tangle ../src/skills.lisp
(defmacro defskill (name &key priority dependencies trigger neuro symbolic)
(defmacro defskill (name &key priority dependencies trigger probabilistic deterministic)
"Registers a new skill into the global registry."
`(setf (gethash (string-downcase (string ,name)) *skills-registry*)
(make-skill :name (string-downcase (string ,name))
:priority (or ,priority 10)
:dependencies ',dependencies
:trigger-fn ,trigger
:neuro-prompt ,neuro
:symbolic-fn ,symbolic)))
:probabilistic-prompt ,probabilistic
:deterministic-fn ,deterministic)))
#+end_src
** Dependency Resolution (resolve-skill-dependencies)
@@ -118,21 +118,19 @@ A robust, low-level scanner that extracts `#+DEPENDS_ON:` and `:ID:` tags from a
#+begin_src lisp :tangle ../src/skills.lisp
(defun parse-skill-metadata (filepath)
"Extracts ID and DEPENDS_ON tags using robust line-scanning."
"Extracts ID and DEPENDS_ON tags using robust regex scanning."
(let ((dependencies nil)
(id nil))
(with-open-file (stream filepath)
(loop for line = (read-line stream nil :eof)
until (eq line :eof)
do (let ((clean (string-trim '(#\Space #\Tab #\Return #\Newline) line)))
(cond
((uiop:string-prefix-p "#+DEPENDS_ON:" (string-upcase clean))
(let* ((deps-part (string-trim " " (subseq clean 13))))
(setf dependencies (append dependencies
(mapcar (lambda (s) (string-trim "[] " s))
(uiop:split-string deps-part :separator '(#\Space #\Tab)))))))
((uiop:string-prefix-p ":ID:" (string-upcase clean))
(setf id (string-trim '(#\Space #\Tab) (subseq clean 4))))))))
(id nil)
(content (uiop:read-file-string filepath)))
;; Extract ID
(multiple-value-bind (match regs)
(ppcre:scan-to-strings "(?im:^:ID:\\s*([^\\s\\r\\n]+))" content)
(when match (setf id (aref regs 0))))
;; Extract all DEPENDS_ON lines
(ppcre:do-register-groups (deps-string)
("(?im:^#\\+DEPENDS_ON:\\s*(.*))" content)
(let ((deps (ppcre:split "\\s+" (string-trim " " deps-string))))
(setf dependencies (append dependencies (mapcar (lambda (s) (string-trim "[] " s)) deps)))))
(values id (remove-if (lambda (s) (= 0 (length s))) dependencies))))
#+end_src
@@ -149,6 +147,7 @@ It performs three critical roles:
"Returns a list of skill filepaths sorted by dependency (dependencies first)."
(let ((files (uiop:directory-files skills-dir "org-skill-*.org"))
(adj (make-hash-table :test 'equal))
(name-to-file (make-hash-table :test 'equal))
(id-to-file (make-hash-table :test 'equal))
(result nil)
(visited (make-hash-table :test 'equal))
@@ -156,7 +155,7 @@ It performs three critical roles:
(dolist (file files)
(let ((filename (pathname-name file)))
(multiple-value-bind (id deps) (parse-skill-metadata file)
(setf (gethash (string-downcase filename) id-to-file) file)
(setf (gethash (string-downcase filename) name-to-file) file)
(when id (setf (gethash (string-downcase id) id-to-file) file))
(setf (gethash (string-downcase filename) adj) deps))))
(labels ((visit (file)
@@ -165,10 +164,12 @@ It performs three critical roles:
(unless (gethash node-key visited)
(setf (gethash node-key stack) t)
(dolist (dep (gethash node-key adj))
(let* ((dep-id (if (and (> (length dep) 3) (uiop:string-prefix-p "id:" (string-downcase dep)))
(subseq dep 3)
dep))
(dep-file (gethash (string-downcase dep-id) id-to-file)))
(let* ((is-id-p (uiop:string-prefix-p "id:" (string-downcase dep)))
(dep-key (string-downcase (if is-id-p (subseq dep 3) dep)))
(dep-file (if is-id-p
(gethash dep-key id-to-file)
(or (gethash dep-key id-to-file)
(gethash dep-key name-to-file)))))
(when dep-file
(let ((dep-filename (pathname-name dep-file)))
(if (gethash (string-downcase dep-filename) stack)
@@ -179,9 +180,9 @@ It performs three critical roles:
(push file result)))))
(let ((filenames (sort (mapcar #'pathname-name files) #'string<)))
(dolist (name filenames)
(let ((file (gethash (string-downcase name) id-to-file)))
(let ((file (gethash (string-downcase name) name-to-file)))
(when file (visit file)))))
result)))
(nreverse result))))
#+end_src
** Syntax Validation (validate-lisp-syntax)

12
literate/system-definition.org
View File

@@ -35,24 +35,24 @@ This system defines the core "Thin Harness." It includes the protocol, the objec
:author "Amr"
:version "0.1.0"
:license "MIT"
:description "The Neurosymbolic Lisp Machine Harness"
:description "The Probabilistic-Deterministic Lisp Machine Harness"
:depends-on (:usocket :cl-json :bordeaux-threads :dexador :uiop :cl-dotenv :cl-ppcre :hunchentoot :ironclad :str)
:serial t
:components ((:file "src/package")
(:file "src/skills")
(:file "src/system-invariants")
(:file "src/engineering-standards")
(:file "src/protocol-validator")
(:file "src/protocol")
(:file "src/communication-validator")
(:file "src/communication")
(:file "src/object-store")
(:file "src/embedding")
(:file "src/embedding-logic")
(:file "src/context")
(:file "src/context-logic")
(:file "src/neuro")
(:file "src/probabilistic")
(:file "src/credentials-vault")
(:file "src/llm-gateway")
(:file "src/symbolic")
(:file "src/deterministic")
(:file "src/lisp-validator")
(:file "src/self-fix")
(:file "src/lisp-repair")
@@ -74,7 +74,7 @@ This system contains the empirical tests required by the Engineering Standards.
#+begin_src lisp :tangle ../org-agent.asd
(defsystem :org-agent/tests
:depends-on (:org-agent :fiveam)
:components ((:file "tests/protocol-tests")
:components ((:file "tests/communication-tests")
(:file "tests/pipeline-tests")
(:file "tests/peripheral-vision-tests")
(:file "tests/lisp-validator-tests")