passepartout/literate/communication.org

#+TITLE: Communication Protocol (communication.lisp)
#+AUTHOR: Amr
#+FILETAGS: :harness:protocol:
#+STARTUP: content

* Communication Protocol (communication.lisp)
** Architectural Intent: Secure Inter-Process Communication & Deterministic Framing

The **communication protocol** defines the exact physical and semantic boundaries for how the isolated Lisp environment talks to the outside world.

The system operates as a perfectly deterministic, highly secure computational engine. When communicating with external processes or actuators—such as an Emacs client, a web dashboard, or a remote Shell script—it cannot rely on unpredictable, "loose" data streams. 

Streaming raw Lisp or JSON over a TCP socket is inherently fragile. If a multi-megabyte Org Abstract Syntax Tree (AST) is fragmented by the operating system's network stack during transmission, a standard stream parser might attempt to evaluate an incomplete string, leading to immediate crashes or desynchronization.

To solve this, we implement the **communication protocol**, which enforces absolute deterministic boundaries around every message.

*** 1. Physical Boundary: Hex-Length Prefixing
Every message crossing the wire is prefixed with a strict 6-character hexadecimal length string (zero-padded). This creates an unbreakable physical boundary. The system reads exactly the number of bytes specified by the hex length. It will never under-read (crashing on a partial form) and never over-read (consuming bytes meant for the next message). 

*** 2. Actuator-Agnosticism ("Dumb Terminal" Architecture)
The protocol keeps the Lisp environment completely agnostic of its clients. The system does not care if the client is written in Emacs Lisp, Python, or Rust. Any environment capable of calculating a byte length and opening a TCP socket can interface with the Lisp Machine.

*** 3. Preventing Reader Macro Injection
Common Lisp's ~read-from-string~ is extremely powerful but dangerous; it allows "reader macros" (like ~#.~) which execute code during the parsing phase. The communication protocol mandates that ~*read-eval*~ is explicitly bound to ~nil~ before any network data is parsed, physically preventing arbitrary code execution.

** Message Framing Logic
#+begin_src mermaid
flowchart LR
    subgraph Client
        A[Lisp Property List] --> B[Calculate Length]
        B --> C[Hex Prefix: 6 Chars]
        C --> D[Concatenate: Length + List]
    end
    D -- TCP Socket --> System
    subgraph System
        System --> E[Read 6 Hex Chars]
        E --> F[Read Exact Byte Count]
        F --> G[Parse S-Expression]
    end
#+end_src

** Package Context
We ensure all protocol logic resides within the isolated system namespace.

#+begin_src lisp :tangle ../src/communication.lisp
(in-package :opencortex)
#+end_src

** Actuator Registry
The system maintains a decoupled registry of target actuators. This allows the system to route messages to Emacs, the Shell, or Web Gateways without hardcoding the routing logic into the protocol itself.

#+begin_src lisp :tangle ../src/communication.lisp
(defvar *actuator-registry* (make-hash-table :test 'equal)
  "Global registry mapping target keywords to their physical actuator functions.")

(defun register-actuator (name fn) 
  "Registers an actuator function. Actuators receive: (ACTION CONTEXT)."
  (setf (gethash name *actuator-registry*) fn))
#+end_src

** Message Framing (frame-message)
The ~frame-message~ function prepares an outgoing Lisp string for transmission. It calculates the byte length, converts it into a 6-character padded hex string, and prefixes it. If ~COMMUNICATION_PROTOCOL_ENFORCE_HMAC~ is enabled in the environment, it also prepends a cryptographic signature to ensure the message hasn't been tampered with.

#+begin_src lisp :tangle ../src/communication.lisp
(defun frame-message (msg-string)
  "Prefixes MSG-STRING with a 6-character hex length.
   If security is enabled, prefixes a 64-char HMAC-SHA256 signature."
  (let ((len (length msg-string))
        (enforce-hmac (uiop:getenv "COMMUNICATION_PROTOCOL_ENFORCE_HMAC")))
    (if (and enforce-hmac (string-equal enforce-hmac "true"))
        (let ((secret (uiop:getenv "COMMUNICATION_PROTOCOL_HMAC_SECRET")))
          (unless secret (error "COMMUNICATION_PROTOCOL_HMAC_SECRET is required when security is enabled."))
          (let* ((key (ironclad:ascii-string-to-byte-array secret))
                 (hmac (ironclad:make-mac :hmac key :sha256))
                 (payload-bytes (ironclad:ascii-string-to-byte-array msg-string)))
            (ironclad:update-mac hmac payload-bytes)
            (let ((signature (ironclad:byte-array-to-hex-string (ironclad:produce-mac hmac))))
              (format nil "~(~6,'0x~)~a~a" len signature msg-string))))
        (format nil "~(~6,'0x~)~a" len msg-string))))
#+end_src

** Message Parsing (parse-message)
Parsing is the high-security inverse of framing. This function acts as the final perimeter defense. It validates the length, verifies the HMAC integrity, and—most importantly—jails the Lisp reader by disabling ~*read-eval*~. 

#+begin_src lisp :tangle ../src/communication.lisp
(defun parse-message (framed-string)
  "Extracts and parses the S-expression from a framed string securely."
  (when (< (length framed-string) 6)
    (error "Framed string too short"))
  (let* ((enforce-hmac (uiop:getenv "COMMUNICATION_PROTOCOL_ENFORCE_HMAC"))
         (use-hmac (and enforce-hmac (string-equal enforce-hmac "true")))
         (prefix-len (if use-hmac 70 6)))
    (when (< (length framed-string) prefix-len)
      (error "Framed string too short for communication protocol prefix"))
    
    (let* ((len-str (subseq framed-string 0 6))
           (signature (when use-hmac (subseq framed-string 6 70)))
           (actual-msg (subseq framed-string prefix-len))
           (expected-len (ignore-errors (parse-integer len-str :radix 16))))
      (unless expected-len
        (error "Invalid hex length prefix: ~a" len-str))
      (unless (= expected-len (length actual-msg))
        (error "Message length mismatch. Expected ~a, got ~a" expected-len (length actual-msg)))
      
      (when use-hmac
        (let ((secret (uiop:getenv "COMMUNICATION_PROTOCOL_HMAC_SECRET")))
          (unless secret (error "COMMUNICATION_PROTOCOL_HMAC_SECRET is required when security is enabled."))
          (let* ((key (ironclad:ascii-string-to-byte-array secret))
                 (hmac (ironclad:make-mac :hmac key :sha256))
                 (payload-bytes (ironclad:ascii-string-to-byte-array actual-msg)))
            (ironclad:update-mac hmac payload-bytes)
            (let ((expected-signature (ironclad:byte-array-to-hex-string (ironclad:produce-mac hmac))))
              (unless (string-equal signature expected-signature)
                (error "communication protocol Integrity Failure: HMAC mismatch"))))))
      
      ;; SECURITY: Disable the reader's ability to execute code during parsing
      (let ((*read-eval* nil))
        (let ((msg (read-from-string actual-msg)))
          (validate-communication-protocol-schema msg)
          msg)))))
#+end_src

** Handshaking (make-hello-message)
Every session begins with a standard ~HELLO~ handshake, allowing the system to announce its capabilities and protocol version to the connecting client.

#+begin_src lisp :tangle ../src/communication.lisp
(defun make-hello-message (version)
  "Constructs the standard HELLO handshake message."
  (list :type :EVENT 
        :payload (list :action :handshake 
                       :version version 
                       :capabilities '(:auth :swank :org-ast))))
#+end_src


** Protocol Reading (read-framed-message)
A robust utility to read a framed message from a stream. It enforces the deterministic hex-length boundary.

#+begin_src lisp :tangle ../src/communication.lisp
(defun read-framed-message (stream)
  "Reads a hex-length prefixed message from the stream securely. Skips leading whitespace."
  (let ((length-buffer (make-string 6)))
    (handler-case
        (progn
          ;; 0. Skip leading whitespace (newlines, spaces, etc.)
          (loop for char = (peek-char nil stream nil :eof)
                while (and (not (eq char :eof)) (member char '(#\Space #\Newline #\Tab #\Return)))
                do (read-char stream))

          ;; 1. Read the 6-char hex length
          (let ((count (read-sequence length-buffer stream)))
            (when (< count 6) (return-from read-framed-message :eof))
            (let ((len (ignore-errors (parse-integer length-buffer :radix 16))))
              (unless len (error "Invalid protocol header: ~a" length-buffer))
              
              ;; 2. Read exactly LEN bytes
              (let ((msg-buffer (make-string len)))
                (read-sequence msg-buffer stream)
                (let ((*read-eval* nil))
                  (let ((msg (read-from-string msg-buffer)))
                    (validate-communication-protocol-schema msg)
                    msg))))))
      (error (c) 
        (harness-log "PROTOCOL READ ERROR: ~a" c)
        :error))))
#+end_src