amr/passepartout
1
0
Fork 0
Code Issues 15 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: Comprehensive literate documentation for harness core

Browse Source 
- loop.org: Explain metabolic pipeline, error recovery, heartbeat
- reason.org: Document dual-engine architecture (probabilistic + deterministic)
- act.org: Detail actuation, actuator pattern, feedback loop
- perceive.org: Explain signal normalization, async vs sync processing
- manifest.org: Add ASDF system design, boundary contract

Each function now has its own code block with detailed docstrings explaining:
- What the function does
- Why it was designed that way
- How it fits into the larger system
- Parameter descriptions
- Return values
This commit is contained in:
Amr Gharbeia
2026-04-22 16:49:27 -04:00
parent b2d85ac4ae
commit 2cdd8fe1a4
5 changed files with 1200 additions and 289 deletions
Download Patch File Download Diff File Expand all files Collapse all files

258
harness/loop.org
View File

@@ -4,128 +4,284 @@
#+STARTUP: content
* The Metabolic Loop (loop.lisp)
** Architectural Intent: The Heartbeat
The Metabolic Loop is the high-level coordinator of the OpenCortex. It orchestrates the flow of energy (information) through the system by calling the three metabolic stages in sequence:
1. **Perceive:** Sensory intake.
2. **Reason:** Cognitive processing.
3. **Act:** Physical side-effects.
** Architectural Intent
** Package and Variables
The loop requires thread-safe interrupt handling to ensure that the agent can be stopped gracefully without leaving the Lisp image in an inconsistent state.
The Metabolic Loop is the /cranial nerve reflex/ of OpenCortex. While skills provide specialized intelligence, the loop provides the fundamental rhythm of existence: the continuous processing of signals from perception through cognition to action.
Unlike a simple event loop, the Metabolic Loop implements a sophisticated error recovery model. When the system encounters an error, it distinguishes between:
1. *Transient errors* (tool failures, network timeouts) - recoverable, no state rollback
2. *Critical errors* (undefined functions, malformed data structures) - require memory rollback
3. *Recursive loops* (signals generating more signals indefinitely) - depth limit enforcement
This design ensures the agent remains stable under adverse conditions while preserving the ability to recover from genuine system failures.
** Why Separate Perceive-Reason-Act?
The three-stage pipeline mirrors the classical sense-think-act paradigm but with a crucial difference: each stage is a pure function that transforms a signal. This allows:
- *Perceive* to normalize raw input into a standardized signal format
- *Reason* to transform the perceived signal into an approved action (or reject it)
- *Act* to execute the approved action and potentially generate a feedback signal
The feedback loop (Act returning a signal that feeds back into Perceive) enables complex multi-step operations where each action can trigger subsequent reasoning.
** Thread Safety
The loop operates in a multi-threaded environment:
- The main thread runs the heartbeat and idle loop
- Async sensors spawn threads for non-blocking I/O
- Interrupt handling requires mutex protection to prevent race conditions
* Package and Thread-Safe Variables
#+begin_src lisp :tangle ../library/loop.lisp
(in-package :opencortex)
(defvar *interrupt-flag* nil)
(defvar *interrupt-lock* (bt:make-lock "harness-interrupt-lock"))
(defvar *heartbeat-thread* nil)
(defvar *interrupt-flag* nil
"Atomic flag set by signal handlers to trigger graceful shutdown.
Using a dedicated variable avoids race conditions in interrupt handling.")
(defvar *interrupt-lock* (bt:make-lock "harness-interrupt-lock")
"Mutex protecting *interrupt-flag* access.
Locking is required because SBCL's interrupt handlers run in uncertain contexts.")
(defvar *heartbeat-thread* nil
"Handle to the heartbeat thread, allowing explicit termination on shutdown.")
#+end_src
** The Metabolic Pipeline
The `process-signal` function is the core metabolic processor. It iterates through the Perceive-Reason-Act gates until the signal is fully processed or an error state is reached. We have refined the error handling to ensure that memory rollbacks only occur on critical system failures, preventing transient tool errors from wiping short-term cognitive state.
* The Metabolic Pipeline
** process-signal: The Core Engine
This function implements the Perceive-Reason-Act pipeline. It processes a signal through all three stages and handles the feedback loop where Actions can generate new signals.
The depth counter prevents infinite recursion—a signal that generates another signal that generates another, etc. By limiting to depth 10, we ensure the system eventually converges or gracefully terminates.
#+begin_src lisp :tangle ../library/loop.lisp
(defun process-signal (signal)
"The entry point to the Metabolic Pipeline: Perceive -> Reason -> Act."
"The entry point to the Metabolic Pipeline: Perceive -> Reason -> Act.
SIGNAL is a property list with the following structure:
- :type - :EVENT, :REQUEST, :RESPONSE, etc.
- :payload - The actual content (sensor data, approved actions, etc.)
- :meta - Metadata including source, session, reply stream
- :depth - Recursion depth counter (starts at 0)
- :status - Processing status (:perceived, :reasoned, :acted)
Returns NIL when processing is complete, or a new signal for feedback loop."
(let ((current-signal signal))
(loop while current-signal do
;; Depth limiting prevents infinite recursion from feedback loops
(let ((depth (getf current-signal :depth 0))
(meta (getf current-signal :meta)))
(when (> depth 10) (harness-log "METABOLISM ERROR: Max depth reached.") (return nil))
(when (> depth 10)
(harness-log "METABOLISM ERROR: Max recursion depth reached.")
(return nil))
;; Check for graceful shutdown interrupt
(when (bt:with-lock-held (*interrupt-lock*) *interrupt-flag*)
(harness-log "METABOLISM: Interrupted.")
(harness-log "METABOLISM: Interrupted by shutdown signal.")
(bt:with-lock-held (*interrupt-lock*) (setf *interrupt-flag* nil))
(return nil))
;; The three-stage pipeline wrapped in error handling
(handler-case
(progn
;; Stage 1: Perceive - normalize sensory input
(setf current-signal (perceive-gate current-signal))
;; Stage 2: Reason - generate and verify action proposals
(setf current-signal (reason-gate current-signal))
;; Stage 3: Act - execute approved actions
(let ((feedback (act-gate current-signal)))
;; feedback generation
(if feedback
;; Action generated a feedback signal - continue processing
(progn
;; Inherit meta from trigger signal
(unless (getf feedback :meta) (setf (getf feedback :meta) meta))
;; Preserve metadata from original signal
(unless (getf feedback :meta)
(setf (getf feedback :meta) meta))
(setf current-signal feedback))
;; No feedback - pipeline complete
(setf current-signal nil))))
;; Error recovery with differentiated response
(error (c)
(let ((sensor (ignore-errors (getf (getf current-signal :payload) :sensor))))
(harness-log "METABOLISM CRASH [~a]: ~a" (or sensor :unknown) c)
;; Only rollback on critical errors, not standard tool or loop errors
;; Only rollback memory on critical errors, not transient tool failures
;; This prevents losing recent context due to a single bad API call
(unless (member sensor '(:loop-error :tool-error :syntax-error))
(harness-log "CRITICAL ERROR: Initiating Micro-Rollback.")
(rollback-memory 0))
;; At deep recursion or known error types, terminate gracefully
(if (or (> depth 2) (member sensor '(:loop-error :tool-error)))
(setf current-signal nil)
(setf current-signal (list :type :EVENT :depth (1+ depth) :meta meta
:payload (list :sensor :loop-error :message (format nil "~a" c) :depth depth)))))))))))
;; Otherwise, convert error to a loop-error signal for retry
(setf current-signal
(list :type :EVENT
:depth (1+ depth)
:meta meta
:payload (list :sensor :loop-error
:message (format nil "~a" c)
:depth depth)))))))))))
#+end_src
** Heartbeat Mechanism
The heartbeat ensures the agent remains "alive" even in the absence of external stimuli, allowing for latent reflection and periodic maintenance. The interval is externalized to the `HEARTBEAT_INTERVAL` environment variable.
** The Feedback Loop Explained
The pipeline implements a feedback loop where Act can return a new signal:
1. User input arrives → Perceive normalizes it
2. Reason generates an action → Act executes it
3. If the action was a tool call that returned new information → Act returns a feedback signal
4. Feedback signal feeds back into step 1 for further reasoning
This enables multi-step workflows where each action can trigger additional analysis.
* Heartbeat Mechanism
The heartbeat thread ensures the agent remains alive even without external input. It drives two critical functions:
1. **Latent reflection** - the agent can think without external prompting
2. **Periodic maintenance** - memory auto-save, orphan detection, etc.
** Heartbeat Configuration Variables
#+begin_src lisp :tangle ../library/loop.lisp
(defvar *auto-save-interval* 300
"Save memory to disk every N seconds. Set from MEMORY_AUTO_SAVE_INTERVAL env.")
"Interval in seconds between automatic memory saves.
Defaults to 300 seconds (5 minutes). Set via MEMORY_AUTO_SAVE_INTERVAL env var.")
(defvar *heartbeat-save-counter* 0
"Counter for auto-save triggers.")
"Tracks heartbeats since last save, used to calculate auto-save timing.")
#+end_src
** start-heartbeat: The Pulsing Heart
#+begin_src lisp :tangle ../library/loop.lisp
(defun start-heartbeat ()
"Starts the background heartbeat thread. Interval is loaded from HEARTBEAT_INTERVAL."
"Starts the background heartbeat thread.
The heartbeat runs in a dedicated thread to avoid blocking the main
signal processing loop. Each heartbeat:
1. Injects a :HEARTBEAT signal into the metabolic pipeline
2. Checks if memory should be auto-saved (based on interval ratio)
Configuration via environment:
- HEARTBEAT_INTERVAL: Seconds between heartbeats (default: 60)
- MEMORY_AUTO_SAVE_INTERVAL: Seconds between auto-saves (default: 300)"
(let ((interval (or (ignore-errors (parse-integer (uiop:getenv "HEARTBEAT_INTERVAL"))) 60))
(auto-save (or (ignore-errors (parse-integer (uiop:getenv "MEMORY_AUTO_SAVE_INTERVAL"))) *auto-save-interval*)))
(setf *auto-save-interval* auto-save)
(setf *heartbeat-save-counter* 0)
(setf *heartbeat-thread*
(bt:make-thread
(lambda ()
(loop
(sleep interval)
(setf *heartbeat-thread*
(bt:make-thread
(lambda ()
(loop
;; Wait for interval
(sleep interval)
;; Update counter and check if it's time to save
(incf *heartbeat-save-counter*)
(when (>= *heartbeat-save-counter* (/ *auto-save-interval* interval))
(setf *heartbeat-save-counter* 0)
(save-memory-to-disk))
;; inject-stimulus is synchronous for heartbeats, preventing accumulation.
(inject-stimulus (list :type :EVENT :payload (list :sensor :heartbeat :unix-time (get-universal-time))))))
;; Inject heartbeat signal - this runs through the full pipeline
;; allowing the agent to do latent reflection even with no input
(inject-stimulus
(list :type :EVENT
:payload (list :sensor :heartbeat
:unix-time (get-universal-time)))))
:name "opencortex-heartbeat"))))
#+end_src
** Main Entry Point
The `main` function initializes the environment, loads skills, and starts the heartbeat. It now includes a graceful shutdown handler for `SIGINT` (Ctrl+C) and uses `DAEMON_SLEEP_INTERVAL` to control its idle rhythm.
* Main Entry Point
** Shutdown Configuration
#+begin_src lisp :tangle ../library/loop.lisp
(defvar *shutdown-save-enabled* t
"If non-nil, save memory to disk on graceful shutdown.")
"When T, save memory to disk on graceful shutdown.
Disable for testing or when memory persistence is handled externally.")
#+end_src
** main: System Bootstrap and Idle Loop
The main function orchestrates system startup:
1. Load environment variables from ~/.local/share/opencortex/.env
2. Restore memory from previous snapshot (crash recovery)
3. Initialize actuators and load all skills
4. Start the heartbeat thread
5. Register SIGINT handler for graceful Ctrl+C shutdown
6. Enter idle loop (sleeping in 1-hour increments)
#+begin_src lisp :tangle ../library/loop.lisp
(defun main ()
"Entry point for the Skeleton MVP. Handles initialization and graceful shutdown."
(let* ((home (uiop:getenv "HOME"))
(env-file (uiop:merge-pathnames* ".local/share/opencortex/.env" (uiop:ensure-directory-pathname home))))
(when (uiop:file-exists-p env-file) (cl-dotenv:load-env env-file)))
"Entry point for OpenCortex. Initializes the system and enters idle loop.
;; Load memory from disk if a snapshot exists
Startup sequence:
1. Load environment from ~/.local/share/opencortex/.env
2. Restore memory from disk (if snapshot exists)
3. Initialize actuators (shell, cli, system)
4. Load all skills from SKILLS_DIR
5. Start heartbeat thread
6. Register SIGINT handler for graceful shutdown
7. Enter idle loop (sleeps in DAEMON_SLEEP_INTERVAL chunks)
The idle loop checks for interrupts and saves memory before exit."
;; Step 1: Load environment variables from standard location
(let* ((home (uiop:getenv "HOME"))
(env-file (uiop:merge-pathnames*
".local/share/opencortex/.env"
(uiop:ensure-directory-pathname home))))
(when (uiop:file-exists-p env-file)
(cl-dotenv:load-env env-file)))
;; Step 2: Crash recovery - load memory from previous snapshot
(load-memory-from-disk)
;; Step 3-4: Initialize actuators and load skills
(initialize-actuators)
(initialize-all-skills)
;; Step 5: Start the heartbeat
(start-heartbeat)
;; Graceful shutdown handler for SBCL
;; Step 6: Register graceful shutdown handler
;; SBCL-specific: catches Ctrl+C (SIGINT) and saves before exit
#+sbcl
(sb-sys:enable-interrupt sb-unix:sigint
(lambda (sig code scp)
(declare (ignore sig code scp))
(sb-sys:enable-interrupt sb-unix:sigint
(lambda (sig code scp)
(declare (ignore sig code scp))
(harness-log "SHUTDOWN: SIGINT received. Saving memory...")
(when *shutdown-save-enabled* (save-memory-to-disk))
(when *shutdown-save-enabled*
(save-memory-to-disk))
(uiop:quit 0)))
(let ((sleep-interval (or (ignore-errors (parse-integer (uiop:getenv "DAEMON_SLEEP_INTERVAL"))) 3600)))
(loop
;; Step 7: Idle loop - sleep in chunks, checking for interrupts
(let ((sleep-interval (or (ignore-errors
(parse-integer (uiop:getenv "DAEMON_SLEEP_INTERVAL")))
3600)))
(loop
;; Check for interrupt before each sleep cycle
(when (bt:with-lock-held (*interrupt-lock*) *interrupt-flag*)
(harness-log "SHUTDOWN: Interrupt flag set. Saving memory...")
(when *shutdown-save-enabled* (save-memory-to-disk))
(when *shutdown-save-enabled*
(save-memory-to-disk))
(return))
;; Sleep in configured intervals (default: 1 hour)
(sleep sleep-interval))))
#+end_src
#+end_src