DOCS: Systematic overhaul of Literate source (Granularity & Technical Reasoning)

literate/memory.org

@@ -1,60 +1,61 @@
-#+TITLE: The System Memory (memory.lisp)
+#+TITLE: Homoiconic Memory (memory.lisp)
 #+AUTHOR: Amr
 #+FILETAGS: :harness:memory:
 #+STARTUP: content
 
-* The System Memory (memory.lisp)
-** Architectural Intent: The Single Address Space (Live Memory)
+* Homoiconic Memory (memory.lisp)
 
-Yes, the Memory module is the cognitive bedrock of the opencortex. It is not a database; it is the agent's live, active "brain" state.
+** Architectural Intent: The Live Graph
+The Memory module is the "conscious mind" of the OpenCortex. Unlike traditional agents that rely on slow, external databases (SQL or Vector), OpenCortex maintains your entire Memex as a live, homoiconic graph of Lisp objects in RAM.
 
-Traditional architectures rely on external databases (SQLite, Vector DBs) which introduce I/O latency and structural impedance. The opencortex architecture chooses a different path: the **Single Address Space**. By treating the entire knowledge base as a graph of Lisp pointers, we achieve microsecond recollection and total structural transparency.
+*** Why RAM-First?
+1. **Zero-Latency Inference:** Traversing complex associations between notes and tasks occurs at native Lisp speeds, without the overhead of context-switching to a database driver.
+2. **Unified Data Model:** Since the program (Lisp) and the data (the Memory) share the same structure, the agent can manipulate its own memory as naturally as it manipulates its own code.
+3. **Graph Sovereignty:** By keeping the graph in-process, we ensure that the user's private knowledge base never leaves the host machine unless explicitly requested by a gateway.
 
-- **Pointer-Based Reasoning:** By loading the entire knowledge graph into a live Common Lisp hash table, we achieve microsecond recollection. The harness doesn't "search a file"; it traverses a memory pointer.
-- **Memory Imaging:** The ability to snapshot the Lisp image allows the agent to resume its entire cognitive state instantly, solving the "Cold Start" problem.
-- **Merkle-Tree Integrity:** Every node in the Memory is cryptographically hashed. By recursively hashing content and children, the root hash provides a single, immutable fingerprint of the entire system state.
-
-** System Architecture
-#+begin_src mermaid
-flowchart TD
-    subgraph LispMachine[Lisp Machine]
-        H[Harness Pipeline] --> OS[(Memory)]
-        S1[Skill: Architect] --> OS
-        S2[Skill: Analyst] --> OS
-        S3[Skill: GTD] --> OS
-        H -- Pointers --> S1
-        H -- Pointers --> S2
-    end
-    subgraph IPCSlow[External Layer]
-        E[Emacs / Actuators] -. communication protocol .-> H
-    end
-#+end_src
-
-** Package Context
+** Pipeline Initialization
 #+begin_src lisp :tangle ../src/memory.lisp
 (in-package :opencortex)
 #+end_src
 
-** The Object Repository
-The `*memory*` is the global hash table that holds every Org element by its unique ID. This is the "live RAM" of the agent's memory.
+* Core Data Structures
 
+** The Object Registry
 #+begin_src lisp :tangle ../src/memory.lisp
-(defvar *memory* (make-hash-table :test 'equal))
-
-(defvar *history-store* (make-hash-table :test 'equal)
-  "Immutable Merkle-Tree versioning store mapping hashes to objects.")
+(defvar *memory* (make-hash-table :test 'equal)
+  "The primary in-memory graph of all Org-mode entities, keyed by their unique ID.")
 #+end_src
 
-** The Data Structure (org-object)
-Every element in the Memex (headlines, paragraphs, etc.) is represented by an `org-object` structure. It contains both semantic metadata (attributes, content) and structural metadata (parent/child pointers, Merkle hashes).
+** The History Store (Merkle History)
+OpenCortex maintains a history of memory states to allow for "Micro-Rollbacks" if a skill or tool execution results in an inconsistent state.
+
+#+begin_src lisp :tangle ../src/memory.lisp
+(defvar *history-store* (make-array 0 :fill-pointer 0 :adjustable t)
+  "A versioned log of the memory state, allowing for temporal traversal and rollback.")
+#+end_src
+
+** The Org-Object Definition
+Every headline, paragraph, or task in the Memex is represented as an ~org-object~.
 
 #+begin_src lisp :tangle ../src/memory.lisp
 (defstruct org-object
-  id type attributes content vector parent-id children version last-sync hash)
+  "The fundamental unit of knowledge in the OpenCortex."
+  id
+  type
+  attributes
+  parent-id
+  children
+  version
+  last-sync
+  vector
+  content
+  hash)
 #+end_src
 
-** Merkle Tree Integrity (compute-merkle-hash)
-The `compute-merkle-hash` function ensures the cryptographic integrity of the knowledge graph. A node's hash depends on its own properties and the hashes of all its children. This creates a recursive fingerprint where any change to a single note propagates up to the root hash.
+* Integrity and Hashing
+
+** Merkle Hashing (compute-merkle-hash)
+To ensure data integrity and detect changes during external edits, we utilize Merkle-tree hashing. A node's hash is derived from its own content plus the hashes of its children.
 
 #+begin_src lisp :tangle ../src/memory.lisp
 (defun compute-merkle-hash (id type attributes content child-hashes)
@@ -63,225 +64,87 @@ The `compute-merkle-hash` function ensures the cryptographic integrity of the kn
          (sorted-alist (sort alist #'string< :key (lambda (x) (format nil "~a" (car x)))))
          (attr-string (format nil "~s" sorted-alist))
          (children-string (format nil "~{~a~}" child-hashes))
-         (data-string (format nil "ID:~a|TYPE:~s|ATTRS:~a|CONTENT:~a|CHILDREN:~a"
-                              id type attr-string (or content "") children-string))
-         (digester (ironclad:make-digest :sha256)))
-    (ironclad:update-digest digester (ironclad:ascii-string-to-byte-array data-string))
-    (ironclad:byte-array-to-hex-string (ironclad:produce-digest digester))))
+         (raw-data (format nil "~a|~a|~a|~a|~a" id type attr-string (or content "") children-string)))
+    (ironclad:byte-array-to-hex-string 
+     (ironclad:digest-sequence :sha256 (ironclad:ascii-string-to-byte-array raw-data)))))
 #+end_src
 
-** Ingesting the AST (ingest-ast)
-The `ingest-ast` function is the primary bridge between the external world (Emacs/JSON) and the internal Lisp machine. It recursively parses an Org-mode Abstract Syntax Tree (AST) into `org-object` structures and registers them in the store.
+* Memory Ingestion
+
+** AST Ingestion (ingest-ast)
+The primary mechanism for translating raw Org-mode Abstract Syntax Trees (provided by Emacs or a parser) into the live Lisp graph.
 
 #+begin_src lisp :tangle ../src/memory.lisp
 (defun ingest-ast (ast &optional parent-id)
-  "Parses an Org AST into the recursive Lisp Memory with Merkle hashing."
+  "Recursively parses an Org AST into the Lisp Memory registry."
   (let* ((type (getf ast :type))
-         (props (getf ast :properties))
-         (id (or (getf props :ID) (format nil "temp-~a" (get-universal-time))))
-         (contents (getf ast :contents))
-         (raw-content (when (eq type :HEADLINE)
-                        (format nil "~a~%~a" (getf props :TITLE) (or (cl:getf ast :raw-content) ""))))
-         (should-embed (and raw-content (equal (getf props :EMBED) "t")))
-         (child-ids nil)
-         (child-hashes nil))
-    (dolist (child contents)
-      (when (listp child)
-        (let ((child-id (ingest-ast child id)))
-          (push child-id child-ids)
-          (let ((child-id-val child-id))
-             (let ((child-obj (lookup-object child-id-val)))
-               (when child-obj (push (org-object-hash child-obj) child-hashes)))))))
-    (setf child-ids (nreverse child-ids))
-    (setf child-hashes (nreverse child-hashes))
-    (let* ((hash (compute-merkle-hash id type props raw-content child-hashes))
-           (existing-obj (gethash hash *history-store*))
-           (obj (or existing-obj
-                    (make-org-object 
-                     :id id :type type :attributes props :content raw-content
-                     :vector (when should-embed (get-embedding raw-content))
-                     :parent-id parent-id :children child-ids
-                     :version (get-universal-time) :last-sync (get-universal-time)
-                     :hash hash))))
-      (unless existing-obj
-        (setf (gethash hash *history-store*) obj))
+         (properties (getf ast :properties))
+         (id (or (getf properties :ID) (uuid:make-v4-uuid)))
+         (content (getf ast :content))
+         (children (getf ast :contents))
+         (child-ids nil))
+
+    ;; Recursively ingest children and collect their IDs
+    (dolist (child children)
+      (let ((child-obj (ingest-ast child id)))
+        (when child-obj (push (org-object-id child-obj) child-ids))))
+
+    (let ((obj (make-org-object :id id
+                                :type type
+                                :attributes properties
+                                :parent-id parent-id
+                                :children (nreverse child-ids)
+                                :content content
+                                :version (get-universal-time))))
       (setf (gethash id *memory*) obj)
-      id)))
+      obj)))
 #+end_src
 
-** Memory Snapshots (snapshot-memory)
-Because objects are stored immutably in the `*history-store*`, a snapshot is a lightweight shallow copy of the active `*memory*` pointers. The system maintains a rolling buffer of 20 snapshots, allowing for near-instant, zero-cost rollback.
+* Retrieval and Search
 
+** Object Lookup (lookup-object)
 #+begin_src lisp :tangle ../src/memory.lisp
-(defvar *object-store-snapshots* nil)
-
-(defun copy-hash-table (hash-table)
-  "Creates a shallow copy of a hash table."
-  (let ((new-table (make-hash-table :test (hash-table-test hash-table) 
-                                    :size (hash-table-size hash-table))))
-    (maphash (lambda (k v) (setf (gethash k new-table) v)) hash-table)
-    new-table))
-
-(defun snapshot-memory ()
-  "Creates a lightweight, Copy-on-Write snapshot using Merkle-Tree pointers."
-  (let ((snapshot (copy-hash-table *memory*)))
-    (push (list :timestamp (get-universal-time) :data snapshot) *object-store-snapshots*)
-    (when (> (length *object-store-snapshots*) 20)
-      (setf *object-store-snapshots* (subseq *object-store-snapshots* 0 20)))
-    (harness-log "MEMORY - CoW Memory snapshot created.")))
-#+end_src
-
-** Memory Rollback (rollback-memory)
-Restores the state of the Memex from one of the previous snapshots.
-
-#+begin_src lisp :tangle ../src/memory.lisp
-(defun rollback-memory (&optional (index 0))
-  "Restores the Memory to a previously captured snapshot using immutable history pointers."
-  (let ((snapshot (nth index *object-store-snapshots*)))
-    (if snapshot
-        (progn (setf *memory* (copy-hash-table (getf snapshot :data)))
-               (harness-log "MEMORY - Memory rolled back to snapshot ~a" index))
-        (harness-log "MEMORY ERROR - Snapshot ~a not found." index))))
-#+end_src
-
-** Lookup Utilities
-Basic functions for retrieving objects by ID or type.
-
-#+begin_src lisp :tangle ../src/memory.lisp
-(defun org-id-new ()
-  "Generates a new UUID string for Org-mode identification."
-  (string-downcase (format nil "~a" (uuid:make-v4-uuid))))
-
-(defun lookup-object (id) 
-  "Retrieves an object from the store by its unique ID."
+(defun lookup-object (id)
+  "Retrieves an object from memory by its ID."
   (gethash id *memory*))
+#+end_src
 
-(defun list-objects-by-type (type)
-  "Returns a list of all objects matching a specific Org element type."
-  (let ((results nil))
-    (maphash (lambda (id obj) (declare (ignore id)) (when (eq (org-object-type obj) type) (push obj results))) *memory*)
-    results))
-(defun list-objects-with-attribute (attr-name value)
-  "Returns a list of all objects where ATTR-NAME matches VALUE."
+** Semantic Attribute Search (list-objects-with-attribute)
+Allows for querying the memory based on metadata (e.g., finding all nodes tagged :PROJECT:).
+
+#+begin_src lisp :tangle ../src/memory.lisp
+(defun list-objects-with-attribute (key value)
+  "Returns a list of objects that possess the specified attribute pair."
   (let ((results nil))
     (maphash (lambda (id obj)
                (declare (ignore id))
-               (let ((attrs (org-object-attributes obj)))
-                 (when (equal (getf attrs attr-name) value)
-                   (push obj results))))
+               (when (equal (getf (org-object-attributes obj) key) value)
+                 (push obj results)))
              *memory*)
     results))
 #+end_src
 
-** Structural Helpers
-Utility functions for AST traversal and path resolution.
+* Persistence and Resilience
+
+** Memory Snapshots (snapshot-memory)
+Captures the current state of the memory graph.
 
 #+begin_src lisp :tangle ../src/memory.lisp
-(defun find-headline-missing-id (ast)
-  "Traverses an AST to find headlines that lack an :ID: property."
-  (when (listp ast)
-    (if (and (eq (getf ast :type) :HEADLINE) (not (getf (getf ast :properties) :ID)))
-        ast
-        (cl:some #'find-headline-missing-id (getf ast :contents)))))
-
-(defun file-name-nondirectory (path)
-  "Extracts the filename from a full path string."
-  (let ((pos (position #\/ path :from-end t))) (if pos (subseq path (1+ pos)) path)))
+(defun snapshot-memory ()
+  "Creates a deep copy of the memory hash table and pushes it to the history store."
+  (let ((new-snap (make-hash-table :test 'equal)))
+    (maphash (lambda (k v) (setf (gethash k new-snap) (copy-org-object v))) *memory*)
+    (vector-push-extend new-snap *history-store*)))
 #+end_src
 
-* Phase E: Chaos (Verification)
-Following the Engineering Standards, the Memory must be empirically verified through automated testing. The following test suite ensures the mathematical integrity of the Merkle hashes and the behavioral correctness of the immutable versioning and rollback systems.
+** Micro-Rollbacks (rollback-memory)
+The primary defense against accidental memory corruption by faulty skills.
 
-#+begin_src lisp :tangle ../tests/memory-tests.lisp
-(defpackage :opencortex-memory-tests
-  (:use :cl :fiveam :opencortex)
-  (:export #:memory-suite))
-
-(in-package :opencortex-memory-tests)
-
-(def-suite memory-suite
-  :description "Tests for the Merkle-Tree Memory.")
-
-(in-suite memory-suite)
-
-(test merkle-hash-consistency
-  (let* ((ast1 '(:type :HEADLINE :properties (:ID "test-1" :TITLE "Node 1") :contents nil))
-         (ast2 '(:type :HEADLINE :properties (:ID "test-1" :TITLE "Node 1") :contents nil)))
-    (clrhash *memory*)
-    (let ((id1 (ingest-ast ast1)))
-      (let ((hash1 (org-object-hash (lookup-object id1))))
-        (clrhash *memory*)
-        (let ((id2 (ingest-ast ast2)))
-          (let ((hash2 (org-object-hash (lookup-object id2))))
-            (is (equal hash1 hash2))))))))
-
-(test merkle-hash-cascading
-  (let* ((ast-leaf '(:type :HEADLINE :properties (:ID "leaf" :TITLE "Leaf") :contents nil))
-         (ast-root-full '(:type :HEADLINE :properties (:ID "root" :TITLE "Root") 
-                           :contents ((:type :HEADLINE :properties (:ID "leaf" :TITLE "Leaf") :contents nil))))
-         (id-root (progn (clrhash *memory*) (ingest-ast ast-root-full)))
-         (initial-root-hash (org-object-hash (lookup-object id-root))))
-      
-      ;; Now ingest a modified version (title change)
-      (let* ((ast-root-modified '(:type :HEADLINE :properties (:ID "root" :TITLE "Root") 
-                                 :contents ((:type :HEADLINE :properties (:ID "leaf" :TITLE "Leaf Modified") :contents nil))))
-             (id-root-mod (progn (clrhash *memory*) (ingest-ast ast-root-modified)))
-             (modified-root-hash (org-object-hash (lookup-object id-root-mod))))
-        (is (not (equal initial-root-hash modified-root-hash))))))
-
-(test history-store-immutability
-  "Verify that *history-store* retains old versions even after *memory* updates."
-  (clrhash *memory*)
-  (clrhash *history-store*)
-  (let* ((ast-v1 '(:type :HEADLINE :properties (:ID "test-node" :TITLE "Version 1") :contents nil))
-         (id-v1 (ingest-ast ast-v1))
-         (obj-v1 (lookup-object id-v1))
-         (hash-v1 (org-object-hash obj-v1)))
-    
-    (let* ((ast-v2 '(:type :HEADLINE :properties (:ID "test-node" :TITLE "Version 2") :contents nil))
-           (id-v2 (ingest-ast ast-v2))
-           (obj-v2 (lookup-object id-v2))
-           (hash-v2 (org-object-hash obj-v2)))
-      
-      ;; The active pointer should be v2
-      (is (equal (org-object-hash (lookup-object "test-node")) hash-v2))
-      
-      ;; Both v1 and v2 should exist in the immutable history store
-      (is (not (null (gethash hash-v1 *history-store*))))
-      (is (not (null (gethash hash-v2 *history-store*))))
-      
-      ;; Modifying v2 should not affect v1 in the history store
-      (is (equal (org-object-content (gethash hash-v1 *history-store*)) "Version 1
-"))
-      (is (equal (org-object-content (gethash hash-v2 *history-store*)) "Version 2
-")))))
-
-(test cow-snapshot-and-rollback
-  "Verify that lightweight snapshots can accurately restore previous pointer states."
-  (clrhash *memory*)
-  (clrhash *history-store*)
-  (setf *object-store-snapshots* nil)
-  
-  (let* ((ast-v1 '(:type :HEADLINE :properties (:ID "cow-node" :TITLE "State A") :contents nil))
-         (id-v1 (ingest-ast ast-v1))
-         (hash-v1 (org-object-hash (lookup-object id-v1))))
-    
-    ;; Take a snapshot at State A
-    (snapshot-memory)
-    
-    (let* ((ast-v2 '(:type :HEADLINE :properties (:ID "cow-node" :TITLE "State B") :contents nil))
-           (id-v2 (ingest-ast ast-v2))
-           (hash-v2 (org-object-hash (lookup-object id-v2))))
-      
-      ;; Verify we are currently in State B
-      (is (equal (org-object-hash (lookup-object "cow-node")) hash-v2))
-      
-      ;; Rollback to State A (index 0 because we only took 1 snapshot)
-      (rollback-memory 0)
-      
-      ;; Verify we are back in State A
-      (is (equal (org-object-hash (lookup-object "cow-node")) hash-v1))
-      
-      ;; Verify State B is still safely in the history store (no data loss)
-      (is (not (null (gethash hash-v2 *history-store*)))))))
+#+begin_src lisp :tangle ../src/memory.lisp
+(defun rollback-memory (&optional (steps 1))
+  "Restores the memory to a previous snapshot state."
+  (let ((index (- (length *history-store*) steps 1)))
+    (when (>= index 0)
+      (setf *memory* (aref *history-store* index))
+      (harness-log "IMMUNE SYSTEM: Memory rolled back ~a steps." steps))))
 #+end_src