passepartout/harness/memory.org

#+TITLE: Homoiconic Memory (memory.lisp)
#+AUTHOR: Amr
#+FILETAGS: :harness:memory:
#+STARTUP: content

* Homoiconic Memory (memory.lisp)

** Architectural Intent: The Live Graph
The Memory module is the "conscious mind" of the OpenCortex. Unlike traditional agents that rely on slow, external databases (SQL or Vector), OpenCortex maintains your entire Memex as a live, homoiconic graph of Lisp objects in RAM.

*** Why RAM-First?
1. **Zero-Latency Inference:** Traversing complex associations between notes and tasks occurs at native Lisp speeds, without the overhead of context-switching to a database driver.
2. **Unified Data Model:** Since the program (Lisp) and the data (the Memory) share the same structure, the agent can manipulate its own memory as naturally as it manipulates its own code.
3. **Graph Sovereignty:** By keeping the graph in-process, we ensure that the user's private knowledge base never leaves the host machine unless explicitly requested by a gateway.

** Pipeline Initialization
#+begin_src lisp :tangle ../library/memory.lisp
(in-package :opencortex)
#+end_src

* Core Data Structures

** The Object Registry
#+begin_src lisp :tangle ../library/memory.lisp
(defvar *memory* (make-hash-table :test 'equal)
  "The primary in-memory graph of all Org-mode entities, keyed by their unique ID.")
#+end_src

** The History Store (Merkle History)
OpenCortex maintains a history of memory states to allow for "Micro-Rollbacks" if a skill or tool execution results in an inconsistent state.

#+begin_src lisp :tangle ../library/memory.lisp
(defvar *history-store* (make-array 0 :fill-pointer 0 :adjustable t)
  "A versioned log of the memory state, allowing for temporal traversal and rollback.")
#+end_src

** The Org-Object Definition
Every headline, paragraph, or task in the Memex is represented as an ~org-object~.

#+begin_src lisp :tangle ../library/memory.lisp
(defstruct org-object
  "The fundamental unit of knowledge in the OpenCortex."
  id
  type
  attributes
  parent-id
  children
  version
  last-sync
  vector
  content
  hash)
#+end_src

* Integrity and Hashing

** Merkle Hashing (compute-merkle-hash)
To ensure data integrity and detect changes during external edits, we utilize Merkle-tree hashing. A node's hash is derived from its own content plus the hashes of its children.

#+begin_src lisp :tangle ../library/memory.lisp
(defun compute-merkle-hash (id type attributes content child-hashes)
  "Computes a SHA-256 Merkle hash for a node based on its core properties and children's hashes."
  (let* ((alist (loop for (k v) on attributes by #'cddr collect (cons k v)))
         (sorted-alist (sort alist #'string< :key (lambda (x) (format nil "~a" (car x)))))
         (attr-string (format nil "~s" sorted-alist))
         (children-string (format nil "~{~a~}" child-hashes))
         (raw-data (format nil "~a|~a|~a|~a|~a" id type attr-string (or content "") children-string)))
    (ironclad:byte-array-to-hex-string 
     (ironclad:digest-sequence :sha256 (ironclad:ascii-string-to-byte-array raw-data)))))
#+end_src

* Memory Ingestion

** AST Ingestion (ingest-ast)
The primary mechanism for translating raw Org-mode Abstract Syntax Trees (provided by Emacs or a parser) into the live Lisp graph.

#+begin_src lisp :tangle ../library/memory.lisp
(defun ingest-ast (ast &optional parent-id)
  "Recursively parses an Org AST into the Lisp Memory registry."
  (let* ((type (getf ast :type))
         (properties (getf ast :properties))
         (id (or (getf properties :ID) (uuid:make-v4-uuid)))
         (content (getf ast :content))
         (children (getf ast :contents))
         (child-ids nil))

    ;; Recursively ingest children and collect their IDs
    (dolist (child children)
      (let ((child-obj (ingest-ast child id)))
        (when child-obj (push (org-object-id child-obj) child-ids))))

    (let ((obj (make-org-object :id id
                                :type type
                                :attributes properties
                                :parent-id parent-id
                                :children (nreverse child-ids)
                                :content content
                                :version (get-universal-time))))
      (setf (gethash id *memory*) obj)
      obj)))
#+end_src

* Retrieval and Search

** Object Lookup (lookup-object)
#+begin_src lisp :tangle ../library/memory.lisp
(defun lookup-object (id)
  "Retrieves an object from memory by its ID."
  (gethash id *memory*))
#+end_src

** Semantic Attribute Search (list-objects-with-attribute)
Allows for querying the memory based on metadata (e.g., finding all nodes tagged :PROJECT:).

#+begin_src lisp :tangle ../library/memory.lisp
(defun list-objects-with-attribute (key value)
  "Returns a list of objects that possess the specified attribute pair."
  (let ((results nil))
    (maphash (lambda (id obj)
               (declare (ignore id))
               (when (equal (getf (org-object-attributes obj) key) value)
                 (push obj results)))
             *memory*)
    results))
#+end_src

* Persistence and Resilience

** Memory Snapshots (snapshot-memory)
Captures the current state of the memory graph.

#+begin_src lisp :tangle ../library/memory.lisp
(defun snapshot-memory ()
  "Creates a deep copy of the memory hash table and pushes it to the history store."
  (let ((new-snap (make-hash-table :test 'equal)))
    (maphash (lambda (k v) (setf (gethash k new-snap) (copy-org-object v))) *memory*)
    (vector-push-extend new-snap *history-store*)))
#+end_src

** Micro-Rollbacks (rollback-memory)
The primary defense against accidental memory corruption by faulty skills.

#+begin_src lisp :tangle ../library/memory.lisp
(defun rollback-memory (&optional (steps 1))
  "Restores the memory to a previous snapshot state."
  (let ((index (- (length *history-store*) steps 1)))
    (when (>= index 0)
      (setf *memory* (aref *history-store* index))
      (harness-log "IMMUNE SYSTEM: Memory rolled back ~a steps." steps))))
#+end_src