passepartout/literate/memory.org

Homoiconic Memory (memory.lisp)

• Homoiconic Memory (memory.lisp)
  • Architectural Intent: The Live Graph
    • Why RAM-First?
  • Pipeline Initialization
• Core Data Structures
  • The Object Registry
  • The History Store (Merkle History)
  • The Org-Object Definition
• Integrity and Hashing
  • Merkle Hashing (compute-merkle-hash)
• Memory Ingestion
  • AST Ingestion (ingest-ast)
• Retrieval and Search
  • Object Lookup (lookup-object)
  • Semantic Attribute Search (list-objects-with-attribute)
• Persistence and Resilience
  • Memory Snapshots (snapshot-memory)
  • Micro-Rollbacks (rollback-memory)

Homoiconic Memory (memory.lisp)

Architectural Intent: The Live Graph

The Memory module is the "conscious mind" of the OpenCortex. Unlike traditional agents that rely on slow, external databases (SQL or Vector), OpenCortex maintains your entire Memex as a live, homoiconic graph of Lisp objects in RAM.

Why RAM-First?

1. Zero-Latency Inference: Traversing complex associations between notes and tasks occurs at native Lisp speeds, without the overhead of context-switching to a database driver.
2. Unified Data Model: Since the program (Lisp) and the data (the Memory) share the same structure, the agent can manipulate its own memory as naturally as it manipulates its own code.
3. Graph Sovereignty: By keeping the graph in-process, we ensure that the user's private knowledge base never leaves the host machine unless explicitly requested by a gateway.

Pipeline Initialization

(in-package :opencortex)

Core Data Structures

The Object Registry

(defvar *memory* (make-hash-table :test 'equal)
  "The primary in-memory graph of all Org-mode entities, keyed by their unique ID.")

The History Store (Merkle History)

OpenCortex maintains a history of memory states to allow for "Micro-Rollbacks" if a skill or tool execution results in an inconsistent state.

(defvar *history-store* (make-array 0 :fill-pointer 0 :adjustable t)
  "A versioned log of the memory state, allowing for temporal traversal and rollback.")

The Org-Object Definition

Every headline, paragraph, or task in the Memex is represented as an org-object.

(defstruct org-object
  "The fundamental unit of knowledge in the OpenCortex."
  id
  type
  attributes
  parent-id
  children
  version
  last-sync
  vector
  content
  hash)

Integrity and Hashing

Merkle Hashing (compute-merkle-hash)

To ensure data integrity and detect changes during external edits, we utilize Merkle-tree hashing. A node's hash is derived from its own content plus the hashes of its children.

(defun compute-merkle-hash (id type attributes content child-hashes)
  "Computes a SHA-256 Merkle hash for a node based on its core properties and children's hashes."
  (let* ((alist (loop for (k v) on attributes by #'cddr collect (cons k v)))
         (sorted-alist (sort alist #'string< :key (lambda (x) (format nil "~a" (car x)))))
         (attr-string (format nil "~s" sorted-alist))
         (children-string (format nil "~{~a~}" child-hashes))
         (raw-data (format nil "~a|~a|~a|~a|~a" id type attr-string (or content "") children-string)))
    (ironclad:byte-array-to-hex-string 
     (ironclad:digest-sequence :sha256 (ironclad:ascii-string-to-byte-array raw-data)))))

Memory Ingestion

AST Ingestion (ingest-ast)

The primary mechanism for translating raw Org-mode Abstract Syntax Trees (provided by Emacs or a parser) into the live Lisp graph.

(defun ingest-ast (ast &optional parent-id)
  "Recursively parses an Org AST into the Lisp Memory registry."
  (let* ((type (getf ast :type))
         (properties (getf ast :properties))
         (id (or (getf properties :ID) (uuid:make-v4-uuid)))
         (content (getf ast :content))
         (children (getf ast :contents))
         (child-ids nil))

    ;; Recursively ingest children and collect their IDs
    (dolist (child children)
      (let ((child-obj (ingest-ast child id)))
        (when child-obj (push (org-object-id child-obj) child-ids))))

    (let ((obj (make-org-object :id id
                                :type type
                                :attributes properties
                                :parent-id parent-id
                                :children (nreverse child-ids)
                                :content content
                                :version (get-universal-time))))
      (setf (gethash id *memory*) obj)
      obj)))

Retrieval and Search

Object Lookup (lookup-object)

(defun lookup-object (id)
  "Retrieves an object from memory by its ID."
  (gethash id *memory*))

Semantic Attribute Search (list-objects-with-attribute)

Allows for querying the memory based on metadata (e.g., finding all nodes tagged :PROJECT:).

(defun list-objects-with-attribute (key value)
  "Returns a list of objects that possess the specified attribute pair."
  (let ((results nil))
    (maphash (lambda (id obj)
               (declare (ignore id))
               (when (equal (getf (org-object-attributes obj) key) value)
                 (push obj results)))
             *memory*)
    results))

Persistence and Resilience

Memory Snapshots (snapshot-memory)

Captures the current state of the memory graph.

(defun snapshot-memory ()
  "Creates a deep copy of the memory hash table and pushes it to the history store."
  (let ((new-snap (make-hash-table :test 'equal)))
    (maphash (lambda (k v) (setf (gethash k new-snap) (copy-org-object v))) *memory*)
    (vector-push-extend new-snap *history-store*)))

Micro-Rollbacks (rollback-memory)

The primary defense against accidental memory corruption by faulty skills.

(defun rollback-memory (&optional (steps 1))
  "Restores the memory to a previous snapshot state."
  (let ((index (- (length *history-store*) steps 1)))
    (when (>= index 0)
      (setf *memory* (aref *history-store* index))
      (harness-log "IMMUNE SYSTEM: Memory rolled back ~a steps." steps))))