passepartout/skills/org-skill-homoiconic-memory.org

SKILL: Homoiconic Memory (Universal Literate Note)

• Overview
• Phase A: Demand (PRD)
  • 1. Purpose
  • 2. User Needs
• Phase B: Blueprint (PROTOCOL)
  • 1. Architectural Intent
  • 2. Semantic Interfaces
• Phase C: Success (QUALITY)
  • 1. Success Criteria
  • 2. TDD Plan
• Phase D: Build (Implementation)
  • Package Context
  • Node Structure Definition
  • ID Generation (org-id-new)
  • ID Injection (memory-ensure-id)
  • Recursive Normalization (memory-normalize-ast)
  • JSON Bridge: Org-to-JSON
  • JSON Bridge: JSON-to-Org
  • Registration
• Phase E: Chaos (Verification)
  • 1. Unit Tests (FiveAM)
  • 2. Chaos Scenarios
• Phase F: Memory (RCA)

Overview

The Homoiconic Memory skill defines the "Grammar of the Memex." It establishes Org-mode as the native Abstract Syntax Tree (AST) for both humans and agents. This skill consolidates the definition of Org nodes, the bidirectional JSON bridge, and the structural normalization logic (ID injection) into a single, high-integrity unit.

Phase A: Demand (PRD)

1. Purpose

Unify the structural rules and programmatic manipulation of the Org-mode AST.

2. User Needs

• Structural Integrity: Mandatory unique IDs and metadata drawers for all headlines.
• Bidirectional Bridge: Lossless conversion between Org-mode text and structured JSON AST.
• Atomic Refactoring: Pure functions for creating, updating, and deleting nodes.
• Normalization: Automatic injection of IDs during ingest or save.

Phase B: Blueprint (PROTOCOL)

1. Architectural Intent

The memory suite uses a "Functional Core" for AST manipulation. Every transformation (normalization, refactoring) returns a new AST version, which is then persisted to the Memory.

2. Semantic Interfaces

(defun memory-org-to-json (source)
  "Converts Org-mode source to JSON AST.")

(defun memory-json-to-org (ast)
  "Converts JSON AST back to Org-mode text.")

(defun memory-normalize-ast (ast)
  "Recursively ensures ID uniqueness across the AST.")

Phase C: Success (QUALITY)

1. Success Criteria

• Round-trip Fidelity: Org -> JSON -> Org must result in identical text (modulo normalization).
• ID Uniqueness: No two headlines may share an ID after normalization.
• Merkle Integration: AST modifications must trigger Memory snapshots.

2. TDD Plan

Tests in `tests/memory-suite-tests.lisp` will verify the round-trip conversion and the recursive ID injection logic.

Phase D: Build (Implementation)

Package Context

Node Structure Definition

We define the standard `org-node` structure used throughout the harness.

(defun make-memory-node (headline &key content properties children)
  "Constructor for a normalized Org node alist."
  (list :type :HEADLINE
        :properties (or properties nil)
        :content content
        :contents children))

ID Generation (org-id-new)

Mandated standard for ID creation. This function ensures that every node in the Memex has a unique, deterministic identifier.

(defun org-id-new ()
  "Generates a new unique ID for an Org node. This is the system-wide standard."
  (format nil "node-~a" (get-universal-time)))

ID Injection (memory-ensure-id)

Ensures every headline has a unique ID property using the system standard `org-id-new`. This is foundational for the Merkle-Tree object store.

(defun memory-ensure-id (node)
  "Injects a unique ID into an Org node if missing, using the standard org-id-new mechanism."
  (let* ((props (getf node :properties))
         (id (getf props :ID)))
    (if (and id (not (equal id "")))
        node
        (let ((new-id (opencortex:org-id-new)))
          (setf (getf node :properties) (append props (list :ID new-id)))
          (harness-log "MEMORY - Injected standard ID ~a" new-id)
          node))))

Recursive Normalization (memory-normalize-ast)

Recursively walks the AST to enforce structural rules.

(defun memory-normalize-ast (ast)
  "Recursively normalizes an Org AST."
  (let ((type (getf ast :type))
        (contents (getf ast :contents)))
    (when (eq type :HEADLINE)
      (setf ast (memory-ensure-id ast)))
    (when contents
      (setf (getf ast :contents)
            (mapcar (lambda (child)
                      (if (listp child)
                          (memory-normalize-ast child)
                          child))
                    contents)))
    ast))

JSON Bridge: Org-to-JSON

Utilizes the Emacs bridge (or local parser) to convert text to JSON.

(defun memory-org-to-json (source-path)
  "Routes to the Emacs-based Org-JSON bridge."
  ;; Future implementation will use the org-json-convert CLI tool
  (harness-log "MEMORY - Parsing ~a to JSON..." source-path)
  nil)

JSON Bridge: JSON-to-Org

Converts a structured AST back into Org-mode text.

(defun memory-json-to-org (ast)
  "Materializes a JSON AST into Org-mode text."
  ;; Placeholder for org-element-interpret-data equivalent
  (harness-log "MEMORY - Rendering AST to text...")
  "")

Registration

(progn
  (defskill :skill-homoiconic-memory
    :priority 300 ; Core foundational skill
    :trigger (lambda (ctx) (member (getf (getf ctx :payload) :sensor) '(:buffer-save :ingest)))
    :probabilistic nil
    :deterministic (lambda (action ctx)
                (let ((ast (getf (getf ctx :payload) :ast)))
                  (when ast (memory-normalize-ast ast))
                  action))))

Phase E: Chaos (Verification)

1. Unit Tests (FiveAM)

(defpackage :opencortex-memory-tests
  #| #| (:use :cl :fiveam :opencortex)) |# |#
(in-package :opencortex-memory-tests)

#| (def-suite memory-suite :description "Tests for Homoiconic Memory.")
(comment (in-suite memory-suite)

(test test-id-injection
  (let* ((node (list :type :HEADLINE :properties nil))
         (normalized (opencortex::memory-ensure-id node)))
    (is (not (null (getf (getf normalized :properties) :ID))))))

2. Chaos Scenarios

• Scenario A (Duplicate IDs): Intentionally inject two nodes with the same ID and verify the normalizer detects the collision and re-generates one.
• Scenario B (Broken AST): Pass a malformed list to `memory-normalize-ast` and verify it fails gracefully with a log entry rather than crashing the harness.

Phase F: Memory (RCA)

• [2026-04-09 Thu]: Consolidated `org-mode`, `org-json-bridge`, and `ast-normalization` into this single skill. Standardized the recursive normalization path.