passepartout/skills/org-skill-scribe.org

SKILL: Autonomous Scribe (Knowledge Distillation)

• Overview
• Phase A: Demand (PRD)
  • 1. Purpose
  • 2. Success Criteria
• Phase B: Blueprint (PROTOCOL)
  • 1. Architectural Intent
  • 2. Semantic Interfaces
• Phase D: Build (Implementation)
  • Package Context
  • State: Checkpoint Management
  • Filtering: Privacy & Relevance
  • Probabilistic: Extraction Prompt
  • Deterministic: Note Committal
  • Skill Registration
  • Initialization

Overview

The Autonomous Scribe is the background architect of the Memex. It is responsible for the "Nightly Distillation": a process that scans chronological daily logs, extracts evergreen concepts, and formalizes them into atomic Zettelkasten notes.

Phase A: Demand (PRD)

1. Purpose

Automate the conversion of ephemeral, time-stamped thoughts into a permanent, structured knowledge graph.

2. Success Criteria

• Capture: Identify new headlines in the `daily/` directory that haven't been distilled yet.
• Privacy: Strictly ignore any node tagged with `@personal`.
• Extraction: Use neural reasoning to extract atomic concepts from raw logs.
• Formalization: Create new `.org` files in the `notes/` directory with proper Org-ID and back-links to the source.

Phase B: Blueprint (PROTOCOL)

1. Architectural Intent

The Scribe reacts to the `:heartbeat` sensor. It maintains a state file (`scribe-state.lisp`) to track the last processed timestamp. It performs a "Read-Reason-Write" loop:

1. Read: Scan `daily/*.org` for nodes updated after the last checkpoint.
2. Reason: Ask the LLM to "Extract atomic notes from this text".
3. Write: Commit the resulting nodes to the `notes/` directory.

2. Semantic Interfaces

• Trigger: `(:sensor :heartbeat)`
• Action: `(:type :REQUEST :target :system :action :create-note :title "…" :content "…" :source-id "…")`

Phase D: Build (Implementation)

Package Context

(in-package :opencortex)

State: Checkpoint Management

We track the last processed universal time to avoid redundant distillation.

(defvar *scribe-last-checkpoint* 0
  "The universal-time of the last successful distillation run.")

(defun scribe-load-state ()
  "Loads the scribe checkpoint from the state directory."
  (let ((state-file (uiop:merge-pathnames* "state/scribe-checkpoint.lisp" (asdf:system-source-directory :opencortex))))
    (if (uiop:file-exists-p state-file)
        (setf *scribe-last-checkpoint* (read-from-string (uiop:read-file-string state-file)))
        (setf *scribe-last-checkpoint* 0))))

(defun scribe-save-state ()
  "Saves the current universal-time as the new checkpoint."
  (let ((state-file (uiop:merge-pathnames* "state/scribe-checkpoint.lisp" (asdf:system-source-directory :opencortex))))
    (ensure-directories-exist state-file)
    (with-open-file (out state-file :direction :output :if-exists :supersede)
      (format out "~a" (get-universal-time)))))

Filtering: Privacy & Relevance

The Scribe only cares about non-personal, non-distilled headlines.

(defun scribe-get-distillable-nodes ()
  "Returns a list of org-objects from the daily/ folder that require distillation."
  (let ((results nil))
    (maphash (lambda (id obj)
               (declare (ignore id))
               (let* ((attrs (org-object-attributes obj))
                      (tags (getf attrs :TAGS))
                      (type (org-object-type obj))
                      (version (org-object-version obj)))
                 (when (and (eq type :HEADLINE)
                            (> version *scribe-last-checkpoint*)
                            (not (member "@personal" tags :test #'string-equal)))
                   (push obj results))))
             *memory*)
    results))

Probabilistic: Extraction Prompt

The LLM is tasked with identifying atomic concepts within the raw text.

(defun probabilistic-skill-scribe (context)
  "Generates the extraction prompt for the Scribe."
  (let* ((payload (getf context :payload))
         (nodes (scribe-get-distillable-nodes)))
    (if nodes
        (let ((text-to-process ""))
          (dolist (node nodes)
            (setf text-to-process (concatenate 'string text-to-process 
                                               (format nil "ID: ~a~%TITLE: ~a~%CONTENT: ~a~%---~%" 
                                                       (org-object-id node)
                                                       (getf (org-object-attributes node) :TITLE)
                                                       (org-object-content node)))))
          (format nil "DISTILLATION TASK:
Below are raw chronological logs from my daily journal.
Extract ATOMIC EVERGREEN NOTES from this text.

RULES:
1. One note per distinct concept.
2. Output a list of Lisp plists: ((:title \"...\" :content \"...\" :source-id \"...\") ...)
3. The content should be in Org-mode format.
4. Keep titles descriptive and snake_case.

TEXT:
~a" text-to-process))
        nil)))

Deterministic: Note Committal

The deterministic gate receives the list of proposed notes and writes them to the filesystem.

(defun scribe-commit-notes (proposals)
  "Writes proposed atomic notes to the notes/ directory. Appends if the note exists."
  (let ((notes-dir (uiop:merge-pathnames* "notes/" (asdf:system-source-directory :opencortex))))
    (ensure-directories-exist notes-dir)
    (dolist (note proposals)
      (let* ((title (getf note :title))
             (content (getf note :content))
             (source-id (getf note :source-id))
             (filename (format nil "~a.org" (string-downcase (cl-ppcre:regex-replace-all " " title "_"))))
             (path (merge-pathnames filename notes-dir)))
        (if (uiop:file-exists-p path)
            (with-open-file (out path :direction :output :if-exists :append)
              (format out "~%~%* Appended insight from ~a~%~a" source-id content))
            (with-open-file (out path :direction :output :if-exists :supersede)
              (format out ":PROPERTIES:~%:ID: ~a~%:SOURCE_ID: ~a~%:END:~%#+TITLE: ~a~%~%~a" 
                      (org-id-new) source-id title content)))
        (harness-log "SCRIBE: Processed evergreen note ~a" filename)))))

(defun verify-skill-scribe (action context)
  "Executes the note creation and marks source nodes as distilled."
  (declare (ignore context))
  (let ((data (cond ((and (listp action) (eq (getf action :type) :REQUEST))
                     (getf (getf action :payload) :payload))
                    ((and (listp action) (not (member (getf action :type) '(:LOG :EVENT))))
                     action)
                    (t nil))))
    (when data
      (harness-log "SCRIBE: Committing ~a atomic notes..." (length data))
      (scribe-commit-notes data)
      (scribe-save-state)
      (harness-log "SCRIBE: Distillation complete.")
      ;; Return a log event to stop the loop
      (list :type :LOG :payload (list :text "Distillation successful.")))))

Skill Registration

(defskill :skill-scribe
  :priority 50
  :trigger (lambda (ctx)
             (let* ((payload (getf ctx :payload))
                    (sensor (getf payload :sensor)))
               (and (eq sensor :heartbeat)
                    ;; Only run once per hour to check if we need to distill
                    (> (- (get-universal-time) *scribe-last-checkpoint*) 3600)
                    (scribe-get-distillable-nodes))))
  :probabilistic #'probabilistic-skill-scribe
  :deterministic #'verify-skill-scribe)

Initialization

(scribe-load-state)