passepartout/skills/org-skill-scribe.org

SKILL: Autonomous Scribe (Knowledge Distillation)

• Overview
  • Architectural Intent: Continuous Distillation
• Implementation
  • Package Initialization
  • State: Checkpoint Management
  • Filtration: Privacy and Relevance
  • Probabilistic Stage: Concept Extraction
  • Deterministic Stage: Knowledge Committal
  • Registration
  • Initialization

Overview

The Autonomous Scribe is the background architect of the Memex. Its primary responsibility is the "Nightly Distillation": a process that scans chronological daily logs, extracts evergreen concepts, and formalizes them into atomic Zettelkasten notes.

Architectural Intent: Continuous Distillation

The Scribe transforms the "Noise" of daily streams into the "Signal" of permanent knowledge. By operating in the background, it ensures that your knowledge graph grows autonomously, even when you aren't actively organizing it.

It utilizes a "Read-Reason-Write" pattern:

1. Read: Identifies new thoughts in the daily/ folder.
2. Reason: Uses the Probabilistic Engine to extract atomic, evergreen concepts.
3. Write: Commits the distilled notes to the notes/ folder with proper back-links.

Implementation

Package Initialization

(in-package :cl-user)
(defpackage :opencortex.skills.org-skill-scribe
  (:use :cl :opencortex))
(in-package :opencortex.skills.org-skill-scribe)

State: Checkpoint Management

The Scribe must be efficient. It tracks the last processed timestamp to avoid redundant distillation and LLM token waste.

(defvar *scribe-last-checkpoint* 0
  "The universal-time of the last successful distillation run.")

(defun scribe-load-state ()
  "Loads the scribe checkpoint from the state directory."
  (let ((state-file (merge-pathnames "system/state/scribe-checkpoint.lisp" 
                                     (asdf:system-source-directory :opencortex))))
    (if (uiop:file-exists-p state-file)
        (setf *scribe-last-checkpoint* (read-from-string (uiop:read-file-string state-file)))
        (setf *scribe-last-checkpoint* 0))))

(defun scribe-save-state ()
  "Saves the current universal-time as the new checkpoint."
  (let ((state-file (merge-pathnames "system/state/scribe-checkpoint.lisp" 
                                     (asdf:system-source-directory :opencortex))))
    (ensure-directories-exist state-file)
    (with-open-file (out state-file :direction :output :if-exists :supersede)
      (format out "~a" (get-universal-time)))))

Filtration: Privacy and Relevance

To protect user privacy, the Scribe strictly ignores any node tagged with @personal.

(defun scribe-get-distillable-nodes ()
  "Returns a list of org-objects from memory that require distillation."
  (let ((results nil))
    (maphash (lambda (id obj)
               (declare (ignore id))
               (let* ((attrs (org-object-attributes obj))
                      (tags (getf attrs :TAGS))
                      (type (org-object-type obj))
                      (version (org-object-version obj)))
                 (when (and (eq type :HEADLINE)
                            (> version *scribe-last-checkpoint*)
                            (not (member "@personal" tags :test #'string-equal)))
                   (push obj results))))
             *memory*)
    results))

Probabilistic Stage: Concept Extraction

This function generates the specific distillation prompt for the LLM. It focuses on atomicity and structured Lisp output.

(defun probabilistic-skill-scribe (context)
  "Generates the extraction prompt for the Scribe distillation task."
  (declare (ignore context))
  (let ((nodes (scribe-get-distillable-nodes)))
    (if nodes
        (let ((text-to-process ""))
          (dolist (node nodes)
            (setf text-to-process (concatenate 'string text-to-process 
                                               (format nil "ID: ~a~%TITLE: ~a~%CONTENT: ~a~%---~%" 
                                                       (org-object-id node)
                                                       (getf (org-object-attributes node) :TITLE)
                                                       (org-object-content node)))))
          (format nil "DISTILLATION TASK:
Below are raw chronological logs from my daily journal.
Extract ATOMIC EVERGREEN NOTES from this text.

RULES:
1. One note per distinct concept.
2. Output a list of Lisp plists: ((:title \"...\" :content \"...\" :source-id \"...\") ...)
3. Keep titles descriptive and snake_case.

TEXT:
~a" text-to-process))
        nil)))

Deterministic Stage: Knowledge Committal

The final physical step. It takes the LLM's structured proposal and writes it to the local filesystem.

(defun scribe-commit-notes (proposals)
  "Writes distilled notes to the MemexHardHard Hard drive."
  (let ((notes-dir (merge-pathnames "notes/" (asdf:system-source-directory :opencortex))))
    (ensure-directories-exist notes-dir)
    (dolist (note proposals)
      (let* ((title (getf note :title))
             (content (getf note :content))
             (source-id (getf note :source-id))
             (filename (format nil "~a.org" (string-downcase (cl-ppcre:regex-replace-all " " title "_"))))
             (path (merge-pathnames filename notes-dir)))
        (with-open-file (out path :direction :output :if-exists :supersede)
          (format out ":PROPERTIES:~%:ID: ~a~%:SOURCE_ID: ~a~%:END:~%#+TITLE: ~a~%~%~a" 
                  (org-id-new) source-id title content))
        (harness-log "SCRIBE: Distilled evergreen note ~a" filename)))))

(defun verify-skill-scribe (action context)
  "Main deterministic gate for Scribe distillation."
  (declare (ignore context))
  (let ((data (cond ((and (listp action) (eq (getf action :type) :REQUEST))
                     (getf (getf action :payload) :payload))
                    ((and (listp action) (not (member (getf action :type) '(:LOG :EVENT))))
                     action)
                    (t nil))))
    (when data
      (scribe-commit-notes data)
      (scribe-save-state)
      (list :type :LOG :payload (list :text "SCRIBE: Distillation cycle complete.")))) )

Registration

(defskill :skill-scribe
  :priority 50
  :trigger (lambda (ctx)
             (let* ((payload (getf ctx :payload))
                    (sensor (getf payload :sensor)))
               (and (eq sensor :heartbeat)
                    (> (- (get-universal-time) *scribe-last-checkpoint*) 3600)
                    (scribe-get-distillable-nodes))))
  :probabilistic #'probabilistic-skill-scribe
  :deterministic #'verify-skill-scribe)

Initialization

(scribe-load-state)