DOCS: Systematic overhaul of Literate source (Granularity & Technical Reasoning)

skills/org-skill-scribe.org

@@ -1,78 +1,63 @@
-:PROPERTIES:
-:ID:       scribe-skill
-:CREATED:  [2026-04-13 Mon 18:40]
-:END:
 #+TITLE: SKILL: Autonomous Scribe (Knowledge Distillation)
-#+STARTUP: content
+#+AUTHOR: Amr
 #+FILETAGS: :scribe:distillation:memex:autonomy:
+#+STARTUP: content
 
 * Overview
-The *Autonomous Scribe* is the background architect of the Memex. It is responsible for the "Nightly Distillation": a process that scans chronological daily logs, extracts evergreen concepts, and formalizes them into atomic Zettelkasten notes.
+The *Autonomous Scribe* is the background architect of the Memex. Its primary responsibility is the "Nightly Distillation": a process that scans chronological daily logs, extracts evergreen concepts, and formalizes them into atomic Zettelkasten notes.
 
-* Phase A: Demand (PRD)
-:PROPERTIES:
-:STATUS: SIGNED
-:END:
+** Architectural Intent: Continuous Distillation
+The Scribe transforms the "Noise" of daily streams into the "Signal" of permanent knowledge. By operating in the background, it ensures that your knowledge graph grows autonomously, even when you aren't actively organizing it.
 
-** 1. Purpose
-Automate the conversion of ephemeral, time-stamped thoughts into a permanent, structured knowledge graph.
+It utilizes a "Read-Reason-Write" pattern:
+1. **Read:** Identifies new thoughts in the ~daily/~ folder.
+2. **Reason:** Uses the Probabilistic Engine to extract atomic, evergreen concepts.
+3. **Write:** Commits the distilled notes to the ~notes/~ folder with proper back-links.
 
-** 2. Success Criteria
-- [ ] *Capture:* Identify new headlines in the `daily/` directory that haven't been distilled yet.
-- [ ] *Privacy:* Strictly ignore any node tagged with `@personal`.
-- [ ] *Extraction:* Use neural reasoning to extract atomic concepts from raw logs.
-- [ ] *Formalization:* Create new `.org` files in the `notes/` directory with proper Org-ID and back-links to the source.
+* Implementation
 
-* Phase B: Blueprint (PROTOCOL)
-:PROPERTIES:
-:STATUS: SIGNED
-:END:
-
-** 1. Architectural Intent
-The Scribe reacts to the `:heartbeat` sensor. It maintains a state file (`scribe-state.lisp`) to track the last processed timestamp. It performs a "Read-Reason-Write" loop:
-1. **Read:** Scan `daily/*.org` for nodes updated after the last checkpoint.
-2. **Reason:** Ask the LLM to "Extract atomic notes from this text".
-3. **Write:** Commit the resulting nodes to the `notes/` directory.
-
-** 2. Semantic Interfaces
-- Trigger: `(:sensor :heartbeat)`
-- Action: `(:type :REQUEST :target :system :action :create-note :title "..." :content "..." :source-id "...")`
-
-* Phase D: Build (Implementation)
-
-** Package Context
+** Package Initialization
 #+begin_src lisp
-(in-package :opencortex)
+(in-package :cl-user)
+(defpackage :opencortex.skills.org-skill-scribe
+  (:use :cl :opencortex))
+(in-package :opencortex.skills.org-skill-scribe)
 #+end_src
 
 ** State: Checkpoint Management
-We track the last processed universal time to avoid redundant distillation.
+The Scribe must be efficient. It tracks the last processed timestamp to avoid redundant distillation and LLM token waste.
 
 #+begin_src lisp
 (defvar *scribe-last-checkpoint* 0
   "The universal-time of the last successful distillation run.")
+#+end_src
 
+#+begin_src lisp
 (defun scribe-load-state ()
   "Loads the scribe checkpoint from the state directory."
-  (let ((state-file (uiop:merge-pathnames* "state/scribe-checkpoint.lisp" (asdf:system-source-directory :opencortex))))
+  (let ((state-file (merge-pathnames "system/state/scribe-checkpoint.lisp" 
+                                     (asdf:system-source-directory :opencortex))))
     (if (uiop:file-exists-p state-file)
         (setf *scribe-last-checkpoint* (read-from-string (uiop:read-file-string state-file)))
         (setf *scribe-last-checkpoint* 0))))
+#+end_src
 
+#+begin_src lisp
 (defun scribe-save-state ()
   "Saves the current universal-time as the new checkpoint."
-  (let ((state-file (uiop:merge-pathnames* "state/scribe-checkpoint.lisp" (asdf:system-source-directory :opencortex))))
+  (let ((state-file (merge-pathnames "system/state/scribe-checkpoint.lisp" 
+                                     (asdf:system-source-directory :opencortex))))
     (ensure-directories-exist state-file)
     (with-open-file (out state-file :direction :output :if-exists :supersede)
       (format out "~a" (get-universal-time)))))
 #+end_src
 
-** Filtering: Privacy & Relevance
-The Scribe only cares about non-personal, non-distilled headlines.
+** Filtration: Privacy and Relevance
+To protect user privacy, the Scribe strictly ignores any node tagged with ~@personal~.
 
 #+begin_src lisp
 (defun scribe-get-distillable-nodes ()
-  "Returns a list of org-objects from the daily/ folder that require distillation."
+  "Returns a list of org-objects from memory that require distillation."
   (let ((results nil))
     (maphash (lambda (id obj)
                (declare (ignore id))
@@ -88,14 +73,14 @@ The Scribe only cares about non-personal, non-distilled headlines.
     results))
 #+end_src
 
-** Probabilistic: Extraction Prompt
-The LLM is tasked with identifying atomic concepts within the raw text.
+** Probabilistic Stage: Concept Extraction
+This function generates the specific distillation prompt for the LLM. It focuses on atomicity and structured Lisp output.
 
 #+begin_src lisp
 (defun probabilistic-skill-scribe (context)
-  "Generates the extraction prompt for the Scribe."
-  (let* ((payload (getf context :payload))
-         (nodes (scribe-get-distillable-nodes)))
+  "Generates the extraction prompt for the Scribe distillation task."
+  (declare (ignore context))
+  (let ((nodes (scribe-get-distillable-nodes)))
     (if nodes
         (let ((text-to-process ""))
           (dolist (node nodes)
@@ -111,21 +96,20 @@ Extract ATOMIC EVERGREEN NOTES from this text.
 RULES:
 1. One note per distinct concept.
 2. Output a list of Lisp plists: ((:title \"...\" :content \"...\" :source-id \"...\") ...)
-3. The content should be in Org-mode format.
-4. Keep titles descriptive and snake_case.
+3. Keep titles descriptive and snake_case.
 
 TEXT:
 ~a" text-to-process))
         nil)))
 #+end_src
 
-** Deterministic: Note Committal
-The deterministic gate receives the list of proposed notes and writes them to the filesystem.
+** Deterministic Stage: Knowledge Committal
+The final physical step. It takes the LLM's structured proposal and writes it to the local filesystem.
 
 #+begin_src lisp
 (defun scribe-commit-notes (proposals)
-  "Writes proposed atomic notes to the notes/ directory. Appends if the note exists."
-  (let ((notes-dir (uiop:merge-pathnames* "notes/" (asdf:system-source-directory :opencortex))))
+  "Writes distilled notes to the MemexHardHard Hard drive."
+  (let ((notes-dir (merge-pathnames "notes/" (asdf:system-source-directory :opencortex))))
     (ensure-directories-exist notes-dir)
     (dolist (note proposals)
       (let* ((title (getf note :title))
@@ -133,16 +117,15 @@ The deterministic gate receives the list of proposed notes and writes them to th
              (source-id (getf note :source-id))
              (filename (format nil "~a.org" (string-downcase (cl-ppcre:regex-replace-all " " title "_"))))
              (path (merge-pathnames filename notes-dir)))
-        (if (uiop:file-exists-p path)
-            (with-open-file (out path :direction :output :if-exists :append)
-              (format out "~%~%* Appended insight from ~a~%~a" source-id content))
-            (with-open-file (out path :direction :output :if-exists :supersede)
-              (format out ":PROPERTIES:~%:ID: ~a~%:SOURCE_ID: ~a~%:END:~%#+TITLE: ~a~%~%~a" 
-                      (org-id-new) source-id title content)))
-        (harness-log "SCRIBE: Processed evergreen note ~a" filename)))))
+        (with-open-file (out path :direction :output :if-exists :supersede)
+          (format out ":PROPERTIES:~%:ID: ~a~%:SOURCE_ID: ~a~%:END:~%#+TITLE: ~a~%~%~a" 
+                  (org-id-new) source-id title content))
+        (harness-log "SCRIBE: Distilled evergreen note ~a" filename)))))
+#+end_src
 
+#+begin_src lisp
 (defun verify-skill-scribe (action context)
-  "Executes the note creation and marks source nodes as distilled."
+  "Main deterministic gate for Scribe distillation."
   (declare (ignore context))
   (let ((data (cond ((and (listp action) (eq (getf action :type) :REQUEST))
                      (getf (getf action :payload) :payload))
@@ -150,15 +133,12 @@ The deterministic gate receives the list of proposed notes and writes them to th
                      action)
                     (t nil))))
     (when data
-      (harness-log "SCRIBE: Committing ~a atomic notes..." (length data))
       (scribe-commit-notes data)
       (scribe-save-state)
-      (harness-log "SCRIBE: Distillation complete.")
-      ;; Return a log event to stop the loop
-      (list :type :LOG :payload (list :text "Distillation successful.")))))
+      (list :type :LOG :payload (list :text "SCRIBE: Distillation cycle complete.")))) )
 #+end_src
 
-** Skill Registration
+** Registration
 #+begin_src lisp
 (defskill :skill-scribe
   :priority 50
@@ -166,7 +146,6 @@ The deterministic gate receives the list of proposed notes and writes them to th
              (let* ((payload (getf ctx :payload))
                     (sensor (getf payload :sensor)))
                (and (eq sensor :heartbeat)
-                    ;; Only run once per hour to check if we need to distill
                     (> (- (get-universal-time) *scribe-last-checkpoint*) 3600)
                     (scribe-get-distillable-nodes))))
   :probabilistic #'probabilistic-skill-scribe