diff --git a/docs/v0.2.x-REMEDIATION.org b/docs/v0.2.x-REMEDIATION.org deleted file mode 100644 index ec0e0d2..0000000 --- a/docs/v0.2.x-REMEDIATION.org +++ /dev/null @@ -1,253 +0,0 @@ -#+TITLE: v0.2.x Remediation Plan -#+AUTHOR: -#+STARTUP: content -#+FILETAGS: :docs:plan:remediation: - -* Summary - -Features marked DONE in the ROADMAP for v0.1.0 and v0.2.0 but whose implementations -are stubs, no-ops, or missing critical functionality. These should have been -completed in their respective versions and must be addressed before v0.3.0 -development proceeds. - -* P0: system-archivist — Proper Distillation and Link Maintenance - -** Claimed status**: =DONE= (v0.1.0: "Scribe + Gardener background workers" + v0.2.0: "31 org files with full literate prose") - -** Actual state**: =archivist-log= is a trivial log wrapper (~10 lines). No knowledge -distillation, no broken link detection, no orphaned node flagging. - -** What it should do**: - -*** Scribe (knowledge distillation) -1. Read daily Org log files from the Memex =daily/= directory -2. Identify new entries (since last processed commit or timestamp) -3. Extract conceptual claims, decisions, and atomic facts from prose -4. Generate atomic Zettelkasten notes in =notes/= with: - - Descriptive snake_case filename (no dates) - - =:CREATED:= property from the source log's date - - =Source:= backlink to the original daily file and headline - - Tags inferred from content and parent file -5. Track processed state to avoid re-distilling the same content - -*** Gardener (structural maintenance) -1. Scan all Org files in the Memex for broken =[[file:...][...]]= links -2. Scan =memory-store= for =memory-object= entries whose =:parent-id= or =:children= - references point to deleted objects (orphaned nodes) -3. Flag broken links and orphans with =:GARDENER: broken-link= or =:GARDENER: orphan= tags -4. Generate a maintenance report as a Org buffer the user can review - -*** Implementation approach -- Wire into =system-event-orchestrator= as cron jobs: - - Scribe: daily cron (="<%%Y-%%m-%%d %%a +1d>"=, tier =:cognition=) - - Gardener: weekly cron (="<%%Y-%%m-%%d %%a +1w>"=, tier =:cognition=) -- Use =orchestrator-register-cron= to schedule -- Replace the trivial =archivist-log= function with real implementation -- Track last-processed state via =memory-store= (:LATEST_PROCESSED_DATETIME property) - or git commit hash - -** Dependencies**: =system-event-orchestrator= (cron scheduling), =core-memory= (object store) - -** Verification**: FiveAM test that creates a daily log with known content, runs the -Scribe, and asserts that an atomic note was created with correct backlinks. - -* P0: system-self-improve — Surgical Self-Editing and Self-Repair - -** Claimed status**: =DONE= (v0.2.0: "Self-editing (error detection, surgical fix, hot-reload)") - -** Actual state**: =self-improve-edit= does =(declare (ignore old-text new-text))= followed by -a log message — no actual text transformation. =self-improve-fix= same pattern. -The skill's trigger is =nil= so it never fires. - -** What it should do**: - -*** Self-edit (surgical text replacement) -1. Accept (=filepath=, =old-text=, =new-text=) and apply the transformation -2. Read the file, locate =old-text= (with exact match verification), replace with =new-text= -3. If the target is an Org file with a =#+begin_src lisp= block, tangling the file - and reloading the skill after edit -4. Create a memory snapshot before editing (rollback safety) -5. Verify the edit succeeded (re-read file, confirm =new-text= appears) -6. Return success/failure with a diff summary - -*** Self-fix (error diagnosis and repair) -1. Accept (=skill-name=, =error-log=) and diagnose the failure -2. Parse the error log for: syntax errors (unmatched parens, invalid forms), - undefined symbol references, semantic issues (prohibited forms) -3. For syntax errors: locate the problematic region, propose a correction - using structural Lisp knowledge -4. For undefined references: check if the symbol exists in another package, - if the skill's =#+DEPENDS_ON:= declaration is missing a dependency -5. For semantic issues: identify the prohibited operation and suggest alternatives -6. Invoke =self-improve-edit= to apply the fix -7. After repair, run the skill's tests if they exist; if tests pass, hot-reload - -*** Implementation approach -- Add an actual =:trigger= function that activates on =:ERROR= or =:STUCK= signal types -- =self-improve-edit=: use =uiop:read-file-string=, string replacement with - =ppcre:regex-replace= or substring operations, write back with =with-open-file= -- =self-improve-fix=: add structural analysis in =programming-lisp.lisp= for error parsing -- Leverage the REPL skill for verification after repair (call =lisp-eval= on the fixed code block) - -** Dependencies**: =programming-lisp= (lisp-structural-check), =programming-org= (tangling), -=core-memory= (snapshot-memory), =core-skills= (jailed reload) - -** Verification**: FiveAM test that creates a file with known content, calls self-improve-edit, -and asserts the replacement was applied. Second test with a file containing a -deliberate error, calls self-improve-fix, and asserts the error was corrected. - -* P1: system-event-orchestrator — Bootstrap Implementation - -** Claimed status**: v0.3.0 partially DONE ("hook-registry + cron-registry + tier classifier") - -** Actual state**: Hook/cron registries, tier dispatching, and heartbeat integration work. -But =orchestrator-bootstrap= is a stub: =(log-message "ORCHESTRATOR: Bootstrap complete")= - -** What it should do**: - -1. Scan the Memex =projects/= and =notes/= directories for Org files containing =#+HOOK:= properties -2. For each =#+HOOK:= property found, call =orchestrator-register-hook= with - the hook name and a gate function -3. For files with =#+CRON:= properties (or cron expressions in timestamps), - register them via =orchestrator-register-cron= -4. Log the count of registered hooks and cron jobs at completion -5. Run bootstrap once at startup (after memory is loaded but before cognitive loop begins) - -*** Implementation approach -- Use =uiop:directory-files= with glob patterns for =*.org= files -- Use =org-element= from Emacs (via =emacs-bridge= or =org-eval= skill) for parsing, - or implement a simple regex-based Org property parser in Lisp -- Walk each file's headlines, extract property drawers, filter for =HOOK:= and =CRON:= keys -- Call existing =orchestrator-register-hook= / =orchestrator-register-cron= - -** Dependencies**: =programming-org= (Org file parsing), file system access - -** Verification**: Create a test Org file with =#+HOOK: on-write=, run bootstrap, -assert the hook registry contains the expected entry. - -* P1: system-memory — Memory Introspection - -** Claimed status**: Skill exists but was never part of a version milestone. - -** Actual state**: =memory-inspect= is a no-op: =(log-message "MEMORY: Self-inspection triggered.")= -The =:trigger= is =nil= so the skill never activates. - -** What it should do**: - -1. Return a structured report of memory state: - - Total objects in =*memory-store*= - - Distribution by type (=:HEADLINE=, =:PARAGRAPH=, etc.) - - Distribution by =:TODO-STATE= (=TODO=, =NEXT=, =DONE=, etc.) - - Count of privacy-filtered objects - - Most recent objects (by =:version= timestamp) - - Current snapshot count and timestamps - - Orphaned objects (parent-id references a deleted ID) -2. Accept an optional filter to narrow the report (by type, by tag, by time range) -3. Wire the trigger to activate on =:INTROSPECTION= signal type or =/memory= commands - -*** Implementation approach -- Iterate =*memory-store*= with =maphash=, collect statistics -- Add to skill trigger: =(eq (getf (getf ctx :payload) :sensor) :introspection)= -- Return results as a plist that can be rendered in the TUI - -** Dependencies**: =core-memory= (memory-store and memory-object struct) - -** Verification**: Ingest known objects, call memory-inspect, assert type counts and -object counts match. - -* P2: core-context — Semantic Retrieval (Embeddings) - -** Claimed status**: The foveal-peripheral model is implemented and tested, but the -embedding pipeline that feeds it is listed as TODO for v0.3.0. - -** Actual state**: The context rendering code (=context-object-render=) computes -=cosine-similarity= correctly, but =org-object-vector= is never populated. -All objects have =nil= vectors, all similarities are =0.0=, and the model -falls back to "include everything within depth 2." This is functionally -equivalent to no retrieval at all. - -** What it should do**: - -1. Add a =populate-vector= function to =core-memory= that calls an embedding - provider and stores the result in the =memory-object= =:vector= slot -2. At ingest time (=ingest-ast=), generate embeddings for new objects -3. Embedding provider options (in priority order): - - Ollama (local, =nomic-embed-text= or =mxbai-embed-large=) - - OpenAI-compatible embedding API (=text-embedding-3-small=) - - Fallback: TF-IDF bag-of-words vector (no external dependency) -4. Updates: when =memory-object= content changes, mark =:vector= as =:pending= - and process in a background batch via the event orchestrator -5. Add an environment variable =EMBEDDING_PROVIDER= with default =ollama= - -*** Implementation approach -- Add an =:embedding-provider= function stored in =*config*= -- =embed-object=: take content string → call provider → store float vector -- Modify =ingest-ast= to call =embed-object= on each new object -- Add batch processing in =system-event-orchestrator= for vector updates -- Use =bordeaux-threads= with a lock for async embedding generation - -** Dependencies**: External embedding provider (Ollama or API), =core-memory= (vector slot) - -** Verification**: Create objects with content, run embedding pipeline, assert vectors -are non-nil and have the correct dimensionality. Verify that =cosine-similarity= -between semantically similar objects exceeds 0.75 threshold. - -* P2: core-context — Subtree-Based Skill Source Loading - -** Claimed status**: DESIGN_DECISIONS §"Org-Mode as Unified AST" describes: "When the -agent needs information about the =openctl-db= function, it queries for the -=openctl-db= subtree specifically." - -** Actual state**: =context-skill-source= reads the ENTIRE Org file as a string via -=uiop:read-file-string=. No subtree query exists. - -** What it should do**: - -1. Add a =context-skill-subtree= function that takes (=skill-name=, =heading-name=) - and returns only the content under that headline -2. Add a =context-skill-function-signature= function that returns only the function - name, lambda list, and docstring -3. Add a =context-skill-tests= function that returns only test blocks -4. Modify =context-skill-source= to optionally accept a =:subtree= keyword argument -5. If the Org file has an Org-element parser available, use it for structural queries; - otherwise fall back to regex-based headline matching - -*** Implementation approach -- Use =org-element= via =org-eval= skill (REPL bridge to Emacs) if available -- Lisp-native fallback: parse Org headlines with regex (=^*+ = pattern), - match heading name by string comparison, extract content until next - headline of equal or higher level -- Cache parsed results to avoid re-parsing on repeated queries - -** Dependencies**: =programming-org= (Org parsing utilities), =emacs-bridge= (if Emacs -Org-element is preferred) - -** Verification**: Create a test Org file with multiple headlines, query for a specific -subtree, assert only that subtree's content is returned. - -* Priority and Sequencing - -The remediation should proceed in this order: - -1. **system-event-orchestrator bootstrap** (P1) — needed as infrastructure for Scribe/Gardener cron scheduling -2. **system-archivist** (P0) — depends on orchestrator for cron scheduling -3. **system-self-improve** (P0) — independent, can proceed in parallel with #2 -4. **core-context embeddings** (P2) — independent, unlocks semantic retrieval -5. **core-context subtree loading** (P2) — independent, improves context efficiency -6. **system-memory inspect** (P1) — lowest priority, nice-to-have introspection - -P0 items must be completed before v0.3.0 development begins. P1 items should be -completed before v0.3.0 is released. P2 items can extend into early v0.3.0. - -* Out of Scope - -Features listed as TODO in the ROADMAP for v0.3.0+ are NOT in this remediation -plan. Specifically excluded: - -- HITL continuation-based suspension (v0.3.0 TODO) -- Model-tier routing / cost optimization (v0.3.0 TODO) -- Memory scope segmentation (v0.3.0 TODO) -- Long-horizon planning / task trees (v0.4.0 TODO) -- Shadow simulation mode (not on roadmap, aspirational) -- Formal verification of dispatcher rules (not on roadmap, aspirational) -- Bouncer rule learning from HITL decisions (not on roadmap, aspirational)