amr/passepartout
1
0
Fork 0
Code Issues 15 Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
4ee85f3df086e0b2683be64a7fae6d8c8e2eed30
passepartout/docs/v0.2.x-REMEDIATION.org
Amr Gharbeia 5a0d1b1c38 remediation: backfill v0.1.0/v0.2.0 gaps (P0+P1) 
- vault: add vault-get-secret/vault-set-secret wrappers
- programming-org: implement org-modify (text search-replace) and org-ast-render (AST to Org text)
- programming-literate: implement literate-block-balance-check (paren validation) and literate-tangle-sync-check (org→lisp diff)
- system-self-improve: replace stubs with surgical text editing and error diagnosis; remove dead first defskill
- system-event-orchestrator: implement orchestrator-bootstrap (scan Org files for HOOK/CRON)
- system-archivist: implement Scribe distillation (daily logs→atomic notes) and Gardener link/orphan repair
- system-memory: implement memory-inspect with type/todo/orphan statistics
- core-skills, core-context: fix path relic (skills/ → lisp/, org/)
- docs: add Token Economics section to DESIGN_DECISIONS, remediation roadmap entries
2026-05-03 10:43:14 -04:00

12 KiB
Raw Blame History

v0.2.x Remediation Plan

Summary

Features marked DONE in the ROADMAP for v0.1.0 and v0.2.0 but whose implementations are stubs, no-ops, or missing critical functionality. These should have been completed in their respective versions and must be addressed before v0.3.0 development proceeds.

P0: system-archivist — Proper Distillation and Link Maintenance

Claimed status**: DONE (v0.1.0: "Scribe + Gardener background workers" + v0.2.0: "31 org files with full literate prose")

Actual state**: archivist-log is a trivial log wrapper (~10 lines). No knowledge

distillation, no broken link detection, no orphaned node flagging.

What it should do**:

Scribe (knowledge distillation)

  1. Read daily Org log files from the Memex daily/ directory
  2. Identify new entries (since last processed commit or timestamp)
  3. Extract conceptual claims, decisions, and atomic facts from prose

  4. Generate atomic Zettelkasten notes in notes/ with:

    • Descriptive snake_case filename (no dates)
    • :CREATED: property from the source log's date
    • Source: backlink to the original daily file and headline
    • Tags inferred from content and parent file
  5. Track processed state to avoid re-distilling the same content

Gardener (structural maintenance)

  1. Scan all Org files in the Memex for broken [[file:...][...]] links
  2. Scan memory-store for memory-object entries whose :parent-id or :children references point to deleted objects (orphaned nodes)
  3. Flag broken links and orphans with :GARDENER: broken-link or :GARDENER: orphan tags
  4. Generate a maintenance report as a Org buffer the user can review

Implementation approach

  • Wire into system-event-orchestrator as cron jobs:

    • Scribe: daily cron ("<%%Y-%%m-%%d %%a +1d>", tier :cognition)
    • Gardener: weekly cron ("<%%Y-%%m-%%d %%a +1w>", tier :cognition)
  • Use orchestrator-register-cron to schedule
  • Replace the trivial archivist-log function with real implementation
  • Track last-processed state via memory-store (:LATEST_PROCESSED_DATETIME property) or git commit hash

Dependencies**: system-event-orchestrator (cron scheduling), core-memory (object store)

Verification**: FiveAM test that creates a daily log with known content, runs the

Scribe, and asserts that an atomic note was created with correct backlinks.

P0: system-self-improve — Surgical Self-Editing and Self-Repair

Claimed status**: DONE (v0.2.0: "Self-editing (error detection, surgical fix, hot-reload)")

Actual state**: self-improve-edit does (declare (ignore old-text new-text)) followed by

a log message — no actual text transformation. self-improve-fix same pattern. The skill's trigger is nil so it never fires.

What it should do**:

Self-edit (surgical text replacement)

  1. Accept (filepath, old-text, new-text) and apply the transformation
  2. Read the file, locate old-text (with exact match verification), replace with new-text
  3. If the target is an Org file with a #+begin_src lisp block, tangling the file and reloading the skill after edit
  4. Create a memory snapshot before editing (rollback safety)
  5. Verify the edit succeeded (re-read file, confirm new-text appears)
  6. Return success/failure with a diff summary

Self-fix (error diagnosis and repair)

  1. Accept (skill-name, error-log) and diagnose the failure
  2. Parse the error log for: syntax errors (unmatched parens, invalid forms), undefined symbol references, semantic issues (prohibited forms)
  3. For syntax errors: locate the problematic region, propose a correction using structural Lisp knowledge
  4. For undefined references: check if the symbol exists in another package, if the skill's #+DEPENDS_ON: declaration is missing a dependency
  5. For semantic issues: identify the prohibited operation and suggest alternatives
  6. Invoke self-improve-edit to apply the fix
  7. After repair, run the skill's tests if they exist; if tests pass, hot-reload

Implementation approach

  • Add an actual :trigger function that activates on :ERROR or :STUCK signal types
  • self-improve-edit: use uiop:read-file-string, string replacement with ppcre:regex-replace or substring operations, write back with with-open-file
  • self-improve-fix: add structural analysis in programming-lisp.lisp for error parsing
  • Leverage the REPL skill for verification after repair (call lisp-eval on the fixed code block)

Dependencies**: programming-lisp (lisp-structural-check), programming-org (tangling),

core-memory (snapshot-memory), core-skills (jailed reload)

Verification**: FiveAM test that creates a file with known content, calls self-improve-edit,

and asserts the replacement was applied. Second test with a file containing a deliberate error, calls self-improve-fix, and asserts the error was corrected.

P1: system-event-orchestrator — Bootstrap Implementation

Claimed status**: v0.3.0 partially DONE ("hook-registry + cron-registry + tier classifier")

Actual state**: Hook/cron registries, tier dispatching, and heartbeat integration work.

But orchestrator-bootstrap is a stub: (log-message "ORCHESTRATOR: Bootstrap complete")

What it should do**:

  1. Scan the Memex projects/ and notes/ directories for Org files containing #+HOOK: properties
  2. For each #+HOOK: property found, call orchestrator-register-hook with the hook name and a gate function
  3. For files with #+CRON: properties (or cron expressions in timestamps), register them via orchestrator-register-cron
  4. Log the count of registered hooks and cron jobs at completion
  5. Run bootstrap once at startup (after memory is loaded but before cognitive loop begins)

Implementation approach

  • Use uiop:directory-files with glob patterns for *.org files
  • Use org-element from Emacs (via emacs-bridge or org-eval skill) for parsing, or implement a simple regex-based Org property parser in Lisp
  • Walk each file's headlines, extract property drawers, filter for HOOK: and CRON: keys
  • Call existing orchestrator-register-hook / orchestrator-register-cron

Dependencies**: programming-org (Org file parsing), file system access

Verification**: Create a test Org file with #+HOOK: on-write, run bootstrap,

assert the hook registry contains the expected entry.

P1: system-memory — Memory Introspection

Claimed status**: Skill exists but was never part of a version milestone.

Actual state**: memory-inspect is a no-op: (log-message "MEMORY: Self-inspection triggered.")

The :trigger is nil so the skill never activates.

What it should do**:

  1. Return a structured report of memory state:

    • Total objects in *memory-store*
    • Distribution by type (:HEADLINE, :PARAGRAPH, etc.)
    • Distribution by :TODO-STATE (TODO, NEXT, DONE, etc.)
    • Count of privacy-filtered objects
    • Most recent objects (by :version timestamp)
    • Current snapshot count and timestamps
    • Orphaned objects (parent-id references a deleted ID)
  2. Accept an optional filter to narrow the report (by type, by tag, by time range)
  3. Wire the trigger to activate on :INTROSPECTION signal type or /memory commands

Implementation approach

  • Iterate *memory-store* with maphash, collect statistics
  • Add to skill trigger: (eq (getf (getf ctx :payload) :sensor) :introspection)
  • Return results as a plist that can be rendered in the TUI

Dependencies**: core-memory (memory-store and memory-object struct)

Verification**: Ingest known objects, call memory-inspect, assert type counts and

object counts match.

P2: core-context — Semantic Retrieval (Embeddings)

Claimed status**: The foveal-peripheral model is implemented and tested, but the

embedding pipeline that feeds it is listed as TODO for v0.3.0.

Actual state**: The context rendering code (context-object-render) computes

cosine-similarity correctly, but org-object-vector is never populated. All objects have nil vectors, all similarities are 0.0, and the model falls back to "include everything within depth 2." This is functionally equivalent to no retrieval at all.

What it should do**:

  1. Add a populate-vector function to core-memory that calls an embedding provider and stores the result in the memory-object :vector slot
  2. At ingest time (ingest-ast), generate embeddings for new objects

  3. Embedding provider options (in priority order):

    • Ollama (local, nomic-embed-text or mxbai-embed-large)
    • OpenAI-compatible embedding API (text-embedding-3-small)
    • Fallback: TF-IDF bag-of-words vector (no external dependency)
  4. Updates: when memory-object content changes, mark :vector as :pending and process in a background batch via the event orchestrator
  5. Add an environment variable EMBEDDING_PROVIDER with default ollama

Implementation approach

  • Add an :embedding-provider function stored in *config*
  • embed-object: take content string → call provider → store float vector
  • Modify ingest-ast to call embed-object on each new object
  • Add batch processing in system-event-orchestrator for vector updates
  • Use bordeaux-threads with a lock for async embedding generation

Dependencies**: External embedding provider (Ollama or API), core-memory (vector slot)

Verification**: Create objects with content, run embedding pipeline, assert vectors

are non-nil and have the correct dimensionality. Verify that cosine-similarity between semantically similar objects exceeds 0.75 threshold.

P2: core-context — Subtree-Based Skill Source Loading

Claimed status**: DESIGN_DECISIONS §"Org-Mode as Unified AST" describes: "When the

agent needs information about the openctl-db function, it queries for the openctl-db subtree specifically."

Actual state**: context-skill-source reads the ENTIRE Org file as a string via

uiop:read-file-string. No subtree query exists.

What it should do**:

  1. Add a context-skill-subtree function that takes (skill-name, heading-name) and returns only the content under that headline
  2. Add a context-skill-function-signature function that returns only the function name, lambda list, and docstring
  3. Add a context-skill-tests function that returns only test blocks
  4. Modify context-skill-source to optionally accept a :subtree keyword argument
  5. If the Org file has an Org-element parser available, use it for structural queries; otherwise fall back to regex-based headline matching

Implementation approach

  • Use org-element via org-eval skill (REPL bridge to Emacs) if available
  • Lisp-native fallback: parse Org headlines with regex (=^*+ = pattern), match heading name by string comparison, extract content until next headline of equal or higher level
  • Cache parsed results to avoid re-parsing on repeated queries

Dependencies**: programming-org (Org parsing utilities), emacs-bridge (if Emacs

Org-element is preferred)

Verification**: Create a test Org file with multiple headlines, query for a specific

subtree, assert only that subtree's content is returned.

Priority and Sequencing

The remediation should proceed in this order:

  1. system-event-orchestrator bootstrap (P1) — needed as infrastructure for Scribe/Gardener cron scheduling
  2. system-archivist (P0) — depends on orchestrator for cron scheduling
  3. system-self-improve (P0) — independent, can proceed in parallel with #2
  4. core-context embeddings (P2) — independent, unlocks semantic retrieval
  5. core-context subtree loading (P2) — independent, improves context efficiency
  6. system-memory inspect (P1) — lowest priority, nice-to-have introspection

P0 items must be completed before v0.3.0 development begins. P1 items should be completed before v0.3.0 is released. P2 items can extend into early v0.3.0.

Out of Scope

Features listed as TODO in the ROADMAP for v0.3.0+ are NOT in this remediation plan. Specifically excluded:

  • HITL continuation-based suspension (v0.3.0 TODO)
  • Model-tier routing / cost optimization (v0.3.0 TODO)
  • Memory scope segmentation (v0.3.0 TODO)
  • Long-horizon planning / task trees (v0.4.0 TODO)
  • Shadow simulation mode (not on roadmap, aspirational)
  • Formal verification of dispatcher rules (not on roadmap, aspirational)
  • Bouncer rule learning from HITL decisions (not on roadmap, aspirational)
Reference in New Issue View Git Blame Copy Permalink