Compare commits
8 Commits
d35aea391e
...
ec882f87fb
| Author | SHA1 | Date | |
|---|---|---|---|
| ec882f87fb | |||
| 5a0d1b1c38 | |||
| 299f72c2bb | |||
| dd478d8a7b | |||
| e2fde5914e | |||
| 0760dc8012 | |||
| 227ad81b30 | |||
| b6923d5584 |
108
README.org
108
README.org
@@ -1,110 +1,66 @@
|
||||
#+TITLE: Passepartout: Your Autonomous, Plain-Text Life Assistant
|
||||
#+TITLE: Passepartout — Your Autonomous, Plain-Text Life Assistant
|
||||
#+AUTHOR: Amr
|
||||
#+FILETAGS: :passepartout:ai:assistant:
|
||||
|
||||
#+HTML: <div style="display: flex; gap: 8px; flex-wrap: wrap; margin-bottom: 1em;">
|
||||
#+HTML: <img src="https://img.shields.io/github/v/tag/amrgharbeia/passepartout?label=version&style=flat-square">
|
||||
#+HTML: <img src="https://img.shields.io/github/license/amrgharbeia/passepartout?style=flat-square">
|
||||
#+HTML: <img src="https://img.shields.io/github/v/tag/amrgharbeia/opencortex?label=version&style=flat-square">
|
||||
#+HTML: <img src="https://img.shields.io/github/license/amrgharbeia/opencortex?style=flat-square">
|
||||
#+HTML: <img src="https://img.shields.io/badge/Lisp-Common%20Lisp-blue?style=flat-square">
|
||||
#+HTML: <img src="https://img.shields.io/badge/docs-Org--mode-green?style=flat-square">
|
||||
#+HTML: </div>
|
||||
|
||||
* Quick Install
|
||||
Passepartout is an AI assistant that runs in your terminal, reads and writes your Org-mode files, executes tasks through a verified safety gate, and works fully offline with local LLMs. Everything it knows is a folder of plain text files that you own.
|
||||
|
||||
**One-line install:**
|
||||
|
||||
#+begin_src bash
|
||||
curl -fsSL https://raw.githubusercontent.com/amrgharbeia/opencortex/main/passepartout | bash -s configure
|
||||
#+end_src
|
||||
|
||||
Then run ~passepartout tui~ to start chatting.
|
||||
|
||||
* Meet Passepartout
|
||||
|
||||
Most AI assistants are just chatbots. You ask a question, they answer, they forget you exist. They trap your conversations in proprietary web apps and silo your data.
|
||||
|
||||
*Passepartout is different. It is an AI that lives inside your own text files.*
|
||||
|
||||
It runs locally on your machine. It reads your notes, organizes your life, executes tasks, and gardens your knowledge base—all while keeping your data in plain text files you own completely.
|
||||
|
||||
* Why Passepartout Exists
|
||||
|
||||
The current generation of AI agents have a fundamental flaw: they prioritize quick demos over long-term reliability and user sovereignty.
|
||||
|
||||
The biggest problem is data ownership. Most agents bury your memories in opaque databases. If you want to see your own data, you have to ask the AI to fetch it. If the app shuts down, your data is gone.
|
||||
|
||||
Passepartout solves this with total plain-text transparency. Your entire life is a folder of text files. Passepartout manages them the same way you do—with any text editor. No database to migrate, no schema to update, no lock-in.
|
||||
|
||||
* What Makes Passepartout Different
|
||||
|
||||
Most AI agents are Python applications that happened to call an LLM. Passepartout is different. It is built in pure Common Lisp—top to bottom, no wrapper, no translation layer.
|
||||
|
||||
The kernel is Lisp. The skills are Lisp. The memory system is Lisp. The TUI is Lisp. One language from the hardware to the agent's thoughts.
|
||||
|
||||
Python agents need a second language for configuration (YAML), a third for memory (JSON or SQLite), and a fourth for deployment (Docker). Passepartout needs SBCL. That's it.
|
||||
|
||||
This is not nostalgia for the 1980s. Lisp has two properties that matter for an autonomous agent.
|
||||
|
||||
First, code is data. In Lisp, the agent can read its own code the same way it reads a text file. It can parse its skills, understand their structure, and modify them at runtime. Python's AST is a foreign object—the agent can inspect it, but not treat it as something it owns.
|
||||
|
||||
Second, stability. The Common Lisp specification has been stable for decades. Your code does not break because a library changed its API. Dependencies do not rot. A Lisp program from 1995 still runs today.
|
||||
|
||||
Your data lives in Org-mode files. Not a database. Not JSON. Not a vector store. Just plain text that you can read in any editor, search with grep, and back up any way you want.
|
||||
|
||||
This matters because every other agent makes your data dependent on their app. Their database schema defines what you can store. Their migration scripts decide whether your data survives an upgrade. Passepartout has no schema. Your memory is a folder of text files. It survives app updates, platform switches, and decades of use.
|
||||
|
||||
The agent fixes itself. When it encounters an error, it can modify its own code, apply surgical fixes, and learn from the outcome to improve and grow. Skills hot-reload at runtime, so you can extend the system without restarting. And if something goes wrong during a complex operation, it snaps back to a known-good state.
|
||||
|
||||
* Three Problems Every Agent Ignores
|
||||
|
||||
Every other AI assistant has three fundamental flaws that Passepartout addresses.
|
||||
|
||||
The first is the data silo. Most agents bury your memories in opaque databases. If you want to see your own data, you have to ask the AI to fetch it. If the app shuts down, your data is gone. Passepartout stores everything in plain text files that you own. No migration needed, no schema to update, no lock-in.
|
||||
|
||||
The second is the hallucination problem. Most agents pipe AI-generated text directly into your terminal. If the model hallucinates, it can delete your files or break your system. Passepartout splits its brain into two parts. The creative brain (the LLM) suggests actions. The strict guard (deterministic logic) intercepts every proposal before it touches a file or runs a command. If the AI hallucinates, the guard blocks it.
|
||||
|
||||
The third is cloud dependency. Most assistants rely entirely on big tech APIs. When your internet drops, or the service goes down, your assistant dies. Worse, your private notes are constantly sent to third-party servers. Passepartout runs on your own hardware using free, open-source models. Your private data never leaves your laptop. Cloud models are optional—used only when you explicitly allow them for complex tasks.
|
||||
Then ~passepartout tui~ to start chatting.
|
||||
|
||||
* Quick Start
|
||||
|
||||
You need SBCL (Steel Bank Common Lisp) installed.
|
||||
You need SBCL (Common Lisp), git, and curl.
|
||||
|
||||
#+begin_src bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/amrgharbeia/opencortex.git ~/projects/passepartout
|
||||
|
||||
# Run the Setup Wizard
|
||||
cd ~/projects/passepartout
|
||||
./passepartout configure
|
||||
|
||||
# Verify System Health
|
||||
passepartout doctor
|
||||
|
||||
# Enter the Brain
|
||||
passepartout tui
|
||||
./passepartout configure # install deps, tangle, setup wizard
|
||||
passepartout tui # launch the terminal interface
|
||||
#+end_src
|
||||
|
||||
* The Onboarding Trifecta
|
||||
See [[file:docs/USER_MANUAL.org][User Manual]] for the full guide.
|
||||
|
||||
`passepartout setup` guides you through configuring LLM providers. Tell it how to talk to Ollama, Groq, OpenRouter, or your own endpoint.
|
||||
* Why Passepartout
|
||||
|
||||
`passepartout gateway link <platform> <token>` connects external chat gateways. Talk to your agent from Telegram while it works on your files.
|
||||
** Your data stays yours.** No database, no vector store, no cloud silo. Your entire memory is a folder of Org files. You can read them with any text editor, search them with grep, and back them up however you like. If Passepartout stops existing, your data doesn't disappear.
|
||||
|
||||
`passepartout doctor` shows you what's working, what's broken, and what needs attention.
|
||||
** The LLM can't do damage.** Every action the LLM proposes passes through a deterministic safety gate before it touches a file, runs a command, or sends a message. The LLM suggests; the gate decides. Hallucinations are blocked, not corrected after the fact.
|
||||
|
||||
** Runs on your hardware.** Works fully offline with Ollama and local models. Cloud providers (OpenRouter, OpenAI, Anthropic, Groq, Gemini, DeepSeek, NVIDIA NIM) are optional add-ons.
|
||||
|
||||
** Written in Common Lisp.** Code is data. The agent reads its own source the same way it reads a text file — it parses, modifies, and hot-reloads its skills without restarting. One language from the kernel to the TUI to the build system.
|
||||
|
||||
* Architecture
|
||||
|
||||
Passepartout has three layers.
|
||||
|
||||
The Harness is the kernel. It runs the [[file:harness/loop.org][metabolic loop]]—Perceive → Reason → Act—each signal moving through normalization, LLM reasoning, skill verification, and action execution. Depth limits prevent infinite loops. The [[file:harness/memory.org][memory system]] persists to plain-text Org-mode files with snapshot and rollback on errors.
|
||||
|
||||
The Skills are userland—thin harness, fat skills. Modular skills load at runtime. Diagnostics, Configuration, LLM Gateway, Shell Actuation, Emacs Editing, Self-Edit, Self-Fix, Credentials Vault, Tool Permissions, Protocol Validator, and more. Each is an independent Org-mode module. Add new skills without touching the kernel.
|
||||
|
||||
The Interface is what you use to talk to the agent. A native Lisp [[file:harness/tui-client.org][TUI]] with semantic highlighting and history. A [[file:harness/communication.org][TCP socket server]] for CLI interaction. External channels via the gateway—Telegram and beyond.
|
||||
- [[file:org/core-loop.org][Metabolic Loop]] — Perceive → Reason → Act, the fundamental cognitive cycle
|
||||
- [[file:org/security-dispatcher.org][Dispatcher]] — 9-vector safety gate: secret scanning, path protection, shell safety, lisp validation, network exfiltration, privacy filtering
|
||||
- [[file:org/core-memory.org][Memory]] — Single-address-space object store with Merkle-tree integrity and snapshot rollback
|
||||
- [[file:org/core-skills.org][Skill Engine]] — 20 hot-reloadable skills loaded at boot, each an independent Org file
|
||||
- [[file:org/gateway-tui.org][TUI]] — Croatoan-based terminal interface connected via framed TCP protocol
|
||||
- [[file:org/gateway-llm.org][LLM Routing]] — Cascade dispatch through multiple providers with tier-based model selection
|
||||
|
||||
* Project Documentation
|
||||
|
||||
Passepartout practices what it preaches—the documentation lives in the code.
|
||||
|
||||
The [[file:USER_MANUAL.org][User Manual]] covers setup, configuration, and commands. The [[file:docs/ROADMAP.org][Evolutionary Roadmap]] shows our plan for reaching state-of-the-art capabilities. The [[file:docs/CONTRIBUTING.org][Contributing]] guide teaches you how to add new skills.
|
||||
| Document | Answers |
|
||||
|----------|---------|
|
||||
| [[file:docs/USER_MANUAL.org][User Manual]] | How do I use it? |
|
||||
| [[file:docs/ARCHITECTURE.org][Architecture]] | How does it work inside? |
|
||||
| [[file:docs/DESIGN_DECISIONS.org][Design Decisions]] | Why was it built this way? |
|
||||
| [[file:docs/ROADMAP.org][Roadmap]] | Where is it going? When? |
|
||||
| [[file:docs/ROADMAP.org][TODO]] | Who is doing what? |
|
||||
| [[file:docs/CONTRIBUTING.org][Contributing]] | How do I contribute? |
|
||||
|
||||
* License
|
||||
|
||||
|
||||
796
TODO.org
796
TODO.org
@@ -1,796 +0,0 @@
|
||||
# Passepartout Project Tasks
|
||||
# All Passepartout-related TODOs live here. gtd.org links to this file.
|
||||
# Evolutionary context: see docs/ROADMAP.org
|
||||
|
||||
* PHASE: AUTONOMOUS MVP (v0.1.0 Released)
|
||||
:PROPERTIES:
|
||||
:ID: proj-mvp-v0-1-0
|
||||
:END:
|
||||
The "Zero-to-One" release. The agent must be mathematically secure, CLI-first, and capable of autonomous Memex maintenance.
|
||||
|
||||
** DONE 1. Harness Hardening (The Final Audit)
|
||||
*** DONE Audit remaining core skills (`org-skill-policy.org`, `org-skill-bouncer.org`) to the new Literate Granularity standard.
|
||||
*** DONE Implement Verification Lock: Ensure `MANDATORY_SKILLS` pass `validate-lisp-syntax` before boot proceeds.
|
||||
*** DONE Logging & Transparency: Ensure `context-get-system-logs` is utilized by the Reason engine to explain blocked actions.
|
||||
|
||||
** DONE 2. The Autonomous Scribe & Gardener (The Primary Value Prop)
|
||||
*** DONE Implement `org-skill-scribe.org`: Background worker that distills daily chronological logs into structured Zettelkasten notes.
|
||||
*** DONE Implement `org-skill-gardener.org`: Heartbeat-driven skill that autonomously flags orphaned nodes and repairs broken links.
|
||||
|
||||
** DONE 3. The Zero-to-One Experience (setup.org)
|
||||
*** DONE Consolidate installation instructions, `onboard.sh`, and `Dockerfile` into a single, literate `setup.org` file.
|
||||
*** DONE Ensure the setup process interactively builds the `.env` and verifies SBCL/Quicklisp dependencies.
|
||||
|
||||
** DONE 4. CLI-First Actuation
|
||||
CLOSED: [2026-04-14 Tue 09:40]
|
||||
*** DONE Verified the `cli` actuator and inbound gateway handle standard I/O interaction gracefully via a stateful `socat` connection.
|
||||
|
||||
* PHASE: PUBLICATION & VERIFICATION (v0.1.0 Post-Release)
|
||||
:PROPERTIES:
|
||||
:ID: proj-pub-v0-1-0
|
||||
:END:
|
||||
Ensuring the system is ready for the world through collaborative testing, documentation, and licensing.
|
||||
|
||||
** DONE 1. Collaborative End-to-End Testing [2026-04-21 Tue]
|
||||
CLOSED: [2026-04-21 Tue 17:30]
|
||||
*** DONE Verified stable foundation at commit `cab0e5a`.
|
||||
*** DONE Verified boot sequence and bidirectional connectivity.
|
||||
|
||||
** DONE 2. Semantic Reorganization & System Stabilization [2026-04-21 Tue]
|
||||
CLOSED: [2026-04-21 Tue 18:30]
|
||||
*** DONE Rename directories: harness/, library/, environment/, infrastructure/.
|
||||
*** DONE Consolidate Probabilistic engine into reason.lisp.
|
||||
*** DONE Embed bidirectional CLI logic into passepartout.sh.
|
||||
*** DONE Stabilize skill engine: 12/12 skills loaded with package jailing.
|
||||
*** DONE Cleanup legacy documentation and deployment artifacts.
|
||||
|
||||
** DONE 2. Comprehensive Documentation <2026-04-14 Tue>
|
||||
CLOSED: [2026-04-20 Mon 18:00]
|
||||
*** DONE Draft `USER_MANUAL.org`: Focus on CLI interaction, installation, and Memex structure.
|
||||
*** DONE Draft `CONTRIBUTING.org`: Explain Literate Granularity and Skill creation standards.
|
||||
|
||||
** DONE 3. License & Legal Finalization <2026-04-14 Tue>
|
||||
CLOSED: [2026-04-17 Fri 11:25]
|
||||
*** DONE Assign the AGPLv3 open-source license.
|
||||
*** DONE Implement a broad Contributor License Agreement (CLA) process.
|
||||
*** DONE Update `LICENSE` and `CHANGELOG` accordingly.
|
||||
|
||||
** DONE 4. GitHub Migration ** TODO 4. GitHub Migration & Repository Setup <2026-04-14 Tue> Repository Setup [2026-05-02 Sat]
|
||||
*** TODO Migrate primary remote to GitHub and configure canonical repository.
|
||||
*** TODO Set repository topics, badges, issue templates, and CI/CD foundations.
|
||||
|
||||
** TODO 5. Marketing & Social Media Launch <2026-04-14 Tue>
|
||||
*** TODO Execute PR plan (Reddit, Hacker News, X/Twitter).
|
||||
*** TODO Create a short, high-quality terminal demo GIF/video of the TUI interaction.
|
||||
|
||||
* PHASE: INTERACTIVE REFINEMENT (v0.2.0 Target)
|
||||
:PROPERTIES:
|
||||
:ID: proj-refinement-v0-2-0
|
||||
:END:
|
||||
Elevating the user interface from raw shell piping to a high-fidelity, native Lisp experience. Priority: Self-editing is the foundation of all future growth. Full org-mode manipulation makes the agent a true Emacs citizen.
|
||||
|
||||
Roadmap basis: Evolutionary roadmap from README.org. Working backwards from SOTA parity.
|
||||
|
||||
** DONE 0. Autonomous Self-Editing Foundation
|
||||
*** DONE org-skill-lisp-repair (Lisp syntax repair)
|
||||
- Deterministic: auto-balance parens via paren-counting
|
||||
- Probabilistic: LLM generates surgical fix on =:syntax-error= events
|
||||
- Memory rollback on failure
|
||||
DONE: Now in org-skill-lisp-utils (merged from contrib)
|
||||
*** DONE org-skill-emacs-edit (full org-mode manipulation)
|
||||
- Read org buffers, parse AST via org-element
|
||||
- Create/update/delete headlines, set properties, manage TODO states
|
||||
- Handle =id:= links and internal links
|
||||
- Pure Lisp implementation (no Emacs subprocess)
|
||||
*** DONE Expose Structural AST Editing Tools
|
||||
- Wrap org-skill-emacs-edit into def-cognitive-tool definitions
|
||||
- Force LLM to use semantic node updates instead of regex file I/O
|
||||
*** DONE Implement Reflection Loops
|
||||
- Feed rejection traces (syntax errors, policy blocks) back to LLM to trigger self-correction
|
||||
*** DONE Harden Actuators
|
||||
- Fix path-traversal vulnerabilities in file I/O tools (e.g. :write-file)
|
||||
- Enforce Merkle-snapshots on all state-modifying actions
|
||||
*** DONE Implement tool permission tiers (ask/allow/deny)
|
||||
- Per-tool permission plist stored in org-object
|
||||
- =generate-tool-belt-prompt= filters denied tools before LLM sees them
|
||||
- Ask-tier prompts user before execution
|
||||
*** DONE Implement skill hot-reload (=:reload-skill= tool)
|
||||
- Swap compiled skill files without breaking active sockets
|
||||
- Reload skill into jailed package namespace
|
||||
- DONE: Added :reload-skill, :read-file, :write-file, :replace-string tools
|
||||
- DONE: Fixed ASDF compilation bug (position tracking issue with :serial t)
|
||||
- DONE: Added explicit :depends-on declarations to passepartout.asd
|
||||
|
||||
** DONE Engineering Process Improvements [2026-04-23 Wed]
|
||||
*** DONE Fix ASDF compilation bug (position tracking at byte 16834)
|
||||
- Root cause: Duplicate proto-get, bt: prefix issues, :serial t position cache
|
||||
- Fix: Removed duplicate, fixed bt:->bordeaux-threads, explicit dependencies
|
||||
- Added eval-when wrapper for new tools (good Lisp practice)
|
||||
*** DONE Add test-first methodology to engineering standards
|
||||
- Rule 10: Test-first - design tests before coding, run chaos testing
|
||||
- Rule 11: Org as thinking medium - document investigations in prose
|
||||
- Rule 12: Engineering decision audit trail - document root cause, tradeoffs
|
||||
- Added to passepartout-contrib/skills/org-skill-engineering-standards.org
|
||||
*** DONE Perform comprehensive architectural review and evolution strategy [2026-04-27 Mon]
|
||||
- Identified hidden gaps: Org-mode round-trip, sandboxing vulnerabilities, and GC scaling.
|
||||
- Defined "Structural AST Editing" and "Reflection Loops" as core strategic requirements.
|
||||
- Captured findings in [[file:notes/passepartout-architectural-evolution.org][passepartout-architectural-evolution.org]].
|
||||
*** DONE Fix API drift in passepartout-contrib [2026-04-27 Mon]
|
||||
- Standardized legacy keywords (:neuro/:symbolic) to new harness standard (:probabilistic/:deterministic).
|
||||
- Updated 16 skills in passepartout-contrib for kernel compatibility.
|
||||
|
||||
** DONE 4. Core Skills Consolidation [2026-04-23 Thu]
|
||||
- Merged lisp-validator + lisp-repair → org-skill-lisp-utils.org
|
||||
- Added lisp utilities: count-char, deterministic-repair, neural-repair
|
||||
- Added validation: structural, syntactic, semantic checks
|
||||
- Moved org-skill-self-fix from contrib → core
|
||||
- Moved org-skill-engineering-standards from contrib → core
|
||||
- Deleted old org-skill-lisp-validator.org
|
||||
|
||||
** DONE 5. Advanced CLI Onboarding Experience
|
||||
*** DONE Implement interactive Lisp CLI wizard (=passepartout setup=)
|
||||
*** DONE Implement =passepartout gateway link= for Telegram/Signal bot connection [2026-05-02 Sat]
|
||||
*** DONE Implement =passepartout gateway unlink= to disable a gateway [2026-05-02 Sat]
|
||||
*** DONE Implement =passepartout gateway list= to show gateway status [2026-05-02 Sat]
|
||||
*** DONE Implement =passepartout install <skill>= for dynamic skill downloading [2026-05-02 Sat]
|
||||
*** DONE Implement =passepartout doctor= for environment health and API key validation [2026-04-28 Tue]
|
||||
- Verified 22/22 skills loading with clean boot.
|
||||
- Fixed macro conflicts and package jailing bugs.
|
||||
|
||||
** DONE Chaos-Driven Bug Fixes (v0.2.0 Pre-Release) [2026-04-28 Tue]
|
||||
- Fixed major conflict between Type A and Type B def-cognitive-tool macros.
|
||||
- Enforced dynamic-only loading by removing skills from passepartout.asd.
|
||||
- Fixed let/let* sequential binding bugs in emacs-edit and self-edit.
|
||||
- Standardized *cognitive-tools* as a centralized hash table.
|
||||
- Resolved missing in-package declarations in core skills.
|
||||
|
||||
** DONE 1. Common Lisp TUI Implementation [2026-04-28 Tue]
|
||||
*** DONE Integrate =croatoan= for native terminal rendering
|
||||
*** DONE Implement scrollable history viewport for chat and thought streams
|
||||
*** DONE Implement fixed bottom input box with multi-line support and command history
|
||||
*** DONE Implement persistent status bar for background workers (Scribe/Gardener)
|
||||
*** DONE Support syntax highlighting for Lisp code blocks and Org-mode syntax
|
||||
|
||||
** DONE 2. Slash Commands & Interactive Control [2026-04-28 Tue]
|
||||
*** DONE Implement =/help= command for system overview
|
||||
*** DONE Implement =/exit= and =/clear= commands
|
||||
*** DONE Implement =/skill-load <name>= for dynamic hot-reloading
|
||||
*** DONE Implement =/status=, =/config=, =/search=, =/commit= slash commands
|
||||
|
||||
** DONE 3. Direct Lisp-to-Terminal Actuation [2026-04-28 Tue]
|
||||
*** DONE Refactor the =:cli= actuator to use native TUI rendering
|
||||
|
||||
** DONE 4. Persistent REPL for Interactive Development [2026-04-30 Thu]
|
||||
*** DONE Implement org-skill-repl for persistent Lisp evaluation
|
||||
- repl-eval: evaluate code with result+output+error separation
|
||||
- repl-inspect: inspect variables and functions
|
||||
- repl-list-vars: list bound symbols in package
|
||||
- repl-load-file: load files into image
|
||||
- Supports REPL-first workflow with literate reflection in org
|
||||
|
||||
* PHASE: EVENT ORCHESTRATION + HITL (v0.3.0)
|
||||
:PROPERTIES:
|
||||
:ID: proj-orchestration-v0-3-0
|
||||
:END:
|
||||
Unified control plane: hooks + cron + routing in one skill. Deep project understanding.
|
||||
|
||||
** DONE 0. Project Renaming (Bouncer → Dispatcher) [2026-05-02 Sat]
|
||||
*** TODO Audit all files for component names to rename
|
||||
*** TODO Rename org-skill-bouncer.org → org-skill-dispatcher.org
|
||||
*** TODO Rename skill-bouncer package → skill-dispatcher
|
||||
*** TODO Rename cognitive tool =:bouncer= → =:dispatcher=
|
||||
*** TODO Update all references in harness, skills, documentation
|
||||
*** TODO Update gtd.org and ROADMAP.org terminology
|
||||
*** TODO Update DESIGN_DECISIONS.org section if applicable
|
||||
*** TODO Verify all tests pass after rename
|
||||
:LOGBOOK:
|
||||
- State "TODO" from "" [2026-05-01 Fri 15:40]
|
||||
:END:
|
||||
|
||||
The Dispatcher's role has evolved beyond security guard. It is the seed of the deterministic engine - it learns to execute procedures without invoking the neural net.
|
||||
|
||||
** TODO 1. Event Orchestrator (unified hooks+cron+routing)
|
||||
*** TODO Integrate contrib org-skill-event-orchestrator
|
||||
- Merge *hook-registry* + *cron-registry* + complexity classifier
|
||||
- Hooks via =#+HOOK:= Org-mode properties
|
||||
- Three complexity tiers: =:REFLEX= (no LLM), =:COGNITION= (light LLM), =:REASONING= (full LLM)
|
||||
- Hook into heartbeat for cron processing
|
||||
|
||||
** TODO 2. Context Manager (project scoping)
|
||||
*** TODO Integrate contrib org-skill-context-manager
|
||||
- Stack-based context with =push-context= / =pop-context=
|
||||
- Path resolution relative to current context
|
||||
- Memory scope: =:scope= property on org-objects (memex/session/project)
|
||||
- Implement lazy-loading proxies for large-scale memory traversal (offload cold nodes to disk)
|
||||
|
||||
** TODO 3. Model-Tier Routing (cost optimization)
|
||||
*** TODO Extend =*model-selector-fn= for complexity-based routing
|
||||
- Heartbeats → smallest model
|
||||
- User input → medium model
|
||||
- Complex reasoning → large model
|
||||
- Source: GBrain sub-agent model routing
|
||||
|
||||
** TODO 4. Memory Scope Segmentation
|
||||
*** TODO Extend org-object with =:scope= property
|
||||
- =:memex= (permanent knowledge)
|
||||
- =:session= (ephemeral context)
|
||||
- =:project= (scoped to current work)
|
||||
- Scope-aware retrieval in memory.lisp
|
||||
|
||||
** TODO 5. Asynchronous Embedding Gateway
|
||||
*** TODO Implement provider-agnostic org-skill-embedding-gateway
|
||||
- Support Ollama, llama.cpp, and OpenAI based on .env config
|
||||
- Implement lazy-loading: edits mark nodes as =:vector :pending=
|
||||
- Background worker thread batches pending nodes and updates Merkle tree silently
|
||||
|
||||
** TODO 6. Slash Commands (TUI ergonomics)
|
||||
*** TODO M-x style command palette
|
||||
*** TODO /- prefix for command mode
|
||||
*** TODO Commands defined in Org-mode
|
||||
|
||||
* PHASE: LONG-HORIZON PLANNING + GIT WORKFLOWS (v0.4.0)
|
||||
:PROPERTIES:
|
||||
:ID: proj-planning-v0-4-0
|
||||
:END:
|
||||
Multi-step task mastery, structured tracking with failure handling and course correction.
|
||||
|
||||
** TODO 0. Long-Horizon Planning (task tree DAG)
|
||||
*** TODO Implement org-skill-long-horizon
|
||||
- Decompose complex tasks into Org-mode headline trees
|
||||
- Terminal states: =:todo= → =:next-action= → =:in-progress= → =:done= / =:blocked= / =:stuck=
|
||||
- Parent summarises child results
|
||||
- Branch pruning when paths fail
|
||||
- Source: Claude Code ULTRAPLAN (reimplemented in Lisp)
|
||||
|
||||
** TODO 1. Git Steward (version control integration)
|
||||
*** TODO Integrate contrib org-skill-git-steward
|
||||
- Status, diff, commit, push, branch operations
|
||||
- Policy: commit-before-modify gate (from contrib engineering-standards)
|
||||
- Log commits to memory
|
||||
|
||||
** TODO 2. TDD Runner Integration
|
||||
*** TODO Integrate contrib org-skill-tdd-runner
|
||||
- Run FiveAM tests on file save
|
||||
- Inject =:test-failure= event on red
|
||||
- Hook into self-fix for auto-repair proposals
|
||||
|
||||
** TODO 3. Deep Emacs Integration
|
||||
*** TODO Full org-agenda awareness
|
||||
- Navigate, clock time, refile, archive
|
||||
- Uses org-element + org-id
|
||||
|
||||
* PHASE: INTERACTIVE ACTUATION & ENVIRONMENT STEWARDSHIP (v0.5.0)
|
||||
:PROPERTIES:
|
||||
:ID: proj-actuation-v0-5-0
|
||||
:END:
|
||||
Interactive terminal sessions and autonomous dependency management.
|
||||
|
||||
** TODO 0. Interactive PTY Actuator
|
||||
*** TODO Stream long-running process output to the context window (e.g., `npm run dev`, REPLs)
|
||||
*** TODO Implement async interrupt control (Ctrl+C emulation)
|
||||
|
||||
** TODO 1. The Environment Steward
|
||||
*** TODO Autonomously detect missing dependencies (e.g., "Command not found")
|
||||
*** TODO Propose an installation command and retry the failed action
|
||||
|
||||
* PHASE: CREATOR + ARCHITECT + GTD (v0.6.0)
|
||||
:PROPERTIES:
|
||||
:ID: proj-creator-v0-5-0
|
||||
:END:
|
||||
Agent bootstraps itself: creates skills autonomously, designs projects from PRDs, tracks work.
|
||||
|
||||
** TODO 0. Skill Creator (autonomous skill generation)
|
||||
*** TODO Integrate contrib org-skill-creator
|
||||
- LLM drafts complete skill org-file from natural language
|
||||
- Mandatory: syntax validation → jail-load → test → register
|
||||
|
||||
** TODO 1. Architect Agent (PRD → PROTOCOL)
|
||||
*** TODO Integrate contrib org-skill-architect
|
||||
- Scan =:STATUS: FROZEN= PRDs
|
||||
- Generate Phase B PROTOCOL from Phase A
|
||||
- Write to same file
|
||||
|
||||
** TODO 2. GTD Integration (project tracking)
|
||||
*** TODO Integrate contrib org-skill-gtd
|
||||
- Full GTD cycle: capture, clarify, organize, reflect, engage
|
||||
- org-gtd v4.0 DAG (=:TRIGGER:=, =:BLOCKER:=)
|
||||
|
||||
** TODO 3. Consensus Loop (multi-model agreement)
|
||||
*** TODO Integrate contrib org-skill-consensus
|
||||
- Run multiple providers for critical decisions
|
||||
- Compare results, detect disagreements
|
||||
- Confidence scoring
|
||||
|
||||
** TODO 4. Web Research (Playwright browsing)
|
||||
*** TODO Integrate contrib org-skill-playwright + org-skill-web-research
|
||||
- Headless Chromium via Python bridge
|
||||
- Text extraction and screenshots
|
||||
- Gemini Web UI automation
|
||||
|
||||
** TODO 5. Memex Management (PARA lifecycle)
|
||||
*** TODO Integrate contrib org-skill-memex + org-skill-workspace-manager
|
||||
- Archive DONE tasks, suggest refiling
|
||||
- Detect orphaned nodes
|
||||
- PARA/Zettelkasten maintenance
|
||||
|
||||
* PHASE: VISUAL GROUNDING & MCP BRIDGE (v0.7.0)
|
||||
:PROPERTIES:
|
||||
:ID: proj-vision-v0-7-0
|
||||
:END:
|
||||
Multimodal visual interaction and ecosystem-wide tool compatibility.
|
||||
|
||||
** TODO 0. Computer Use / Vision
|
||||
*** TODO Allow the agent to request host OS or browser screenshots
|
||||
*** TODO Analyze UI and issue precise X/Y coordinate click/type commands via an X11/Wayland bridge
|
||||
|
||||
** TODO 1. MCP Gateway Bridge
|
||||
*** TODO Build a Lisp-native client for the Model Context Protocol
|
||||
*** TODO Connect Passepartout to external tools and data sources
|
||||
|
||||
* PHASE: THE EVALUATION HARNESS (v0.8.0)
|
||||
:PROPERTIES:
|
||||
:ID: proj-eval-v0-8-0
|
||||
:END:
|
||||
Automated benchmarking to mathematically prove the agent's reasoning capabilities.
|
||||
|
||||
** TODO 0. SWE-Bench Harness
|
||||
*** TODO Automated pipeline that clones repositories and feeds GitHub issues
|
||||
*** TODO Track multi-step resolution trajectory, run tests, and score success
|
||||
|
||||
* PHASE: SOTA PARITY (v1.0.0)
|
||||
:PROPERTIES:
|
||||
:ID: proj-sota-v1-0-0
|
||||
:END:
|
||||
Feature-complete agent competitive with commercial agents. All borrowed concepts reimplemented in pure Lisp.
|
||||
|
||||
All features from v0.2.0 through v0.8.0 combined, verified, and tested end-to-end.
|
||||
|
||||
| Area | Parity Target |
|
||||
|------|--------------|
|
||||
| Self-improvement | Claude Code self-debug |
|
||||
| Planning | ULTRAPLAN equivalent |
|
||||
| Tool ecosystem | 10+ cognitive tools |
|
||||
| Context window | Semantic search + scope segmentation |
|
||||
| Safety | 6 Policy invariants + formal verification |
|
||||
| Multi-step tasks | Task trees with terminal states |
|
||||
| Code editing | Full file read/write via org manipulation |
|
||||
| Memory | Vector recall in org-object |
|
||||
| Emacs integration | Full org-mode control (exceeds Claude Code) |
|
||||
| Autonomy | 100% local capable (exceeds Claude Code) |
|
||||
|
||||
* PHASE: LISP MACHINE EMERGENCE (v2.0.0)
|
||||
:PROPERTIES:
|
||||
:ID: proj-lisp-v2-0-0
|
||||
:END:
|
||||
From Lisp-using agent to true Lisp machine. Agent IS the Emacs process.
|
||||
|
||||
** TODO Lish: Lisp editor as Org-mode IDE
|
||||
- Org-babel for interactive Lisp evaluation
|
||||
- Full REPL in TUI
|
||||
- No bridge needed — direct memory access
|
||||
** TODO Lish: Shell replacement
|
||||
- Lisp-based shell that speaks plists
|
||||
- Org-mode buffers as file system
|
||||
- No bash dependency
|
||||
|
||||
* PHASE: NEUROSYMBOLIC MATURITY (v3.0.0)
|
||||
:PROPERTIES:
|
||||
:ID: proj-neuro-v3-0-0
|
||||
:END:
|
||||
Deterministic planner takes the wheel. LLM relegated to semantic translation.
|
||||
|
||||
** TODO Deterministic planner
|
||||
- Planner as pure Lisp function
|
||||
- No LLM needed for scheduling
|
||||
- Generates task graphs without probabilistic inference
|
||||
** TODO Self-correcting gates
|
||||
- Gates learn from false positives (user override patterns)
|
||||
- Adaptive threshold adjustment
|
||||
|
||||
* PHASE: AI STACK INTERNALIZED (v4.0.0)
|
||||
:PROPERTIES:
|
||||
:ID: proj-ai-v4-0-0
|
||||
:END:
|
||||
The agent understands its own weights. No external inference.
|
||||
|
||||
** TODO Llama.cpp in Lisp
|
||||
- FFI binding to llama.cpp
|
||||
- No Python subprocess
|
||||
- Pure Common Lisp inference
|
||||
** TODO Weights as sexps
|
||||
- Neural weights represented as Lisp data structures
|
||||
- Homoiconic model introspection
|
||||
|
||||
* PHASE: TRUE AGENCY (v5.0.0)
|
||||
:PROPERTIES:
|
||||
:ID: proj-agency-v5-0-0
|
||||
:END:
|
||||
World models, temporal reasoning, goal persistence across restarts.
|
||||
|
||||
** TODO World models
|
||||
- Agent builds predictive models of user behavior
|
||||
- Project dynamics awareness
|
||||
- System state forecasting
|
||||
** TODO Temporal reasoning
|
||||
- Scheduling and deadline awareness
|
||||
- Elapsed duration tracking
|
||||
- Calendar integration
|
||||
** TODO Goal persistence
|
||||
- Goals survive restarts
|
||||
- Long-term projects tracked in org-objects
|
||||
- Cross-session continuity
|
||||
|
||||
* PHASE: EVOLUTIONARY ROADMAP (Previous — Superseded by Critical Analysis)
|
||||
:PROPERTIES:
|
||||
:ID: proj-old-roadmap
|
||||
:END:
|
||||
Superseded by the critical analysis-informed roadmap above (v0.2.0 through v5.0.0). This section kept for historical reference.
|
||||
|
||||
** TODO v0.1.0: The Autonomous Foundation (Current Release) — Now COMPLETE
|
||||
** TODO v1.0.0 (Phase 2.5): The Verified Wrapper (SOTA Parity) — Now v1.0.0
|
||||
** TODO v2.0.0 (Phase 3): Cannibalizing the Toolchain — Now v2.0.0
|
||||
** TODO v3.0.0 (Phase 4): True Symbolic Determinism — Now v3.0.0
|
||||
|
||||
* PHASE: FOUNDATION (Complete)
|
||||
** DONE Draft Swank/Socket communication protocol between CL and Emacs
|
||||
:PROPERTIES:
|
||||
:CREATED: [2026-03-22 Sun 14:00]
|
||||
:ASSIGNED: Agent
|
||||
:END:
|
||||
** DONE Implement core Perceive-Think-Act loop in Common Lisp
|
||||
:PROPERTIES:
|
||||
:CREATED: [2026-03-22 Sun 14:00]
|
||||
:ASSIGNED: Agent
|
||||
:END:
|
||||
** DONE Implement Persistent Object-Store for Org entities in CL
|
||||
:PROPERTIES:
|
||||
:CREATED: [2026-03-22 Sun 16:30]
|
||||
:ASSIGNED: Agent
|
||||
:END:
|
||||
** DONE Implement LLM Connector (Probabilistic Engine) in CL Daemon
|
||||
:PROPERTIES:
|
||||
:CREATED: [2026-03-22 Sun 17:30]
|
||||
:ASSIGNED: Agent
|
||||
:END:
|
||||
** DONE Design Deterministic Engine Heuristics (Lisp logic over Memory)
|
||||
:PROPERTIES:
|
||||
:CREATED: [2026-03-22 Sun 17:30]
|
||||
:END:
|
||||
** DONE Achieve Phase 3: The Self-Editing Kernel
|
||||
:PROPERTIES:
|
||||
:CREATED: [2026-03-23 Mon 16:30]
|
||||
:END:
|
||||
- Jailing & Sandboxing implemented
|
||||
- Org-Native Skill Standard established
|
||||
- Telemetry & Introspection API active
|
||||
|
||||
* PHASE: THE SOVEREIGN BOUNDARY (Core vs Skills Refactor)
|
||||
:PROPERTIES:
|
||||
:ID: proj-autonomous-boundary
|
||||
:END:
|
||||
Slim down the passepartout microharness by moving non-essential cognitive functions to hot-reloadable user-space skills.
|
||||
|
||||
** DONE Extract LLM Provider Routing to a Skill (neuro.lisp)
|
||||
** DONE Extract Vector Embedding Algorithms to a Skill (embedding.lisp)
|
||||
CLOSED: [2026-04-12 Sun 14:10]
|
||||
:PROPERTIES:
|
||||
:ID: extract-embedding-skill
|
||||
:END:
|
||||
- Created `org-skill-embedding.org`.
|
||||
- Moved logic to `src/embedding-logic.lisp` via tangling.
|
||||
- Updated `system-definition.org`.
|
||||
** DONE Extract Sparse Tree Context Pruning Strategies to a Skill (context.lisp)
|
||||
CLOSED: [2026-04-12 Sun 14:25]
|
||||
:PROPERTIES:
|
||||
:ID: extract-context-skill
|
||||
:END:
|
||||
- Created `org-skill-peripheral-vision.org`.
|
||||
- Moved logic to `src/context-logic.lisp` via tangling.
|
||||
- Updated `system-definition.org`.
|
||||
** DONE Implement `org-skill-peripheral-vision` (Moving embedding logic out of core)
|
||||
CLOSED: [2026-04-12 Sun 14:25]
|
||||
:PROPERTIES:
|
||||
:ID: impl-peripheral-vision
|
||||
:END:
|
||||
** DONE Implement communication protocol Schema Validation (Prevent reader macro injection in communication.lisp)
|
||||
CLOSED: [2026-04-12 Sun 14:45]
|
||||
:PROPERTIES:
|
||||
:ID: communication-protocol-schema-validation
|
||||
:END:
|
||||
- Created `org-skill-protocol-validator.org`.
|
||||
- Integrated `validate-communication-protocol-schema` into `communication.org`.
|
||||
- Added `protocol-validator.lisp` to system definition.
|
||||
** DONE Implement Pluggable communication protocol Integrity Hashing (Core interface, Skill-based algorithms)
|
||||
CLOSED: [2026-04-12 Sun 15:15]
|
||||
:PROPERTIES:
|
||||
:ID: communication-protocol-integrity-hashing
|
||||
:END:
|
||||
- Integrated HMAC-SHA256 (`ironclad:make-mac`) in `literate/communication.org`.
|
||||
** DONE Implement Native Lisp Merkle-Tree Versioning (Short-term undo buffer in memory.lisp)
|
||||
CLOSED: [2026-04-12 Sun 19:15]
|
||||
** DONE Performance: Implement Copy-on-Write (CoW) or Persistent Data Structures for Memory
|
||||
CLOSED: [2026-04-12 Sun 19:15]
|
||||
** DONE Feature: Implement Latent Reflection (Proactive Gardening) using heartbeat idle cycles
|
||||
CLOSED: [2026-04-12 Sun 19:15]
|
||||
** DONE Simplification: Refactor Cognitive Cycle into a Unified Reactive Signal Pipeline
|
||||
CLOSED: [2026-04-12 Sun 19:15]
|
||||
** DONE Resilience: Implement Micro-Rollbacks for the Immune System
|
||||
CLOSED: [2026-04-12 Sun 19:15]
|
||||
** DONE Implement `org-skill-memory-archivist` (Long-term IPFS checkpointing and P2P sync)
|
||||
CLOSED: [2026-04-12 Sun 19:15]
|
||||
** DONE Implement True Lisp Sandboxing (eval-safe mechanism in core and policy in skills)
|
||||
CLOSED: [2026-04-12 Sun 19:15]
|
||||
** DONE Decouple Vendor Logic from Probabilistic Engine (Move Google/Anthropic/OpenAI to Skills)
|
||||
CLOSED: [2026-04-12 Sun 19:15]
|
||||
** DONE Component IV: Comprehensive Core Skill Audit (Review all 39 skills)
|
||||
CLOSED: [2026-04-12 Sun 19:45]
|
||||
:PROPERTIES:
|
||||
:ID: core-skill-audit-task
|
||||
:END:
|
||||
|
||||
** DONE Consolidation I: Unified LLM Gateway (Anthropic, Gemini, Groq, OpenAI, etc.)
|
||||
** DONE Consolidation II: Credentials Vault (Secure Enclave & Masked Logging)
|
||||
** DONE Consolidation III: Homoiconic Memory (Unified Grammar, Bridge, & ID Generation)
|
||||
** DONE Consolidation IV: State Persistence Layer (Unified Local & IPFS Checkpointing)
|
||||
** DONE Consolidation V: Event Orchestrator (Unified Cron, Hooks, & Cognitive Routing)
|
||||
** DONE Consolidation VI: Task Orchestrator (Task Integrity, Delegation, & Consensus)
|
||||
CLOSED: [2026-04-11 Sat 13:45]
|
||||
:PROPERTIES:
|
||||
:ID: task-orchestrator-consolidation
|
||||
:END:
|
||||
- Implemented Parallel Multi-Backend Consensus in neuro.lisp.
|
||||
- Implemented Task Integrity (GTD semantics) in symbolic.lisp.
|
||||
- Integrated Consensus Gate and Delegation hooks in core.lisp.
|
||||
- Verified with new task-orchestrator-tests.lisp.
|
||||
** DONE Implement Unified Envelope Architecture & Channel-Awareness
|
||||
CLOSED: [2026-04-20 Mon 13:20]
|
||||
- Removed specialized :CHAT type; reverted to semantic :REQUEST/:EVENT protocol.
|
||||
- Decoupled routing metadata into a :META envelope (SOURCE, SESSION-ID).
|
||||
- Updated TUI, Emacs, and CLI gateways to use the unified protocol.
|
||||
- Verified end-to-end loop with TUI; kernel correctly routes responses back to origin interface.
|
||||
- Achieved "Equality of Clients" mandate.
|
||||
** DONE Full review of passepartout's harness
|
||||
CLOSED: [2026-05-01 Fri 12:46]
|
||||
:PROPERTIES:
|
||||
:CREATED: [2026-04-13 Mon 13:30]
|
||||
:ASSIGNED: Agent
|
||||
:END:
|
||||
- [X] Audit terminology: Replaced OACP with "communication protocol" workspace-wide.
|
||||
- [X] Audit boot sequence: Synchronized loader with `org-skill-policy.org`.
|
||||
- [X] Implement Unified Envelope (Channel-Aware Routing).
|
||||
- [X] Audit core Perceive-Think-Act loop.
|
||||
- [X] Verified protocol framing and reader jailing (`*read-eval* nil`).
|
||||
- [X] Refactored `loop.org` for literate granularity and configuration externalization.
|
||||
- [X] Improved error handling (restricted rollback) and added graceful shutdown.
|
||||
- [X] **FIXED:** Implemented symbolic guard check in `act-gate` via Dispatcher skill refactoring.
|
||||
- [X] **FIXED:** Harness `deterministic-verify` now correctly respects skill triggers.
|
||||
- [X] **FIXED:** Resolved TUI crash by removing `--non-interactive` from `passepartout.sh` and adding defensive coordinate handling.
|
||||
- [X] **VERIFIED:** Confirmed bidirectional TUI communication and signed off v0.2.0.
|
||||
- [X] Ensure alignment with System Policy and Engineering Standards.
|
||||
- [X] Restored structural integrity by fixing `manifest.org` loading sequence.
|
||||
|
||||
** TODO Wake up the Scribe (Implement autonomous weekly Journal-to-Ledger distillation in org-skill-scribe.org)
|
||||
** TODO Implement `org-skill-lisp-repair` (Self-correcting syntax gate for Deterministic Engine)
|
||||
CLOSED: [2026-04-11 Sat 15:10]
|
||||
:PROPERTIES:
|
||||
:ID: lisp-repair-gate
|
||||
:END:
|
||||
- Implemented asynchronous, event-driven repair logic.
|
||||
- Decoupled core from repair logic (emits `:syntax-error` event).
|
||||
- Proven via lisp-repair-tests.lisp (Asynchronous flow verified).
|
||||
** DONE Implement `org-skill-formal-verification` (Prove safety of high-impact actions)
|
||||
CLOSED: [2026-04-11 Sat 18:15]
|
||||
:PROPERTIES:
|
||||
:ID: formal-verification-task
|
||||
:END:
|
||||
- Implemented `org-skill-formal-verification.org`.
|
||||
- Created Lisp-Native Symbolic Prover for security invariants.
|
||||
- Implemented `path-confinement` invariant (restricted to memex root).
|
||||
- Implemented `no-network-exfil` invariant (blocking nc, ssh, etc).
|
||||
- Verified with `formal-verification-tests.lisp`.
|
||||
|
||||
* PHASE: DETERMINISTIC ENGINE REFINEMENT
|
||||
** DONE Verify Autonomous Self-Fix Loop
|
||||
CLOSED: [2026-04-11 Sat 14:20]
|
||||
:PROPERTIES:
|
||||
:CREATED: [2026-03-23 Mon 16:30]
|
||||
:END:
|
||||
- Proven repair capability via self-fix-tests.lisp.
|
||||
- Verified surgical code patching and hot-reloading.
|
||||
- Documentation and RCA complete.
|
||||
** DONE Implement "Planning Mode" (Deterministic Engine Dispatcher) for Complex Actions
|
||||
CLOSED: [2026-04-11 Sat 15:30]
|
||||
:PROPERTIES:
|
||||
:CREATED: [2026-04-01 Wed 17:00]
|
||||
:END:
|
||||
- Implemented `dispatcher-check` interceptor in `symbolic.lisp`.
|
||||
- Created `org-skill-dispatcher.org` for flight plan serialization.
|
||||
- Verified asynchronous Org-native approval loop via `dispatcher-tests.lisp`.
|
||||
** DONE Implement Authorization Gate (communication protocol) for "Planning Mode"
|
||||
CLOSED: [2026-04-11 Sat 15:30]
|
||||
:PROPERTIES:
|
||||
:CREATED: [2026-04-01 Wed 17:00]
|
||||
:END:
|
||||
- Integrated with Org-mode state transitions (`PLAN` -> `APPROVED`).
|
||||
- Leveraged Memory event bus for asynchronous re-injection.
|
||||
|
||||
** DONE Refactor Architecture Terminology (Associative -> Probabilistic, Deliberate -> Deterministic)
|
||||
CLOSED: [2026-04-12 Sun 21:00]
|
||||
:PROPERTIES:
|
||||
:ID: terminology-refactor-task
|
||||
:END:
|
||||
- Updated codebase-wide terminology to use Probabilistic/Deterministic Engines.
|
||||
- Replaced System 1/2 with Probabilistic/Deterministic Engines respectively.
|
||||
|
||||
** DONE Refactor org-skill-policy.org: Concrete Invariants, Conflict Hierarchy, and Auditable Gate
|
||||
CLOSED: [2026-04-22 Wed 11:50]
|
||||
:PROPERTIES:
|
||||
:ID: policy-refactor-concrete-invariants
|
||||
:END:
|
||||
- Added explicit Override Hierarchy (Transparency > Autonomy > Bloat > Mentorship > Sustainability).
|
||||
- Implemented `policy-check-transparency`: blocks user-facing actions without :explanation.
|
||||
- Implemented `policy-check-autonomy`: flags proprietary domain references as autonomy debt.
|
||||
- Implemented `policy-check-bloat`: warns on :create-skill actions exceeding size threshold.
|
||||
- Implemented `policy-check-mentorship`: blocks high-impact actions missing :mentorship-note.
|
||||
- Implemented `policy-check-sustainability`: logs cloud-provider usage as sustainability debt.
|
||||
- Implemented `policy-explain`: formats auditable rationale for every policy decision.
|
||||
- Implemented `policy-find-engineering-standards-gate`: robust cross-package search for standards skill.
|
||||
- Hardened `policy-deterministic-gate`: never returns NIL silently; always returns action or auditable log.
|
||||
- Raised skill priority from 100 to 500 to ensure it runs before other deterministic gates.
|
||||
|
||||
** DONE Add Invariant 6 (Modularity) and Harness Boundary Contract to Policy/Manifest
|
||||
CLOSED: [2026-04-22 Wed 12:10]
|
||||
:PROPERTIES:
|
||||
:ID: policy-modularity-invariant
|
||||
:END:
|
||||
- Added Modularity as Invariant 6 in `org-skill-policy.org`: general life principle that complexity must live at the edges.
|
||||
- Implemented `policy-check-modularity`: blocks modifications to protected core paths unless `:modularity-justification` is provided.
|
||||
- Defined `*modularity-protected-paths*` as project-configurable variable (defaults: harness/, passepartout.asd).
|
||||
- Updated Override Hierarchy to include Modularity between Bloat and Mentorship.
|
||||
- Added Harness Boundary Contract section to `harness/manifest.org` documenting primary boundary files and generated artifacts.
|
||||
- Converted checkbox sub-tasks to hierarchical TODO headlines per GTD standard.
|
||||
|
||||
** DONE Implement `org-skill-lisp-validator` (3-phase deterministic validation gate)
|
||||
CLOSED: [2026-04-22 Wed 12:30]
|
||||
:PROPERTIES:
|
||||
:ID: lisp-validator-implementation
|
||||
:END:
|
||||
- Created 3-phase validation pipeline: Structural (O(n) paren scanner), Syntactic (reader with *read-eval* nil), Semantic (whitelist AST walk).
|
||||
- Implemented `lisp-validator-validate` returning structured plists for machine parsing.
|
||||
- Exposed `:validate-lisp` cognitive tool for Probabilistic Engine self-correction.
|
||||
- Replaced `validate-lisp-syntax` in `harness/skills.org` with delegation to the validator.
|
||||
- Added mandatory validation rule to Probabilistic Engine system prompt in `harness/reason.org`.
|
||||
- Fixed paren balance and `return-from` compilation errors in org source; tangled and validated in SBCL.
|
||||
|
||||
** DONE Fix Skill Loader to Respect `:tangle` Blocks and Eliminate Circular Dependency
|
||||
CLOSED: [2026-04-22 Wed 12:45]
|
||||
:PROPERTIES:
|
||||
:ID: skill-loader-tangle-fix
|
||||
:END:
|
||||
- Updated `load-skill-from-org` in `harness/skills.org` to only collect blocks with `:tangle` directives pointing to runtime `.lisp` files, excluding `tests/` and `test/` paths.
|
||||
- Added fallback to `validate-lisp-syntax` so it uses a basic reader check when `lisp-validator-validate` is not yet loaded (breaks circular harness->skill dependency).
|
||||
- Verified full boot: 13/13 skills loaded successfully into SBCL, including `skill-lisp-validator` at priority 900 and `skill-policy` at priority 500.
|
||||
|
||||
* TRACK: SECURITY & CONTAINMENT (The 5-Vector Dispatcher Matrix)
|
||||
** DONE Implement Path-Based Scoping for File Writes (DNA/State vs Work)
|
||||
CLOSED: [2026-04-12 Sun 15:15]
|
||||
:PROPERTIES:
|
||||
:ID: path-based-scoping
|
||||
:END:
|
||||
- Implemented as `path-confinement` invariant in `org-skill-formal-verification.org`.
|
||||
** DONE Implement Network Exfiltration Gate (Intercept generic HTTP requests)
|
||||
CLOSED: [2026-04-12 Sun 15:15]
|
||||
:PROPERTIES:
|
||||
:ID: network-exfiltration-gate
|
||||
:END:
|
||||
- Implemented as `no-network-exfil` invariant in `org-skill-formal-verification.org`.
|
||||
** TODO Implement Secret Exposure Gate (Intercept reads to .env, keys)
|
||||
|
||||
* TRACK: INTELLIGENCE & ACTUATION (The Engines)
|
||||
** DONE Verify individual provider track (Anthropic, Gemini, Groq, OpenAI, OpenRouter, Ollama)
|
||||
CLOSED: [2026-04-11 Sat 15:45]
|
||||
:PROPERTIES:
|
||||
:ID: provider-verification-track
|
||||
:END:
|
||||
- Added unit tests for each provider in `llm-gateway-tests.lisp`.
|
||||
- Mocked `dex:post` to verify JSON payload formatting and response parsing.
|
||||
- Implemented robust `get-nested` helper to handle various provider structures.
|
||||
- Integrated `llm-gateway` and `credentials-vault` into `passepartout.asd`.
|
||||
** TODO Verify org-skill-shell-actuator formal safety harnesses
|
||||
** DONE Build Playwright-Python Bridge for high-fidelity browsing
|
||||
CLOSED: [2026-04-11 Sat 18:30]
|
||||
:PROPERTIES:
|
||||
:ID: playwright-bridge-task
|
||||
:END:
|
||||
- Created `scripts/browser-bridge.py` (Playwright wrapper).
|
||||
- Implemented `org-skill-playwright.org`.
|
||||
- Registered `:browser` cognitive tool (JS-rendering, text extraction, screenshots).
|
||||
- Updated `Dockerfile` with Python/Playwright dependencies.
|
||||
- Verified with `playwright-tests.lisp`.
|
||||
|
||||
|
||||
* TRACK: COMMUNICATION & INTERFACES
|
||||
** DONE Implement org-skill-gateway-telegram
|
||||
CLOSED: [2026-04-11 Sat 16:15]
|
||||
:PROPERTIES:
|
||||
:ID: gateway-telegram-task
|
||||
:END:
|
||||
- Implemented `org-skill-gateway-telegram.org`.
|
||||
- Added automated background polling for Telegram GetUpdates.
|
||||
- Implemented `:telegram` actuator for outbound responses.
|
||||
- Refactored `org-skill-chat` to be channel-aware.
|
||||
- Verified with `gateway-telegram-tests.lisp`.
|
||||
** DONE Implement org-skill-gateway-signal
|
||||
CLOSED: [2026-04-11 Sat 16:50]
|
||||
:PROPERTIES:
|
||||
:ID: gateway-signal-task
|
||||
:END:
|
||||
- Implemented `org-skill-gateway-signal.org` (signal-cli wrapper).
|
||||
- Added background polling for `signal-cli receive --json`.
|
||||
- Implemented `:signal` actuator for outbound responses.
|
||||
- Updated `org-skill-chat` to support Signal channel.
|
||||
- Verified with `gateway-signal-tests.lisp`.
|
||||
** DONE Implement org-skill-gateway-matrix
|
||||
CLOSED: [2026-04-11 Sat 17:15]
|
||||
:PROPERTIES:
|
||||
:ID: gateway-matrix-task
|
||||
:END:
|
||||
- Implemented `org-skill-gateway-matrix.org` (Client-Server API).
|
||||
- Added background polling for `/sync` with token persistence.
|
||||
- Implemented `:matrix` actuator for `m.room.message` delivery.
|
||||
- Updated `org-skill-chat` to support Matrix channel and room IDs.
|
||||
- Verified with `gateway-matrix-tests.lisp`.
|
||||
|
||||
* TRACK: DEPLOYMENT & INFRASTRUCTURE
|
||||
** DONE Create Dockerfile and docker-compose.yml for containerized setup
|
||||
CLOSED: [2026-04-11 Sat 17:30]
|
||||
:PROPERTIES:
|
||||
:ID: docker-infra-task
|
||||
:END:
|
||||
- Created `Dockerfile` (Debian-based, SBCL + Quicklisp + signal-cli).
|
||||
- Created `docker-compose.yml` with host-volume mapping for memex.
|
||||
- Created `docs/deployment.org` guide.
|
||||
** TODO Create Bare Metal installation scripts/playbooks
|
||||
** TODO Create LXC (Linux Containers) template/guide
|
||||
** TODO Create VM Vagrantfiles/Cloud-init configs
|
||||
|
||||
* TRACK: MAINTENANCE & HYGIENE
|
||||
** TODO [RECURRING: Monthly] Review and test Infrastructure Dependency Upgrades
|
||||
:PROPERTIES:
|
||||
:ID: monthly-infra-audit
|
||||
:REPEAT_TO_STATE: TODO
|
||||
:END:
|
||||
*** TODO Check for new Debian security patches (`apt-get update` check)
|
||||
*** TODO Check for new `signal-cli` releases (compare vs v0.14.0)
|
||||
*** TODO Check for new Quicklisp distribution (monthly snapshot)
|
||||
*** TODO Verification: Update `Dockerfile`, run `docker-compose build --no-cache`, and execute full test suite
|
||||
*** TODO If all tests pass, commit updated `Dockerfile` and `.asd` dependencies
|
||||
|
||||
* TRACK: COMMUNITY & DOCS
|
||||
** TODO Write Quickstart Guide
|
||||
** TODO Write Skill Creation Guide
|
||||
** TODO Write Architecture Deep-Dive
|
||||
** TODO Clean up GitHub repository structure and add CI/CD
|
||||
** TODO Create Marketing Material (Landing page copy, diagrams)
|
||||
** TODO Draft Release Plan checklist
|
||||
|
||||
* SUB-PROJECT: THE BOOT SEQUENCE (skills.lisp)
|
||||
:PROPERTIES:
|
||||
:ID: proj-skill-boot-sequence
|
||||
:END:
|
||||
** DONE Refactor `skills.lisp` into a Micro-Loader (Harness)
|
||||
CLOSED: [2026-04-12 Sun 19:10]
|
||||
** DONE Implement Topological Sort based on `#+DEPENDS_ON:` tags
|
||||
CLOSED: [2026-04-12 Sun 15:15]
|
||||
:PROPERTIES:
|
||||
:ID: topological-sort-skills
|
||||
:END:
|
||||
- Implemented in `literate/skills.org`.
|
||||
** DONE Enforce `org-skill-system-invariants` as the mandatory Gateway Skill (Loaded first)
|
||||
CLOSED: [2026-04-12 Sun 15:15>
|
||||
:PROPERTIES:
|
||||
:ID: enforce-mandatory-skill
|
||||
:END:
|
||||
- Enforced in `initialize-all-skills` in `literate/skills.org`.
|
||||
** DONE Formalize the "Minimal Boot Set" (Router, Vision, Steward, Actuator)
|
||||
CLOSED: [2026-04-12 Sun 19:10>
|
||||
@@ -1,61 +0,0 @@
|
||||
#+TITLE: Passepartout User Manual
|
||||
#+AUTHOR: Agent
|
||||
#+STARTUP: content
|
||||
#+FILETAGS: :docs:manual:
|
||||
|
||||
* Introduction
|
||||
Welcome to the Passepartout User Manual. This guide provides the operational knowledge required to manage your sovereign Lisp Machine and its neural skills.
|
||||
|
||||
* System Architecture
|
||||
Passepartout follows a "Purified Kernel" model. The core harness handles essential I/O, while all high-level logic resides in sovereign skills.
|
||||
|
||||
** XDG Directory Standard
|
||||
To ensure POSIX compliance, Passepartout stores its files in standard Linux locations:
|
||||
|
||||
| Type | Path | Purpose |
|
||||
| :--- | :--- | :--- |
|
||||
| **Config** | `~/.config/passepartout/` | User settings, `.env` secrets, and provider registry. |
|
||||
| **Data** | `~/.local/share/passepartout/` | Tangled Lisp artifacts and the compiled engine. |
|
||||
| **State** | `~/.local/state/passepartout/` | Brain snapshots, logs, and Merkle-memory. |
|
||||
| **Bin** | `~/.local/bin/passepartout` | The global CLI shim. |
|
||||
|
||||
* Command Reference
|
||||
|
||||
** `passepartout setup`
|
||||
The interactive configuration wizard. Use this to:
|
||||
- Define your identity and the Agent's name.
|
||||
- Register LLM providers (Ollama, Groq, Anthropic, etc.).
|
||||
- The wizard automatically splits sensitive tokens into `~/.config/passepartout/.env`.
|
||||
|
||||
** `passepartout gateway link <platform> <token>`
|
||||
Connects Passepartout to external communication gateways.
|
||||
- **Example:** `passepartout gateway link telegram <my_bot_token>`
|
||||
- **Example:** `passepartout gateway unlink telegram` to disable
|
||||
- **Example:** `passepartout gateway list` to see status
|
||||
|
||||
** `passepartout doctor`
|
||||
Your primary diagnostic tool. Run this if the system feels sluggish or fails to boot. It verifies:
|
||||
- External dependencies (sbcl, git, socat).
|
||||
- XDG directory existence and permissions.
|
||||
- LLM connectivity.
|
||||
|
||||
** `passepartout tui`
|
||||
Launches the native Lisp Terminal User Interface.
|
||||
- **Highlighting:** Semantic color-coding for Lisp and Org syntax.
|
||||
- **Scrolling:** Use `PgUp`/`PgDn` to navigate history.
|
||||
- **Exit:** Type `/exit` or `Ctrl+C` to close.
|
||||
|
||||
* Configuration Strategy
|
||||
Passepartout uses a **Hybrid Storage** model for maximum security and flexibility.
|
||||
|
||||
** 1. Secrets (`.env`)
|
||||
Found in `~/.config/passepartout/.env`. This file stores raw API tokens. It is never automatically read by the Lisp structural parser to prevent accidental leakage into logs.
|
||||
|
||||
** 2. Metadata (`providers.lisp`)
|
||||
Found in `~/.config/passepartout/providers.lisp`. This stores non-sensitive configuration like model names, base URLs, and user preferences as native Lisp S-expressions.
|
||||
|
||||
* Troubleshooting
|
||||
If `passepartout doctor` reports a `FAIL`:
|
||||
1. Check that your `PATH` includes `/usr/bin` and `/usr/local/bin`.
|
||||
2. Ensure `sbcl` is installed.
|
||||
3. If LLM connectivity fails, verify your API key in `~/.config/passepartout/.env`.
|
||||
90
docs/ARCHITECTURE.org
Normal file
90
docs/ARCHITECTURE.org
Normal file
@@ -0,0 +1,90 @@
|
||||
#+TITLE: Passepartout Architecture
|
||||
#+AUTHOR: Agent
|
||||
#+STARTUP: content
|
||||
|
||||
* The Four Quadrants
|
||||
|
||||
Passepartout divides cognition along two axes: **Foreground vs Background** (initiated by the user vs running autonomously) and **Probabilistic vs Deterministic** (LLM-driven vs pure Lisp logic).
|
||||
|
||||
| | Probabilistic (LLM) | Deterministic (Lisp) |
|
||||
|----------------|--------------------|---------------------|
|
||||
| **Foreground** | Chat responses, task execution, code generation | Shell execution, file I/O, safety gates, dispatcher checks |
|
||||
| **Background** | Scribe distillation, vector embedding, autonomous decisions | Heartbeat, cron jobs, memory auto-save, gateway polling |
|
||||
|
||||
The Probabilistic engine proposes. The Deterministic engine verifies and executes. No proposal from the LLM touches a file, runs a command, or sends a message without passing through at least one deterministic gate.
|
||||
|
||||
* Code Map
|
||||
|
||||
The project is organized into ~org/~ (source of truth) and ~lisp/~ (generated by tangle).
|
||||
|
||||
** Core pipeline (loaded by ASDF, committed to git)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| ~org/core-defpackage.org~ | Package definition and export list |
|
||||
| ~org/core-skills.org~ | Skill engine: ~defskill~ macro, topological sorter, jailed loading |
|
||||
| ~org/core-communication.org~ | Framed TCP protocol, actuator registry, daemon server |
|
||||
| ~org/core-memory.org~ | ~memory-object~ struct, Merkle hashing, snapshots, persistence |
|
||||
| ~org/core-context.org~ | Foveal-peripheral rendering, context assembly for LLM |
|
||||
| ~org/core-loop-perceive.org~ | Stage 1: normalize raw signals into pipeline format |
|
||||
| ~org/core-loop-reason.org~ | Stage 2: LLM proposal + deterministic verification |
|
||||
| ~org/core-loop-act.org~ | Stage 3: dispatch approved actions to actuators |
|
||||
| ~org/core-loop.org~ | Orchestration: process-signal, heartbeat, main entry point |
|
||||
| ~org/system-diagnostics.org~ | Boot-time health check, doctor CLI |
|
||||
|
||||
** Skills (loaded at runtime by the skill engine)
|
||||
|
||||
| Category | Files | Purpose |
|
||||
|----------|-------|---------|
|
||||
| **gateway-** | ~gateway-cli~, ~gateway-llm~, ~gateway-manager~, ~gateway-provider~, ~gateway-tui~ | External communication channels |
|
||||
| **security-** | ~security-dispatcher~, ~security-policy~, ~security-permissions~, ~security-vault~, ~security-validator~ | Safety and authorization |
|
||||
| **programming-** | ~programming-lisp~, ~programming-org~, ~programming-standards~, ~programming-literate~, ~programming-repl~ | Lisp and Org tooling |
|
||||
| **system-** | ~system-config~, ~system-archivist~, ~system-self-improve~, ~system-memory~, ~system-actuator-shell~, ~system-event-orchestrator~ | Background services |
|
||||
|
||||
* Pipeline Flow
|
||||
|
||||
Every signal moves through three stages:
|
||||
|
||||
```
|
||||
Signal → Perceive (normalize) → Reason (think + verify) → Act (dispatch)
|
||||
```
|
||||
|
||||
The signal is a plist: ~(:TYPE :EVENT :META (...) :PAYLOAD (:SENSOR :user-input :TEXT "..."))~
|
||||
|
||||
1. **Perceive** normalizes raw input from any gateway into a uniform signal
|
||||
2. **Reason** calls the LLM to generate a proposal, then runs the proposal through all registered deterministic gates (sorted by priority). If a gate rejects the proposal, the rejection trace feeds back to the LLM for self-correction (up to 3 retries)
|
||||
3. **Act** dispatches the approved action to the registered actuator (~:cli~, ~:tool~, ~:system~, ~:shell~, ~:telegram~, ~:signal~)
|
||||
|
||||
Each stage can produce feedback signals that loop back to Perceive (e.g., a tool-execute action produces a ~:tool-output~ event that becomes the next perception).
|
||||
|
||||
** Depth limiting
|
||||
|
||||
A depth counter prevents infinite loops. If a signal's depth exceeds 10, it is silently dropped. This is the circuit breaker for runaway recursive cycles.
|
||||
|
||||
* Skill Lifecycle
|
||||
|
||||
1. **Discovery:** ~skill-initialize-all~ scans the skills directory, globs for ~*.lisp~ files (excluding ~core-*~ files which are loaded by ASDF)
|
||||
2. **Sorting:** ~skill-topological-sort~ orders skills by their ~#+DEPENDS_ON:~ declarations
|
||||
3. **Loading:** Each skill is loaded into a jailed package (~passepartout.skills.<skill-name>~). The loader removes ~in-package~ forms, evaluates the remaining code in the jailed package, and exports symbols matching the skill's short name to ~passepartout~
|
||||
4. **Registration:** The skill's ~defskill~ call creates a ~skill~ struct in ~*skill-registry*~, registering its trigger function, probabilistic prompt generator, deterministic gate, and system-prompt augment
|
||||
5. **Triggering:** On each cognitive cycle, ~skill-triggered-find~ iterates the registry and returns the highest-priority skill whose trigger matches the context
|
||||
6. **Hot-reload:** A skill can be replaced at runtime by loading a new version into its jailed package — no restart needed
|
||||
|
||||
* Protocol Format
|
||||
|
||||
All communication between the daemon and its gateways (TUI, CLI, Emacs) uses length-prefixed plists over TCP:
|
||||
|
||||
```
|
||||
00002C(:TYPE :EVENT :PAYLOAD (:ACTION :handshake :VERSION "0.2.0"))
|
||||
```
|
||||
|
||||
The 6-character hex prefix encodes the payload length. The payload is a ~prin1~-serialized plist. ~*read-eval*~ is bound to nil on the receiving end to prevent code injection.
|
||||
|
||||
** Standard message envelope:
|
||||
|
||||
| Key | Value | Meaning |
|
||||
|-----|-------|---------|
|
||||
| ~:TYPE~ | ~:REQUEST~, ~:EVENT~, ~:RESPONSE~, ~:LOG~, ~:STATUS~ | Message category |
|
||||
| ~:META~ | plist | ~:SOURCE~, ~:SESSION-ID~, ~:reply-stream~ |
|
||||
| ~:PAYLOAD~ | plist | Action-specific data (~:SENSOR~, ~:ACTION~, ~:TEXT~) |
|
||||
| ~:DEPTH~ | integer | Recursion counter for loop prevention |
|
||||
@@ -1,6 +1,41 @@
|
||||
#+TITLE: Changelog
|
||||
#+STARTUP: content
|
||||
|
||||
* v0.2.1 — Rename, Safety, and Deployment (2026-05-02)
|
||||
This release renames the project to Passepartout, adds content-level safety gates, professionalizes deployment, and documents every function with full explanatory prose.
|
||||
|
||||
** Project Rename
|
||||
- **Passepartout:** Project renamed from OpenCortex to Passepartout. All files, packages, functions, and environment variables updated.
|
||||
- **Org/lisp split:** Source of truth lives in ~org/~, tangled to ~lisp/~. Core files committed, skills generated at configure time.
|
||||
- **31 org files:** Every file renamed to ~category-subject.org~ convention. Harness and skills unified under one directory.
|
||||
|
||||
** Safety
|
||||
- **Secret Exposure Gate:** Content scanning for API keys, PEM blocks, PGP keys, credentials, and tokens in all outgoing text.
|
||||
- **Path Protection:** File reads blocked for ~.env~, SSH keys, PEM/PGP, cloud configs, and credential stores.
|
||||
- **Shell Safety:** Destructive commands (~rm -rf /~, ~dd~, ~mkfs~, ~shred~) and injection patterns (backtick, ~$()~) blocked with timeout and output limits.
|
||||
- **Lisp Validation Gate:** Writes to ~.lisp~ and ~.org~ files validated for syntax errors before they reach disk.
|
||||
- **REPL Verification Lint:** Warns if defuns are written without REPL prototyping.
|
||||
|
||||
** Deployment
|
||||
- **Multi-distro:** Automatic detection of Debian vs Fedora, correct package names and managers.
|
||||
- **systemd service:** User-level auto-start on boot via ~passepartout install service~.
|
||||
- **Backup/Restore:** ~passepartout backup~ and ~passepartout restore~ commands.
|
||||
- **Docker:** Updated to ~debian:trixie-slim~, fixed build context.
|
||||
- **CI/CD:** GitHub Actions workflows for lint, test, and release. Gitea deploy workflow fixed.
|
||||
|
||||
** Engineering Process
|
||||
- **REPL-first Lifecycle:** Two-track workflow: Org-first for prose and tests, REPL-first for implementation. Every function prototyped in the REPL before reaching Org.
|
||||
- **Verification Loop:** Bouncer rejects bad lisp; rejection trace feeds back to LLM for self-correction.
|
||||
- **System-prompt-augment:** Skills can inject domain-specific mandates into the LLM prompt via ~:system-prompt-augment~.
|
||||
|
||||
** Documentation
|
||||
- **Literate Prose Restored:** Every Org file now has an Architectural Intent overview and explanatory prose before each function block, following the style established in the v0.1.0 era.
|
||||
- **AGENTS.md:** Thinned to a routing layer — the skill org files are authoritative.
|
||||
|
||||
** Contributors
|
||||
- **gitignore:** ~skills/*.lisp~ and ~tests/*.lisp~ as generated artifacts (source of truth is ~.org~).
|
||||
- **DeepSeek and NVIDIA NIM:** Added as LLM providers (OpenAI-compatible). Use ~DEEPSEEK_API_KEY~ and ~NVIDIA_API_KEY~ env vars.
|
||||
|
||||
* v0.2.0 - Interactive Refinement (2026-04-29)
|
||||
This release focuses on professionalizing the environment and enhancing the agent's structural capabilities.
|
||||
|
||||
|
||||
@@ -336,4 +336,125 @@ The long-term goal is a single =passepartout= binary that the user runs. It star
|
||||
|
||||
This stands in stark contrast to most AI agent systems, which require managing Python environments, npm packages, API keys, environment variables, and configuration files. OpenAI's agents SDK requires pip install, a Python environment, and external API access. OpenClaw requires Node.js, npm, and a plugin ecosystem that must be individually installed. LangChain requires a Python environment with dozens of dependencies that must be kept compatible.
|
||||
|
||||
Passepartout's dependency model is SBCL plus Quicklisp. Quicklisp loads libraries on demand from the internet, but caches them locally. A system with internet access can fetch any library it needs. A system without internet access uses only the libraries it has already loaded - and those are preserved in the cache. The agent does not require internet access to function after initial setup.
|
||||
Passepartout's dependency model is SBCL plus Quicklisp. Quicklisp loads libraries on demand from the internet, but caches them locally. A system with internet access can fetch any library it needs. A system without internet access uses only the libraries it has already loaded - and those are preserved in the cache. The agent does not require internet access to function after initial setup.
|
||||
|
||||
* Token Economics and Performance Advantage
|
||||
:PROPERTIES:
|
||||
:ID: design-token-economics
|
||||
:END:
|
||||
|
||||
This section analyzes how Passepartout's architectural decisions translate into token usage, latency, and cost versus competing agent designs (OpenClaw, Hermes, Claude Code).
|
||||
|
||||
** The Core Insight: LLM as Expensive Resource, Not Default Engine
|
||||
|
||||
Passepartout treats the LLM as a resource to be minimized. Every operation is designed to reduce LLM dependency. Competitors treat the LLM as the core engine through which all operations flow. This is not a difference of degree but of architecture.
|
||||
|
||||
The three structural multipliers are:
|
||||
|
||||
1. *Sparse tree retrieval* — loading relevant subtrees (200-800 tokens per file) rather than full files (1,500-5,000 tokens) = ~5-10x reduction per file access
|
||||
2. *Deterministic safety* — 9-vector dispatcher gate runs in pure Lisp (0 LLM tokens per verification) versus prompt-based guardrails (200-500 tokens per action) = infinite multiplier
|
||||
3. *REPL verification* — catches errors in-image (milliseconds, 0 LLM tokens) versus LLM correction round-trips (500-2,000 tokens per retry)
|
||||
|
||||
These compound. A coding session touching 20 files, performing 10 actions, and triggering 3 errors saves ~50,000-100,000 tokens compared to the same session with Claude Code.
|
||||
|
||||
** Per-Task Type Analysis
|
||||
|
||||
*** Coding (debugging, refactoring, PR review)
|
||||
|
||||
| Operation | Passepartout | Claude Code | Hermes (3-agent) | Savings vs Claude |
|
||||
|-----------|-------------|-------------|-------------------|--------------------|
|
||||
| File access (30 files) | 30 × 400 tok = 12,000 | 30 × 3,000 tok = 90,000 | 30 × 3,000 tok × 3 = 270,000 | 78,000 tok |
|
||||
| Reasoning rounds (20) | 20 × 3,000 tok = 60,000 | 20 × 4,000 tok = 80,000 | 20 × 3,000 tok × 3 = 180,000 | 20,000 tok |
|
||||
| Error correction (5 caught by REPL) | 0 (REPL) | 5 × 1,000 tok = 5,000 | 5 × 1,000 tok × 3 = 15,000 | 5,000 tok |
|
||||
| Safety verification | 0 (deterministic) | 500 tok/round × 20 = 10,000 | 200 tok/round × agents | 10,000 tok |
|
||||
| Agent coordination | 0 | 0 | 3,000-5,000 tok/task | 0 |
|
||||
| *Total* | *~72,000 tok* | *~185,000 tok* | *~475,000 tok* | *~113,000 tok (2.6x)* |
|
||||
|
||||
Over a month of daily coding (20 sessions): ~2.3 million tokens saved. At typical API pricing ($2-15/M tokens), this saves $5-35/month.
|
||||
|
||||
*** Knowledge Management (Zettelkasten, research, note-taking)
|
||||
|
||||
Passepartout's strongest domain. The Org-mode native format and sparse tree retrieval create a 10-40x advantage because knowledge bases are the worst case for "load everything" architectures.
|
||||
|
||||
| Operation | Passepartout | Competitor | Savings |
|
||||
|-----------|-------------|------------|---------|
|
||||
| Context assembly (500-node KB) | Peripheral outline + ~5 foveal nodes = 2,000-4,000 tok | Full serialization = 80,000-150,000 tok | 40-75x |
|
||||
| Semantic search (10 queries) | Vector lookup in-image = 0 LLM tok | LLM-assisted search = 5,000 tok | 5,000 tok |
|
||||
| Note creation (10 notes) | Deterministic Org writes = 0 LLM tok | 10 × 800 tok = 8,000 | 8,000 tok |
|
||||
| *Total per session* | *~7,000 tok* | *~95,000-165,000 tok* | *~13-24x* |
|
||||
|
||||
*** Day-to-Day Life Management (calendar, tasks, reminders)
|
||||
|
||||
| Operation | Passepartout | Competitor | Savings |
|
||||
|-----------|-------------|------------|---------|
|
||||
| Background maintenance | Deterministic heartbeat-driven = 0 LLM tok | Scheduled LLM calls or skipped | Variable |
|
||||
| User interactions (30/day) | 30 × 2,000 tok = 60,000 | 30 × 4,000 tok = 120,000 | 60,000 tok |
|
||||
| Context queries by TODO/tag | Hash table scan = 0 LLM tok | LLM-based search = 2,500 tok | 2,500 tok |
|
||||
| *Total per day* | *~60,000 tok* | *~122,500 tok* | *~2x* |
|
||||
|
||||
The defining advantage: background maintenance (compaction, archiving, link repair) costs zero LLM tokens. Competing systems either skip this or pay LLM costs for it.
|
||||
|
||||
*** Chatting (casual conversation)
|
||||
|
||||
Chatting is inherently LLM-bound. Passepartout's edge is privacy filtering before content reaches the LLM and slightly smaller context footprint. Token savings are marginal (~1.3x).
|
||||
|
||||
** The Dispatcher Learning Curve: Cost Decreases Over Time
|
||||
|
||||
A unique architectural property: Passepartout's cost curve descends while competitors' ascends.
|
||||
|
||||
Passepartout: As the dispatcher accumulates deterministic rules from Human-in-the-Loop decisions, fewer actions require LLM proposals. A file write that initially triggered a full LLM proposal → dispatcher review → HITL approval → rule extraction loop eventually becomes a deterministic rule check. Each hardened rule permanently reduces future token costs.
|
||||
|
||||
Competitors: As context histories grow, safety instructions accumulate, and guardrails become more elaborate, each interaction costs more than the last. The only way to reduce cost is to cap context — sacrificing capability.
|
||||
|
||||
After 12 months of learning, Passepartout's core reasoning costs could drop to 40-60% of baseline, while competitors' costs rise to 125-140% of baseline.
|
||||
|
||||
The crossover point where Passepartout becomes structurally cheaper is estimated at 3-6 months depending on usage volume and task diversity.
|
||||
|
||||
** Local LLM Viability
|
||||
|
||||
Reduced context requirements change which model sizes deliver acceptable performance:
|
||||
|
||||
| Model | Passepartout Viability | Competitor Viability |
|
||||
|-------|----------------------|---------------------|
|
||||
| Phi-3-mini 3.8B (4K ctx) | Viable for structured tasks | Context starvation |
|
||||
| Llama 3.1 8B (8K ctx) | Comfortable daily driver | Marginal |
|
||||
| Qwen 2.5 7B (4K ctx) | Viable for most tasks | Not viable |
|
||||
| Mistral 7B (8K ctx) | Comfortable | Marginal |
|
||||
| Llama 3.1 70B (128K ctx) | Overkill (but works) | Comfortable |
|
||||
|
||||
KV cache memory scales with context length:
|
||||
|
||||
| Context Window | KV Cache (Llama 3.1 8B, FP16) |
|
||||
|---------------|-------------------------------|
|
||||
| 4K tokens | ~67 MB |
|
||||
| 32K tokens | ~540 MB |
|
||||
| 128K tokens | ~2.1 GB |
|
||||
|
||||
Passepartout at 4K effective context: ~67 MB KV cache. Competitor at 128K: ~2.1 GB. A 7-8B model on an RTX 3060 Ti (8 GB VRAM) or MacBook (16 GB unified memory) is a practical daily driver with Passepartout. Competitors at full context require 16-32 GB VRAM or cloud APIs.
|
||||
|
||||
** Open Questions and Risks
|
||||
|
||||
1. *Retrieval accuracy is the bottleneck.* If sparse tree retrieval loads the wrong subtree (low-similarity but causally relevant), the LLM makes unfixable errors. The architecture assumes embedding quality is "good enough" — this is untested at scale.
|
||||
|
||||
2. *System prompt overhead can consume savings.* Every =think= cycle iterates all registered skills and calls every =system-prompt-augment= function. With 20+ skills, a trivial interaction could carry 3,000-8,000 tokens of overhead before user input is even processed. This overhead is flat per-call, so it disproportionately affects short interactions.
|
||||
|
||||
3. *Model size vs context quality.* A 3.8B model with perfect context cannot match a 70B model on complex multi-file refactors regardless of context quality. Model size independently determines reasoning depth. The minimum viable model is likely 7-13B parameters for engineering work.
|
||||
|
||||
4. *The 3-retry dispatcher loop.* When the dispatcher rejects a proposal, the rejection trace feeds back to the LLM for self-correction (up to 3 retries). If the dispatcher rejects 30% of proposals, the effective token multiplier is 1.39x per action. At 50% rejection (plausible during early use), it is 1.75x. This penalty decreases as the dispatcher accumulates rules.
|
||||
|
||||
5. *Competitor evolution.* Sparse retrieval is not patentable. Claude Code, Copilot, and others will implement similar mechanisms. The architectural advantage is real but finite in duration. The deterministic safety gate is the harder-to-replicate differentiator.
|
||||
|
||||
** Comparison Summary
|
||||
|
||||
| Metric | Passepartout | Claude Code | Hermes | OpenClaw |
|
||||
|--------|-------------|-------------|--------|----------|
|
||||
| Active context (tokens) | 2,000-4,000 | 10,000-50,000+ | 5,000-15,000/agent | 10,000-40,000 |
|
||||
| File access cost (per file) | 200-800 tok | 1,500-5,000 tok | 1,500-5,000 tok × agents | 1,500-5,000 tok |
|
||||
| Safety verification cost | 0 (deterministic) | 200-500 tok/action | 200-500 tok/action × agents | 100-300 tok/action |
|
||||
| Agent coordination cost | 0 | 0 | 1,000-3,000 tok/task | 500-2,000 tok/task |
|
||||
| Error recovery cost | 0 (REPL) | 500-2,000 tok/retry | 500-2,000 tok/retry × agents | 500-2,000 tok/retry |
|
||||
| Long-term cost trend | Decreasing | Increasing | Increasing | Flat/Increasing |
|
||||
| Min viable local model | 3-4B params, 4K ctx | 30-70B params, 32K+ ctx | 30-70B params, 32K+ ctx | 7-13B params, 8K+ ctx |
|
||||
| Min VRAM for local | 4-6 GB | 16-32 GB | 24-48 GB | 8-16 GB |
|
||||
|
||||
*Conclusion:* Passepartout's architecture is designed to produce 2-3x token savings for coding, 13-24x for knowledge management, and 2x for life management at v1.0.0 maturity. The three structural advantages — sparse trees, deterministic safety, and REPL verification — compound. The critical risk is implementation gap: achieving the retrieval precision, dispatcher learning, and REPL integration depth required to realize the design.
|
||||
557
docs/ROADMAP.org
557
docs/ROADMAP.org
@@ -1,11 +1,12 @@
|
||||
#+TITLE: Passepartout Evolutionary Roadmap
|
||||
#+STARTUP: content
|
||||
#+FILETAGS: :docs:roadmap:
|
||||
|
||||
* The Evolutionary Roadmap
|
||||
|
||||
The roadmap is designed working backwards from SOTA parity (V 1.0.0), guiding each version toward a fully autonomous, self-editing agent. Each version builds on the previous, with features designed to be implemented in pure Common Lisp + Org-mode.
|
||||
The roadmap is designed working backwards from SOTA parity (v1.0.0), guiding each version toward a fully autonomous, self-editing agent. Each version builds on the previous, with features designed to be implemented in pure Common Lisp + Org-mode.
|
||||
|
||||
Per-version task tracking: [[file:../TODO.org][TODO.org]]
|
||||
The TODO states in each version's Tasks section are the authoritative task tracker. The feature tables describe what each version delivers.
|
||||
|
||||
** Non-Negotiable Identity
|
||||
- Pure Common Lisp + Org-mode. No JSON. No YAML. No external databases.
|
||||
@@ -16,156 +17,522 @@ Per-version task tracking: [[file:../TODO.org][TODO.org]]
|
||||
|
||||
** Version Roadmap
|
||||
|
||||
*** v0.1.0: The Autonomous Foundation — CURRENT RELEASE ✅
|
||||
*** v0.1.0: The Autonomous Foundation — RELEASED 2026-04-20
|
||||
|
||||
The secure, auditable Lisp kernel. All core infrastructure in place.
|
||||
|
||||
| Component | Status | Notes |
|
||||
|-----------------------------------+--------+-----------------------------------------------------------------------|
|
||||
| Perceive-Reason-Act pipeline | ✅ | 3-stage metabolic loop |
|
||||
| Skills engine with jailed loading | ✅ | defskill, topological sort, hot-reload |
|
||||
| Policy skill (6 invariants) | ✅ | Transparency, Autonomy, Bloat, Modularity, Mentorship, Sustainability |
|
||||
| Bouncer skill | ✅ | Command whitelist guard functions |
|
||||
| Memory (org-object + Merkle) | ✅ | Hash tables, snapshots, rollback |
|
||||
| Lisp validator skill | ✅ | Syntax validation before eval |
|
||||
| Scribe + Gardener skills | ✅ | Heartbeat-driven distillation + audit |
|
||||
| LLM gateway (OpenRouter + Ollama) | ✅ | Provider cascade |
|
||||
| Shell actuator | ✅ | Safe command execution |
|
||||
| Emacs bridge via Swank | ✅ | Point/buffer updates |
|
||||
| FiveAM test suite | ✅ | Memory, boot, pipeline, act, communication |
|
||||
| Credentials vault | ✅ | Encrypted storage |
|
||||
**** DONE Perceive-Reason-Act pipeline
|
||||
:PROPERTIES:
|
||||
:ID: id-06f10b9a-4054-4dea-a927-b0935fbdcd2f
|
||||
:CREATED: [2026-03-22 Sun]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-04-20 Mon]
|
||||
:END:
|
||||
|
||||
*** v0.2.0: Interactive Refinement ✅
|
||||
**** DONE Skills engine with jailed loading
|
||||
:PROPERTIES:
|
||||
:ID: id-dc83944f-3923-4142-b324-c317dacd6b0b
|
||||
:CREATED: [2026-03-22 Sun]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-04-20 Mon]
|
||||
:END:
|
||||
|
||||
**** DONE Policy skill (6 invariants)
|
||||
:PROPERTIES:
|
||||
:ID: id-929c84b7-d6ae-42b9-a8b5-d9df962db826
|
||||
:CREATED: [2026-03-22 Sun]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-04-20 Mon]
|
||||
:END:
|
||||
|
||||
**** DONE Memory (memory-object + Merkle hashing)
|
||||
:PROPERTIES:
|
||||
:ID: id-3a96b384-cacf-4da0-8faa-1647739feba9
|
||||
:CREATED: [2026-03-22 Sun]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-04-20 Mon]
|
||||
:END:
|
||||
|
||||
**** DONE Scribe + Gardener background workers
|
||||
:PROPERTIES:
|
||||
:ID: id-3f618a38-ec23-4034-ba3c-ef272e212e2b
|
||||
:CREATED: [2026-03-22 Sun]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-04-20 Mon]
|
||||
:END:
|
||||
|
||||
**** DONE LLM gateway (OpenRouter, Ollama)
|
||||
:PROPERTIES:
|
||||
:ID: id-f5d870e2-cbd2-4c00-a8d4-174ab4118afc
|
||||
:CREATED: [2026-04-11 Sat]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-04-20 Mon]
|
||||
:END:
|
||||
|
||||
**** DONE Shell actuator, Emacs bridge, credentials vault
|
||||
:PROPERTIES:
|
||||
:ID: id-7ca3167f-8353-4bb7-8b97-c039017716b0
|
||||
:CREATED: [2026-04-11 Sat]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-04-20 Mon]
|
||||
:END:
|
||||
|
||||
**** DONE FiveAM test suite
|
||||
:PROPERTIES:
|
||||
:ID: id-925d4180-764b-4219-8bdc-8e1849572da1
|
||||
:CREATED: [2026-04-11 Sat]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-04-20 Mon]
|
||||
:END:
|
||||
|
||||
*** v0.2.0: Interactive Refinement — RELEASED 2026-04-29
|
||||
|
||||
The "Brain" meets the "Machine." Standardization and professionalization of the user interface and environment.
|
||||
|
||||
| Feature | Status | Notes |
|
||||
| :--- | :---: | :--- |
|
||||
| Minimalist Kernel | ✅ | Purified harness targeting I/O & Memory only. |
|
||||
| Sovereign Skills | ✅ | Diagnostics and Configuration extracted to Userland. |
|
||||
| POSIX/XDG Compliance | ✅ | Standardized paths (~/.config, ~/.local). |
|
||||
| Professional TUI | ✅ | Styled, scrollable, and verified Lisp interface. |
|
||||
| Onboarding Wizard | ✅ | Modular Lisp setup for multiple LLM providers. |
|
||||
| Linkage Command | ✅ | Real-time verification of external gateways (Telegram). |
|
||||
| Self-Editing | ✅ | Detects errors, applies fixes, learns from outcomes. |
|
||||
| Enhanced Utilities | ✅ | Structural Lisp/Org manipulation + REPL evaluation. |
|
||||
| Memory Rollback | ✅ | Snap back to known-good state on critical errors. |
|
||||
**** DONE Professional TUI (Croatoan-based, styled, scrollable)
|
||||
:PROPERTIES:
|
||||
:ID: id-57cef382-fe14-42e6-aade-03e05e3e920b
|
||||
:CREATED: [2026-04-28 Tue]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-04-29 Wed]
|
||||
:END:
|
||||
|
||||
**** DONE Self-editing (error detection, surgical fix, hot-reload)
|
||||
:PROPERTIES:
|
||||
:ID: id-459b8275-9979-4d0f-8d61-a9af883930d4
|
||||
:CREATED: [2026-04-23 Wed]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-04-29 Wed]
|
||||
:END:
|
||||
|
||||
**** DONE Enhanced utilities (structural Lisp/Org manipulation + REPL)
|
||||
:PROPERTIES:
|
||||
:ID: id-23f37c0d-4e77-4dc3-ab43-52a5987eb426
|
||||
:CREATED: [2026-04-23 Wed]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-04-29 Wed]
|
||||
:END:
|
||||
|
||||
**** DONE Onboarding wizard (modular Lisp setup for LLM providers)
|
||||
:PROPERTIES:
|
||||
:ID: id-bd497de7-3533-4056-b89f-2c992d2ea28b
|
||||
:CREATED: [2026-04-28 Tue]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-04-29 Wed]
|
||||
:END:
|
||||
|
||||
**** DONE Memory rollback (snapshot and restore)
|
||||
:PROPERTIES:
|
||||
:ID: id-fd2fb6e3-03e7-4e22-b9e9-a7eecfd06718
|
||||
:CREATED: [2026-04-12 Sun]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-04-29 Wed]
|
||||
:END:
|
||||
|
||||
**** DONE Secret Exposure Gate, Shell Safety, Lisp Validation
|
||||
:PROPERTIES:
|
||||
:ID: id-aa53c128-195b-42d4-9838-2def59faf7cf
|
||||
:CREATED: [2026-05-02 Sat]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-05-02 Sat]
|
||||
:END:
|
||||
|
||||
**** DONE Multi-distro deployment (Debian+Fedora, systemd, Docker)
|
||||
:PROPERTIES:
|
||||
:ID: id-783df999-f7fe-45c8-896d-2fd07c604d64
|
||||
:CREATED: [2026-05-02 Sat]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-05-02 Sat]
|
||||
:END:
|
||||
|
||||
**** DONE Project rename to Passepartout (files, packages, env vars)
|
||||
:PROPERTIES:
|
||||
:ID: id-91724874-aa0d-4804-9220-8bc5551f1366
|
||||
:CREATED: [2026-05-02 Sat]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-05-02 Sat]
|
||||
:END:
|
||||
|
||||
**** DONE 31 org files with full literate prose
|
||||
:PROPERTIES:
|
||||
:ID: id-597b2a92-aac6-481a-b2c4-4f9842ced97c
|
||||
:CREATED: [2026-05-02 Sat]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-05-02 Sat]
|
||||
:END:
|
||||
|
||||
*** v0.3.0: Event Orchestration + HITL
|
||||
|
||||
Unified control plane and Human-in-the-Loop (HITL) state management.
|
||||
Unified control plane and Human-in-the-Loop state management.
|
||||
|
||||
| Feature | Description |
|
||||
|--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| org-skill-event-orchestrator | Unified hooks + cron + routing. Three tiers: =:REFLEX= (no LLM), =:COGNITION= (light LLM), =:REASONING= (full LLM). |
|
||||
| Human-in-the-Loop (HITL) | Continuation-based interaction. The agent can "suspend" its cognitive loop to ask for permission or clarification and resume precisely where it left off. |
|
||||
| org-skill-context-manager | Stack-based project scoping. =push-context= / =pop-context=. Path resolution relative to context. |
|
||||
| Memory scope segmentation | =:scope= property on org-objects: memex/session/project. Scope-aware retrieval. |
|
||||
| Model-tier routing | Complexity-based model selection: heartbeat → tiny, user → medium, reasoning → large. |
|
||||
| Slash commands | =M-x= style command palette in TUI. Commands defined in Org-mode. |
|
||||
| Asynchronous Embedding Gateway | Provider-agnostic vector generation (Ollama, local llama.cpp) via background worker. |
|
||||
| Telegram Gateway Skill | Full implementation of the message receiver for linked Telegram bots. |
|
||||
** Tasks
|
||||
|
||||
*** Remediation: Backfill v0.1.0/v0.2.0 Gaps
|
||||
|
||||
These features were marked DONE in prior versions but are stubs, no-ops, or
|
||||
missing. They must be completed before v0.3.0 feature work proceeds.
|
||||
|
||||
**** DONE P0: Add vault-get-secret / vault-set-secret wrappers :backfill:
|
||||
CLOSED: [2026-05-03 Sun 10:42]
|
||||
:PROPERTIES:
|
||||
:ID: id-vault-secret-wrappers
|
||||
:CREATED: [2026-05-03 Sun]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
|
||||
:END:
|
||||
=vault-get-secret= and =vault-set-secret= are exported from =core-defpackage=
|
||||
and called from =gateway-manager.org= (lines 36, 86, 180) but never defined.
|
||||
=gateway-link= crashes at runtime. Add one-line wrappers in =security-vault.org=
|
||||
that delegate to the existing =vault-get=/=vault-set= with ~:type :secret~.
|
||||
|
||||
**** DONE P0: system-archivist — Scribe + Gardener :backfill:
|
||||
CLOSED: [2026-05-03 Sun 10:42]
|
||||
:PROPERTIES:
|
||||
:ID: id-archivist-distillation
|
||||
:CREATED: [2026-05-03 Sun]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
|
||||
:END:
|
||||
Scribe: distill daily Org logs into atomic Zettelkasten notes with backlinks.
|
||||
Gardener: scan for broken =[[file:]]= links and orphaned =memory-object= entries.
|
||||
Wire both as cron jobs via =system-event-orchestrator=.
|
||||
Depends on: orchestrator bootstrap (P1 item below).
|
||||
|
||||
**** DONE P0: system-self-improve — surgical edit + error fix :backfill:
|
||||
CLOSED: [2026-05-03 Sun 10:42]
|
||||
:PROPERTIES:
|
||||
:ID: id-self-improve-real
|
||||
:CREATED: [2026-05-03 Sun]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
|
||||
:END:
|
||||
= self-improve-edit=: =org-read-file= → text replace → =snapshot-memory= →
|
||||
=org-write-file= → =literate-block-balance-check= → tangle → reload.
|
||||
=self-improve-fix=: parse error log → =lisp-structural-check= →
|
||||
=lisp-extract= → surgical repair → =repl-eval= verify.
|
||||
Remove the dead first =defskill= registration (trigger nil, overwritten by second).
|
||||
Depends on: =programming-org=, =programming-literate= (P0 items below).
|
||||
|
||||
**** DONE P0: programming-org — fix org-modify + org-ast-render :backfill:
|
||||
CLOSED: [2026-05-03 Sun 10:42]
|
||||
:PROPERTIES:
|
||||
:ID: id-org-modify-render
|
||||
:CREATED: [2026-05-03 Sun]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
|
||||
:END:
|
||||
=org-modify(filepath, id, changes)= ignores ~changes~ and only logs. Should locate
|
||||
node by ID in file and apply changes to its content.
|
||||
=org-ast-render(ast)= returns a hardcoded placeholder. Should convert plist AST
|
||||
back to Org text.
|
||||
|
||||
**** DONE P0: programming-literate — fix both stubs :backfill:
|
||||
CLOSED: [2026-05-03 Sun 10:42]
|
||||
:PROPERTIES:
|
||||
:ID: id-literate-real
|
||||
:CREATED: [2026-05-03 Sun]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
|
||||
:END:
|
||||
=literate-block-balance-check=: verify all =#+begin_src lisp= blocks in an Org file
|
||||
have balanced parentheses. Returns T if all balanced, error message otherwise.
|
||||
=literate-tangle-sync-check=: verify =.lisp= file matches tangled output of =.org= file.
|
||||
|
||||
**** DONE P1: system-event-orchestrator — bootstrap implementation :backfill:
|
||||
CLOSED: [2026-05-03 Sun 10:42]
|
||||
:PROPERTIES:
|
||||
:ID: id-orchestrator-bootstrap
|
||||
:CREATED: [2026-05-03 Sun]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
|
||||
:END:
|
||||
=orchestrator-bootstrap= currently only logs. Should scan Org files for =#+HOOK:=
|
||||
and =#+CRON:= properties and register them via the existing registries.
|
||||
Prerequisite for archivist cron jobs.
|
||||
|
||||
**** DONE P1: system-memory — memory introspection :backfill:
|
||||
CLOSED: [2026-05-03 Sun 10:42]
|
||||
:PROPERTIES:
|
||||
:ID: id-memory-inspect
|
||||
:CREATED: [2026-05-03 Sun]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
|
||||
:END:
|
||||
=memory-inspect= only logs. Should return structured statistics: object count
|
||||
by type, TODO state distribution, orphan count, snapshot list. Trigger on
|
||||
=:INTROSPECTION= sensor type.
|
||||
|
||||
**** DONE P1: Path relic — skills/ → lisp/ in skill-initialize-all :backfill:
|
||||
CLOSED: [2026-05-03 Sun 10:42]
|
||||
:PROPERTIES:
|
||||
:ID: id-path-relic
|
||||
:CREATED: [2026-05-03 Sun]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
|
||||
:END:
|
||||
=skill-initialize-all= and =context-skill-source= resolve against =skills/=
|
||||
under =$PASSEPARTOUT_DATA_DIR=. Core and skills were merged into =lisp/=.
|
||||
Update both functions to point at =lisp/=.
|
||||
|
||||
**** TODO P2: core-context — semantic retrieval (embeddings) :backfill:
|
||||
:PROPERTIES:
|
||||
:ID: id-embeddings
|
||||
:CREATED: [2026-05-03 Sun]
|
||||
:END:
|
||||
=org-object-vector= is never populated; all similarities are 0.0. Generate
|
||||
embeddings via Ollama =nomic-embed-text= at ingest time. Store in
|
||||
=memory-object.vector=. Fallback: TF-IDF bag-of-words.
|
||||
|
||||
**** TODO P2: core-context — subtree-based skill source loading :backfill:
|
||||
:PROPERTIES:
|
||||
:ID: id-skill-subtree
|
||||
:CREATED: [2026-05-03 Sun]
|
||||
:END:
|
||||
=context-skill-source= reads entire Org files. Add =context-skill-subtree=
|
||||
for targeted retrieval of specific function docs or test blocks by heading name.
|
||||
|
||||
**** TODO P3: Variable name drift normalization (out of scope for now) :backfill:
|
||||
:PROPERTIES:
|
||||
:ID: id-name-normalization
|
||||
:CREATED: [2026-05-03 Sun]
|
||||
:END:
|
||||
=*memory*= (context) vs =*memory-store*= (memory). =*skills-registry*= with
|
||||
underscore (reason/context) vs =*skill-registry*= with hyphen (defpackage).
|
||||
Normalization pass across all modules. Touches every file — do after P0-P2
|
||||
are stable. Do not mix with functional changes.
|
||||
|
||||
*** DONE Project Renaming (Bouncer → Dispatcher)
|
||||
:PROPERTIES:
|
||||
:ID: id-9e779580-287b-b3d1-37b9-bcefd750bf9e
|
||||
:CREATED: [2026-05-01 Fri 15:40]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-05-02 Sat 22:00]
|
||||
:END:
|
||||
The Dispatcher's role has evolved beyond security guard. It is the seed of the deterministic engine — it learns to execute procedures without invoking the neural net.
|
||||
|
||||
*** DONE Event Orchestrator (unified hooks+cron+routing)
|
||||
:PROPERTIES:
|
||||
:ID: id-d35aea3d-2e5f-4a12-a9b0-1c2d3e4f5a6b
|
||||
:CREATED: [2026-05-02 Sat 14:00]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-05-02 Sat 22:36]
|
||||
:END:
|
||||
Unified control plane for hooks, cron, and complexity-based routing.
|
||||
- *hook-registry* + *cron-registry* + tier classifier
|
||||
- Hooks via ~#+HOOK:~ Org-mode properties
|
||||
- Three complexity tiers: ~:REFLEX~ (no LLM), ~:COGNITION~ (light LLM), ~:REASONING~ (full LLM)
|
||||
- Hooked into heartbeat for cron processing
|
||||
- Rule-based tier classifier (overrideable via ~*tier-classifier*~)
|
||||
|
||||
*** TODO Context Manager (project scoping)
|
||||
:PROPERTIES:
|
||||
:ID: id-a10ed34e-9f37-4a15-b499-46672c00d951
|
||||
:CREATED: [2026-05-02 Sat 23:00]
|
||||
:END:
|
||||
Stack-based context with ~push-context~ / ~pop-context~.
|
||||
Path resolution relative to current context.
|
||||
Memory scope: ~:scope~ property on memory-objects (memex/session/project).
|
||||
Implement lazy-loading proxies for large-scale memory traversal.
|
||||
|
||||
*** TODO Model-Tier Routing (cost optimization)
|
||||
Extend ~*model-selector-fn*~ for complexity-based routing.
|
||||
- Heartbeats → smallest model
|
||||
- User input → medium model
|
||||
- Complex reasoning → large model
|
||||
|
||||
*** TODO Memory Scope Segmentation
|
||||
Extend memory-object with ~:scope~ property.
|
||||
- ~:memex~ (permanent knowledge), ~:session~ (ephemeral), ~:project~ (current work)
|
||||
- Scope-aware retrieval in memory layer
|
||||
|
||||
*** TODO Asynchronous Embedding Gateway
|
||||
Provider-agnostic vector generation (Ollama, llama.cpp, OpenAI).
|
||||
Edits mark nodes as ~:vector :pending~; background worker batches and updates Merkle tree.
|
||||
|
||||
*** TODO TUI Experience (Daily Driver Quality)
|
||||
The TUI is a standalone Croatoan app in ~org/gateway-tui.org~.
|
||||
None of these changes require daemon modifications — the protocol between TUI and
|
||||
daemon (port 9105, framed plists) is stable.
|
||||
|
||||
- P0: Chat scrollback (Page Up/Down) — ~2h
|
||||
- P0: Input history (up/down arrows) — ~1h
|
||||
- P1: Status bar (daemon, model, time) — ~3h
|
||||
- P1: Message rendering (timestamps, colors, wrapping) — ~2h
|
||||
- P2: Command palette (/help redesign) — ~4h
|
||||
- P2: Multi-line input (Shift+Enter) — ~3h
|
||||
- P3: Background activity indicator — ~2h
|
||||
- P4: Tab completion for / commands — ~3h
|
||||
- P4: Configurable theme — ~4h
|
||||
|
||||
*** TODO Human-in-the-Loop (HITL)
|
||||
Continuation-based interaction. The agent can suspend its cognitive loop to ask for
|
||||
permission or clarification and resume precisely where it left off. Builds on the
|
||||
dispatcher's existing Flight Plan mechanism.
|
||||
|
||||
*** v0.4.0: Long-Horizon Planning + Git Workflows
|
||||
|
||||
Structured tracking, failure handling, and course correction for multi-step engineering work.
|
||||
|
||||
| Feature | Description |
|
||||
|------------------------+---------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| org-skill-long-horizon | Decompose tasks into Org-mode headline trees. Terminal states: =:done= / =:blocked= / =:stuck=. Parent summarises children. Branch pruning. |
|
||||
| org-skill-git-steward | Status, diff, commit, push, branch. Policy enforces commit-before-modify. |
|
||||
| TDD runner | FiveAM on file save. =:test-failure= events. Hook into self-fix for auto-repair. |
|
||||
| Deep Emacs integration | Full org-agenda awareness. Navigate, clock time, refile, archive. |
|
||||
** Tasks
|
||||
|
||||
*** TODO Long-Horizon Planning (task tree DAG)
|
||||
Decompose complex tasks into Org-mode headline trees.
|
||||
Terminal states: ~:todo~ → ~:next-action~ → ~:in-progress~ → ~:done~ / ~:blocked~ / ~:stuck~.
|
||||
Parent summarises child results.
|
||||
Branch pruning when paths fail.
|
||||
|
||||
*** TODO Git Steward (version control integration)
|
||||
Status, diff, commit, push, branch operations.
|
||||
Policy enforces commit-before-modify gate.
|
||||
Log commits to memory.
|
||||
|
||||
*** TODO TDD Runner Integration
|
||||
Run FiveAM tests on file save.
|
||||
Inject ~:test-failure~ event on red.
|
||||
Hook into self-fix for auto-repair proposals.
|
||||
|
||||
*** TODO Deep Emacs Integration
|
||||
Full org-agenda awareness: navigate, clock time, refile, archive.
|
||||
Uses org-element + org-id.
|
||||
|
||||
*** v0.5.0: Interactive Actuation & Environment Stewardship
|
||||
|
||||
Interactive terminal sessions and autonomous dependency management.
|
||||
|
||||
| Feature | Description |
|
||||
|--------------------------+-------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| Interactive PTY Actuator | Stream long-running process output to the context window (e.g., `npm run dev`, REPLs) with async interrupt control. |
|
||||
| The Environment Steward | Autonomously detect missing dependencies (e.g., "Command not found"), propose an installation command, and retry the failed action. |
|
||||
** Tasks
|
||||
|
||||
*** TODO Interactive PTY Actuator
|
||||
Stream long-running process output to the context window (e.g., ~npm run dev~, REPLs).
|
||||
Async interrupt control (Ctrl+C emulation).
|
||||
|
||||
*** TODO The Environment Steward
|
||||
Autonomously detect missing dependencies ("Command not found").
|
||||
Propose installation command and retry the failed action.
|
||||
|
||||
*** v0.6.0: Concurrency + Creator + GTD
|
||||
|
||||
The agent bootstraps itself and manages parallel workstreams.
|
||||
|
||||
| Feature | Description |
|
||||
|-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| org-skill-sub-agent-manager | Lightweight Lisp-native sub-agents (via bordeaux-threads) that share memory but have isolated execution contexts for background work. |
|
||||
| org-skill-creator | LLM drafts complete skill org-file from natural language. Mandatory: syntax validation → jail-load → test → register. |
|
||||
| org-skill-architect | Scan =:STATUS: FROZEN= PRDs. Generate Phase B PROTOCOL. |
|
||||
| org-skill-gtd | Full GTD cycle: capture, clarify, organize, reflect, engage. org-gtd v4.0 DAG (=:TRIGGER:=, =:BLOCKER:=). |
|
||||
| Consensus loop | Run multiple providers for critical decisions. Compare results, detect disagreements. |
|
||||
| Web research | Headless Chromium via Python bridge. Text extraction, screenshots, Gemini Web UI automation. |
|
||||
** Tasks
|
||||
|
||||
*** TODO Skill Creator (autonomous skill generation)
|
||||
LLM drafts complete skill org-file from natural language.
|
||||
Mandatory: syntax validation → jail-load → test → register.
|
||||
|
||||
*** TODO Architect Agent (PRD → PROTOCOL)
|
||||
Scan ~:STATUS: FROZEN~ PRDs. Generate Phase B PROTOCOL from Phase A.
|
||||
|
||||
*** TODO GTD Integration (project tracking)
|
||||
Full GTD cycle: capture, clarify, organize, reflect, engage.
|
||||
org-gtd v4.0 DAG (~:TRIGGER:~, ~:BLOCKER:~).
|
||||
|
||||
*** TODO Consensus Loop (multi-model agreement)
|
||||
Run multiple providers for critical decisions.
|
||||
Compare results, detect disagreements.
|
||||
Confidence scoring.
|
||||
|
||||
*** TODO Web Research (Playwright browsing)
|
||||
Headless Chromium via Python bridge.
|
||||
Text extraction, screenshots, Gemini Web UI automation.
|
||||
|
||||
*** TODO Memex Management (PARA lifecycle)
|
||||
Archive DONE tasks, suggest refiling.
|
||||
Detect orphaned nodes.
|
||||
PARA/Zettelkasten maintenance.
|
||||
|
||||
*** v0.7.0: Visual Grounding & MCP Bridge
|
||||
|
||||
Multimodal visual interaction and ecosystem-wide tool compatibility.
|
||||
|
||||
| Feature | Description |
|
||||
|-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| Computer Use / Vision | Allow the agent to request host OS or browser screenshots, analyze the UI, and issue precise X/Y coordinate click/type commands via an X11/Wayland bridge. |
|
||||
| MCP Gateway Bridge | Lisp-native client for the Model Context Protocol, allowing Passepartout to connect to the entire ecosystem of external tools and data sources. |
|
||||
** Tasks
|
||||
|
||||
*** TODO Computer Use / Vision
|
||||
Allow the agent to request host OS or browser screenshots.
|
||||
Analyze UI and issue precise X/Y coordinate click/type commands via X11/Wayland bridge.
|
||||
|
||||
*** TODO MCP Gateway Bridge
|
||||
Lisp-native client for the Model Context Protocol.
|
||||
Connect Passepartout to external tools and data sources.
|
||||
|
||||
*** v0.8.0: The Evaluation Harness
|
||||
|
||||
Automated benchmarking to mathematically prove the agent's reasoning capabilities.
|
||||
|
||||
| Feature | Description |
|
||||
|-------------------+------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| SWE-Bench Harness | Automated pipeline that clones repositories, feeds GitHub issues, tracks the multi-step resolution trajectory, runs tests, and scores success. |
|
||||
** Tasks
|
||||
|
||||
*** TODO SWE-Bench Harness
|
||||
Automated pipeline that clones repositories and feeds GitHub issues.
|
||||
Track multi-step resolution trajectory, run tests, and score success.
|
||||
|
||||
*** v1.0.0: SOTA Parity
|
||||
|
||||
Feature-complete agent competitive with commercial agents. All features reimplemented in pure Lisp.
|
||||
Feature-complete agent competitive with commercial agents. All features from v0.2.0 through v0.8.0 combined, verified, and tested end-to-end.
|
||||
|
||||
| Area | Status | Notes |
|
||||
|-------------------+-----------+-------------------------------------------|
|
||||
| Self-improvement | ✅ v0.2.0 | Self-edit + lisp-repair |
|
||||
| Planning | ✅ v0.4.0 | Task tree DAGs with terminal states |
|
||||
| Tool ecosystem | 🟡 v0.4.0 | 10+ cognitive tools |
|
||||
| Context window | ✅ v0.3.0 | Semantic search + scope segmentation |
|
||||
| Safety | ✅ v0.1.0 | 6 Policy invariants + formal verification |
|
||||
| Multi-step tasks | ✅ v0.4.0 | Task trees with failure handling |
|
||||
| Code editing | ✅ v0.2.0 | Full org-mode file read/write |
|
||||
| Memory | ✅ v0.2.0 | Vector recall in org-object |
|
||||
| Emacs integration | ✅ v0.2.0 | Full org-mode control |
|
||||
| Autonomy | ✅ v0.1.0 | 100% local capable (Ollama) |
|
||||
| Area | Parity Target |
|
||||
|------|--------------|
|
||||
| Self-improvement | Claude Code self-debug |
|
||||
| Planning | ULTRAPLAN equivalent |
|
||||
| Tool ecosystem | 10+ cognitive tools |
|
||||
| Context window | Semantic search + scope segmentation |
|
||||
| Safety | 6 Policy invariants + formal verification |
|
||||
| Multi-step tasks | Task trees with terminal states |
|
||||
| Code editing | Full file read/write via org manipulation |
|
||||
| Memory | Vector recall in memory-object |
|
||||
| Emacs integration | Full org-mode control (exceeds Claude Code) |
|
||||
| Autonomy | 100% local capable (exceeds Claude Code) |
|
||||
|
||||
*** v2.0.0: Lisp Machine Emergence
|
||||
|
||||
From Lisp-using agent to true Lisp machine. Agent IS the Emacs process.
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| Lish: Lisp editor | Org-mode as IDE. Org-babel for interactive evaluation. Full REPL in TUI. No bridge needed. |
|
||||
| Lish: Shell replacement | Lisp-based shell that speaks plists. Org-mode buffers as file system. |
|
||||
- Lish: Lisp editor — Org-mode as IDE. Org-babel for interactive evaluation. Full REPL in TUI.
|
||||
- Lish: Shell replacement — Lisp-based shell that speaks plists. Org-mode buffers as file system.
|
||||
|
||||
*** v3.0.0: Neurosymbolic Maturity
|
||||
|
||||
Deterministic planner takes the wheel. LLM relegated to semantic translation.
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| Deterministic planner | Pure Lisp task scheduler. No LLM needed for planning. |
|
||||
| Self-correcting gates | Gates learn from false positives (user override patterns). |
|
||||
- Deterministic planner: Pure Lisp task scheduler. No LLM needed for scheduling.
|
||||
- Self-correcting gates: Gates learn from false positives (user override patterns).
|
||||
|
||||
*** v4.0.0: AI Stack Internalized
|
||||
|
||||
The agent understands its own weights. No external inference.
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| Llama.cpp in Lisp | FFI binding. No Python subprocess. Pure Common Lisp inference. |
|
||||
| Weights as sexps | Neural weights as Lisp data structures. Homoiconic model introspection. |
|
||||
- Llama.cpp in Lisp: FFI binding. No Python subprocess. Pure Common Lisp inference.
|
||||
- Weights as sexps: Neural weights as Lisp data structures. Homoiconic model introspection.
|
||||
|
||||
*** v5.0.0: True Agency
|
||||
|
||||
World models, temporal reasoning, goal persistence across restarts.
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| World models | Predictive models of user behavior, project dynamics, system state. |
|
||||
| Temporal reasoning | Scheduling, deadlines, elapsed duration awareness. |
|
||||
| Goal persistence | Goals survive restarts. Long-term projects in org-objects. |
|
||||
- World models: Predictive models of user behavior, project dynamics, system state.
|
||||
- Temporal reasoning: Scheduling, deadlines, elapsed duration awareness.
|
||||
- Goal persistence: Goals survive restarts. Long-term projects in memory-objects.
|
||||
|
||||
253
docs/v0.2.x-REMEDIATION.org
Normal file
253
docs/v0.2.x-REMEDIATION.org
Normal file
@@ -0,0 +1,253 @@
|
||||
#+TITLE: v0.2.x Remediation Plan
|
||||
#+AUTHOR:
|
||||
#+STARTUP: content
|
||||
#+FILETAGS: :docs:plan:remediation:
|
||||
|
||||
* Summary
|
||||
|
||||
Features marked DONE in the ROADMAP for v0.1.0 and v0.2.0 but whose implementations
|
||||
are stubs, no-ops, or missing critical functionality. These should have been
|
||||
completed in their respective versions and must be addressed before v0.3.0
|
||||
development proceeds.
|
||||
|
||||
* P0: system-archivist — Proper Distillation and Link Maintenance
|
||||
|
||||
** Claimed status**: =DONE= (v0.1.0: "Scribe + Gardener background workers" + v0.2.0: "31 org files with full literate prose")
|
||||
|
||||
** Actual state**: =archivist-log= is a trivial log wrapper (~10 lines). No knowledge
|
||||
distillation, no broken link detection, no orphaned node flagging.
|
||||
|
||||
** What it should do**:
|
||||
|
||||
*** Scribe (knowledge distillation)
|
||||
1. Read daily Org log files from the Memex =daily/= directory
|
||||
2. Identify new entries (since last processed commit or timestamp)
|
||||
3. Extract conceptual claims, decisions, and atomic facts from prose
|
||||
4. Generate atomic Zettelkasten notes in =notes/= with:
|
||||
- Descriptive snake_case filename (no dates)
|
||||
- =:CREATED:= property from the source log's date
|
||||
- =Source:= backlink to the original daily file and headline
|
||||
- Tags inferred from content and parent file
|
||||
5. Track processed state to avoid re-distilling the same content
|
||||
|
||||
*** Gardener (structural maintenance)
|
||||
1. Scan all Org files in the Memex for broken =[[file:...][...]]= links
|
||||
2. Scan =memory-store= for =memory-object= entries whose =:parent-id= or =:children=
|
||||
references point to deleted objects (orphaned nodes)
|
||||
3. Flag broken links and orphans with =:GARDENER: broken-link= or =:GARDENER: orphan= tags
|
||||
4. Generate a maintenance report as a Org buffer the user can review
|
||||
|
||||
*** Implementation approach
|
||||
- Wire into =system-event-orchestrator= as cron jobs:
|
||||
- Scribe: daily cron (="<%%Y-%%m-%%d %%a +1d>"=, tier =:cognition=)
|
||||
- Gardener: weekly cron (="<%%Y-%%m-%%d %%a +1w>"=, tier =:cognition=)
|
||||
- Use =orchestrator-register-cron= to schedule
|
||||
- Replace the trivial =archivist-log= function with real implementation
|
||||
- Track last-processed state via =memory-store= (:LATEST_PROCESSED_DATETIME property)
|
||||
or git commit hash
|
||||
|
||||
** Dependencies**: =system-event-orchestrator= (cron scheduling), =core-memory= (object store)
|
||||
|
||||
** Verification**: FiveAM test that creates a daily log with known content, runs the
|
||||
Scribe, and asserts that an atomic note was created with correct backlinks.
|
||||
|
||||
* P0: system-self-improve — Surgical Self-Editing and Self-Repair
|
||||
|
||||
** Claimed status**: =DONE= (v0.2.0: "Self-editing (error detection, surgical fix, hot-reload)")
|
||||
|
||||
** Actual state**: =self-improve-edit= does =(declare (ignore old-text new-text))= followed by
|
||||
a log message — no actual text transformation. =self-improve-fix= same pattern.
|
||||
The skill's trigger is =nil= so it never fires.
|
||||
|
||||
** What it should do**:
|
||||
|
||||
*** Self-edit (surgical text replacement)
|
||||
1. Accept (=filepath=, =old-text=, =new-text=) and apply the transformation
|
||||
2. Read the file, locate =old-text= (with exact match verification), replace with =new-text=
|
||||
3. If the target is an Org file with a =#+begin_src lisp= block, tangling the file
|
||||
and reloading the skill after edit
|
||||
4. Create a memory snapshot before editing (rollback safety)
|
||||
5. Verify the edit succeeded (re-read file, confirm =new-text= appears)
|
||||
6. Return success/failure with a diff summary
|
||||
|
||||
*** Self-fix (error diagnosis and repair)
|
||||
1. Accept (=skill-name=, =error-log=) and diagnose the failure
|
||||
2. Parse the error log for: syntax errors (unmatched parens, invalid forms),
|
||||
undefined symbol references, semantic issues (prohibited forms)
|
||||
3. For syntax errors: locate the problematic region, propose a correction
|
||||
using structural Lisp knowledge
|
||||
4. For undefined references: check if the symbol exists in another package,
|
||||
if the skill's =#+DEPENDS_ON:= declaration is missing a dependency
|
||||
5. For semantic issues: identify the prohibited operation and suggest alternatives
|
||||
6. Invoke =self-improve-edit= to apply the fix
|
||||
7. After repair, run the skill's tests if they exist; if tests pass, hot-reload
|
||||
|
||||
*** Implementation approach
|
||||
- Add an actual =:trigger= function that activates on =:ERROR= or =:STUCK= signal types
|
||||
- =self-improve-edit=: use =uiop:read-file-string=, string replacement with
|
||||
=ppcre:regex-replace= or substring operations, write back with =with-open-file=
|
||||
- =self-improve-fix=: add structural analysis in =programming-lisp.lisp= for error parsing
|
||||
- Leverage the REPL skill for verification after repair (call =lisp-eval= on the fixed code block)
|
||||
|
||||
** Dependencies**: =programming-lisp= (lisp-structural-check), =programming-org= (tangling),
|
||||
=core-memory= (snapshot-memory), =core-skills= (jailed reload)
|
||||
|
||||
** Verification**: FiveAM test that creates a file with known content, calls self-improve-edit,
|
||||
and asserts the replacement was applied. Second test with a file containing a
|
||||
deliberate error, calls self-improve-fix, and asserts the error was corrected.
|
||||
|
||||
* P1: system-event-orchestrator — Bootstrap Implementation
|
||||
|
||||
** Claimed status**: v0.3.0 partially DONE ("hook-registry + cron-registry + tier classifier")
|
||||
|
||||
** Actual state**: Hook/cron registries, tier dispatching, and heartbeat integration work.
|
||||
But =orchestrator-bootstrap= is a stub: =(log-message "ORCHESTRATOR: Bootstrap complete")=
|
||||
|
||||
** What it should do**:
|
||||
|
||||
1. Scan the Memex =projects/= and =notes/= directories for Org files containing =#+HOOK:= properties
|
||||
2. For each =#+HOOK:= property found, call =orchestrator-register-hook= with
|
||||
the hook name and a gate function
|
||||
3. For files with =#+CRON:= properties (or cron expressions in timestamps),
|
||||
register them via =orchestrator-register-cron=
|
||||
4. Log the count of registered hooks and cron jobs at completion
|
||||
5. Run bootstrap once at startup (after memory is loaded but before cognitive loop begins)
|
||||
|
||||
*** Implementation approach
|
||||
- Use =uiop:directory-files= with glob patterns for =*.org= files
|
||||
- Use =org-element= from Emacs (via =emacs-bridge= or =org-eval= skill) for parsing,
|
||||
or implement a simple regex-based Org property parser in Lisp
|
||||
- Walk each file's headlines, extract property drawers, filter for =HOOK:= and =CRON:= keys
|
||||
- Call existing =orchestrator-register-hook= / =orchestrator-register-cron=
|
||||
|
||||
** Dependencies**: =programming-org= (Org file parsing), file system access
|
||||
|
||||
** Verification**: Create a test Org file with =#+HOOK: on-write=, run bootstrap,
|
||||
assert the hook registry contains the expected entry.
|
||||
|
||||
* P1: system-memory — Memory Introspection
|
||||
|
||||
** Claimed status**: Skill exists but was never part of a version milestone.
|
||||
|
||||
** Actual state**: =memory-inspect= is a no-op: =(log-message "MEMORY: Self-inspection triggered.")=
|
||||
The =:trigger= is =nil= so the skill never activates.
|
||||
|
||||
** What it should do**:
|
||||
|
||||
1. Return a structured report of memory state:
|
||||
- Total objects in =*memory-store*=
|
||||
- Distribution by type (=:HEADLINE=, =:PARAGRAPH=, etc.)
|
||||
- Distribution by =:TODO-STATE= (=TODO=, =NEXT=, =DONE=, etc.)
|
||||
- Count of privacy-filtered objects
|
||||
- Most recent objects (by =:version= timestamp)
|
||||
- Current snapshot count and timestamps
|
||||
- Orphaned objects (parent-id references a deleted ID)
|
||||
2. Accept an optional filter to narrow the report (by type, by tag, by time range)
|
||||
3. Wire the trigger to activate on =:INTROSPECTION= signal type or =/memory= commands
|
||||
|
||||
*** Implementation approach
|
||||
- Iterate =*memory-store*= with =maphash=, collect statistics
|
||||
- Add to skill trigger: =(eq (getf (getf ctx :payload) :sensor) :introspection)=
|
||||
- Return results as a plist that can be rendered in the TUI
|
||||
|
||||
** Dependencies**: =core-memory= (memory-store and memory-object struct)
|
||||
|
||||
** Verification**: Ingest known objects, call memory-inspect, assert type counts and
|
||||
object counts match.
|
||||
|
||||
* P2: core-context — Semantic Retrieval (Embeddings)
|
||||
|
||||
** Claimed status**: The foveal-peripheral model is implemented and tested, but the
|
||||
embedding pipeline that feeds it is listed as TODO for v0.3.0.
|
||||
|
||||
** Actual state**: The context rendering code (=context-object-render=) computes
|
||||
=cosine-similarity= correctly, but =org-object-vector= is never populated.
|
||||
All objects have =nil= vectors, all similarities are =0.0=, and the model
|
||||
falls back to "include everything within depth 2." This is functionally
|
||||
equivalent to no retrieval at all.
|
||||
|
||||
** What it should do**:
|
||||
|
||||
1. Add a =populate-vector= function to =core-memory= that calls an embedding
|
||||
provider and stores the result in the =memory-object= =:vector= slot
|
||||
2. At ingest time (=ingest-ast=), generate embeddings for new objects
|
||||
3. Embedding provider options (in priority order):
|
||||
- Ollama (local, =nomic-embed-text= or =mxbai-embed-large=)
|
||||
- OpenAI-compatible embedding API (=text-embedding-3-small=)
|
||||
- Fallback: TF-IDF bag-of-words vector (no external dependency)
|
||||
4. Updates: when =memory-object= content changes, mark =:vector= as =:pending=
|
||||
and process in a background batch via the event orchestrator
|
||||
5. Add an environment variable =EMBEDDING_PROVIDER= with default =ollama=
|
||||
|
||||
*** Implementation approach
|
||||
- Add an =:embedding-provider= function stored in =*config*=
|
||||
- =embed-object=: take content string → call provider → store float vector
|
||||
- Modify =ingest-ast= to call =embed-object= on each new object
|
||||
- Add batch processing in =system-event-orchestrator= for vector updates
|
||||
- Use =bordeaux-threads= with a lock for async embedding generation
|
||||
|
||||
** Dependencies**: External embedding provider (Ollama or API), =core-memory= (vector slot)
|
||||
|
||||
** Verification**: Create objects with content, run embedding pipeline, assert vectors
|
||||
are non-nil and have the correct dimensionality. Verify that =cosine-similarity=
|
||||
between semantically similar objects exceeds 0.75 threshold.
|
||||
|
||||
* P2: core-context — Subtree-Based Skill Source Loading
|
||||
|
||||
** Claimed status**: DESIGN_DECISIONS §"Org-Mode as Unified AST" describes: "When the
|
||||
agent needs information about the =openctl-db= function, it queries for the
|
||||
=openctl-db= subtree specifically."
|
||||
|
||||
** Actual state**: =context-skill-source= reads the ENTIRE Org file as a string via
|
||||
=uiop:read-file-string=. No subtree query exists.
|
||||
|
||||
** What it should do**:
|
||||
|
||||
1. Add a =context-skill-subtree= function that takes (=skill-name=, =heading-name=)
|
||||
and returns only the content under that headline
|
||||
2. Add a =context-skill-function-signature= function that returns only the function
|
||||
name, lambda list, and docstring
|
||||
3. Add a =context-skill-tests= function that returns only test blocks
|
||||
4. Modify =context-skill-source= to optionally accept a =:subtree= keyword argument
|
||||
5. If the Org file has an Org-element parser available, use it for structural queries;
|
||||
otherwise fall back to regex-based headline matching
|
||||
|
||||
*** Implementation approach
|
||||
- Use =org-element= via =org-eval= skill (REPL bridge to Emacs) if available
|
||||
- Lisp-native fallback: parse Org headlines with regex (=^*+ = pattern),
|
||||
match heading name by string comparison, extract content until next
|
||||
headline of equal or higher level
|
||||
- Cache parsed results to avoid re-parsing on repeated queries
|
||||
|
||||
** Dependencies**: =programming-org= (Org parsing utilities), =emacs-bridge= (if Emacs
|
||||
Org-element is preferred)
|
||||
|
||||
** Verification**: Create a test Org file with multiple headlines, query for a specific
|
||||
subtree, assert only that subtree's content is returned.
|
||||
|
||||
* Priority and Sequencing
|
||||
|
||||
The remediation should proceed in this order:
|
||||
|
||||
1. **system-event-orchestrator bootstrap** (P1) — needed as infrastructure for Scribe/Gardener cron scheduling
|
||||
2. **system-archivist** (P0) — depends on orchestrator for cron scheduling
|
||||
3. **system-self-improve** (P0) — independent, can proceed in parallel with #2
|
||||
4. **core-context embeddings** (P2) — independent, unlocks semantic retrieval
|
||||
5. **core-context subtree loading** (P2) — independent, improves context efficiency
|
||||
6. **system-memory inspect** (P1) — lowest priority, nice-to-have introspection
|
||||
|
||||
P0 items must be completed before v0.3.0 development begins. P1 items should be
|
||||
completed before v0.3.0 is released. P2 items can extend into early v0.3.0.
|
||||
|
||||
* Out of Scope
|
||||
|
||||
Features listed as TODO in the ROADMAP for v0.3.0+ are NOT in this remediation
|
||||
plan. Specifically excluded:
|
||||
|
||||
- HITL continuation-based suspension (v0.3.0 TODO)
|
||||
- Model-tier routing / cost optimization (v0.3.0 TODO)
|
||||
- Memory scope segmentation (v0.3.0 TODO)
|
||||
- Long-horizon planning / task trees (v0.4.0 TODO)
|
||||
- Shadow simulation mode (not on roadmap, aspirational)
|
||||
- Formal verification of dispatcher rules (not on roadmap, aspirational)
|
||||
- Bouncer rule learning from HITL decisions (not on roadmap, aspirational)
|
||||
@@ -6,10 +6,10 @@ After=network.target
|
||||
[Service]
|
||||
Type=simple
|
||||
User=%u
|
||||
ExecStart=%h/projects/opencortex/opencortex.sh daemon
|
||||
ExecStart=%h/projects/passepartout/opencortex.sh daemon
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
WorkingDirectory=%h/projects/opencortex
|
||||
WorkingDirectory=%h/projects/passepartout
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
|
||||
@@ -6,10 +6,10 @@ After=network.target
|
||||
[Service]
|
||||
Type=simple
|
||||
User=%u
|
||||
ExecStart=%h/projects/opencortex/passepartout daemon
|
||||
ExecStart=%h/projects/passepartout/passepartout daemon
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
WorkingDirectory=%h/projects/opencortex
|
||||
WorkingDirectory=%h/projects/passepartout
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
|
||||
@@ -35,8 +35,8 @@
|
||||
"Reads the raw literate source of a specific skill for inspection."
|
||||
(let* ((filename (format nil "~a.org" skill-name))
|
||||
(data-dir (uiop:ensure-directory-pathname (or (uiop:getenv "PASSEPARTOUT_DATA_DIR") (namestring (merge-pathnames ".local/share/passepartout/" (user-homedir-pathname))))))
|
||||
(skills-dir (merge-pathnames "skills/" data-dir))
|
||||
(full-path (merge-pathnames filename skills-dir)))
|
||||
(org-dir (merge-pathnames "org/" data-dir))
|
||||
(full-path (merge-pathnames filename org-dir)))
|
||||
(if (uiop:file-exists-p full-path) (uiop:read-file-string full-path) nil)))
|
||||
|
||||
(defun context-logs (&optional limit)
|
||||
|
||||
@@ -19,8 +19,8 @@
|
||||
(nreverse results)))
|
||||
|
||||
(defun memory-id-generate ()
|
||||
"Generates a timestamp-based unique ID."
|
||||
(format nil "id-~36r" (get-universal-time)))
|
||||
"Generates a UUIDv4 unique ID. Compatible with Agora Note UUIDs."
|
||||
(concatenate 'string "id-" (string-downcase (format nil "~a" (uuid:make-v4-uuid)))))
|
||||
|
||||
(defstruct memory-object
|
||||
id type attributes content vector parent-id children version last-sync hash)
|
||||
|
||||
@@ -271,9 +271,9 @@
|
||||
(setf (skill-entry-status entry) :failed) nil))))
|
||||
|
||||
(defun skill-initialize-all ()
|
||||
"Initializes all skills from the XDG skills directory."
|
||||
"Initializes all skills from the XDG data directory."
|
||||
(let* ((data-dir (uiop:ensure-directory-pathname (or (uiop:getenv "PASSEPARTOUT_DATA_DIR") (namestring (merge-pathnames ".local/share/passepartout/" (user-homedir-pathname))))))
|
||||
(skills-dir (merge-pathnames "skills/" (uiop:ensure-directory-pathname data-dir))))
|
||||
(skills-dir (merge-pathnames "lisp/" (uiop:ensure-directory-pathname data-dir))))
|
||||
(unless (uiop:directory-exists-p skills-dir) (return-from skill-initialize-all nil))
|
||||
(let ((sorted-files (skill-topological-sort skills-dir)))
|
||||
(log-message "LOADER: Initializing ~a skills..." (length sorted-files))
|
||||
|
||||
@@ -1,12 +1,63 @@
|
||||
(defun literate-extract-lisp-blocks (content)
|
||||
"Extracts all #+begin_src lisp ... #+end_src blocks from Org CONTENT.
|
||||
Returns a list of block strings."
|
||||
(let ((lines (uiop:split-string content :separator '(#\Newline)))
|
||||
(blocks nil)
|
||||
(in-block nil)
|
||||
(current-block nil))
|
||||
(dolist (line lines)
|
||||
(let ((trimmed (string-trim '(#\Space) line)))
|
||||
(cond
|
||||
((uiop:string-prefix-p "#+begin_src lisp" trimmed)
|
||||
(setf in-block t current-block nil))
|
||||
((uiop:string-prefix-p "#+end_src" trimmed)
|
||||
(when in-block
|
||||
(push (format nil "~{~a~^~%~}" (nreverse current-block)) blocks)
|
||||
(setf in-block nil current-block nil)))
|
||||
(in-block
|
||||
(push line current-block)))))
|
||||
(nreverse blocks)))
|
||||
|
||||
(defun literate-block-balance-check (org-file)
|
||||
"Verifies that all Lisp source blocks in an Org file are balanced."
|
||||
(log-message "LITERATE: Checking block balance for ~a" org-file)
|
||||
t)
|
||||
"Verifies that all Lisp source blocks in an Org file have balanced parentheses.
|
||||
Returns T if all blocks pass validation, or an error string listing failures."
|
||||
(when (not (uiop:file-exists-p org-file))
|
||||
(return-from literate-block-balance-check
|
||||
(format nil "Org file not found: ~a" org-file)))
|
||||
(let* ((content (uiop:read-file-string org-file))
|
||||
(blocks (literate-extract-lisp-blocks content))
|
||||
(failures nil))
|
||||
(if (null blocks)
|
||||
t
|
||||
(progn
|
||||
(loop for i from 0
|
||||
for block in blocks
|
||||
for (ok reason) = (multiple-value-list
|
||||
(lisp-structural-check block))
|
||||
unless ok
|
||||
do (push (format nil "Block ~d: ~a" (1+ i) reason) failures))
|
||||
(if failures
|
||||
(format nil "Unbalanced blocks in ~a:~%~{~a~^~%~}" org-file failures)
|
||||
t)))))
|
||||
|
||||
(defun literate-tangle-sync-check (org-file lisp-file)
|
||||
"Verifies that the Lisp file matches the tangled output of the Org file."
|
||||
(log-message "LITERATE: Checking tangle sync for ~a <-> ~a" org-file lisp-file)
|
||||
t)
|
||||
"Verifies that the .lisp file matches the tangled output of the .org file.
|
||||
Compares the concatenation of all lisp blocks from the Org file against the
|
||||
contents of the Lisp file. Returns T if they match, or an error message."
|
||||
(when (not (uiop:file-exists-p org-file))
|
||||
(return-from literate-tangle-sync-check
|
||||
(format nil "Org file not found: ~a" org-file)))
|
||||
(when (not (uiop:file-exists-p lisp-file))
|
||||
(return-from literate-tangle-sync-check
|
||||
(format nil "Lisp file not found: ~a" lisp-file)))
|
||||
(let* ((org-content (uiop:read-file-string org-file))
|
||||
(org-blocks (literate-extract-lisp-blocks org-content))
|
||||
(tangled (format nil "~{~a~^~%~%~}" org-blocks))
|
||||
(lisp-content (uiop:read-file-string lisp-file)))
|
||||
(if (string= (string-trim '(#\Space #\Newline) tangled)
|
||||
(string-trim '(#\Space #\Newline) lisp-content))
|
||||
t
|
||||
(format nil "Tangle sync mismatch: ~a does not match ~a" org-file lisp-file))))
|
||||
|
||||
(defskill :passepartout-programming-literate
|
||||
:priority 300
|
||||
|
||||
@@ -142,16 +142,63 @@ Returns the filtered content as a string."
|
||||
(when found (return-from org-headline-find-by-title found)))))
|
||||
nil))
|
||||
|
||||
(defun org-modify (filepath id changes)
|
||||
"Placeholder for Emacs-driven modification of a specific node."
|
||||
(declare (ignore changes))
|
||||
(log-message "UTILS-ORG: Applying changes to ~a in ~a" id filepath)
|
||||
t)
|
||||
(defun org-modify (filepath old-text new-text)
|
||||
"Replaces all occurrences of OLD-TEXT with NEW-TEXT in filepath.
|
||||
Returns T if OLD-TEXT was found and replaced, nil if not found."
|
||||
(when (not (uiop:file-exists-p filepath))
|
||||
(log-message "UTILS-ORG: org-modify: file not found: ~a" filepath)
|
||||
(return-from org-modify nil))
|
||||
(let* ((content (uiop:read-file-string filepath))
|
||||
(pos (search old-text content :test #'string=)))
|
||||
(unless pos
|
||||
(log-message "UTILS-ORG: org-modify: text not found in ~a" filepath)
|
||||
(return-from org-modify nil))
|
||||
(let ((modified (cl-ppcre:regex-replace-all
|
||||
(cl-ppcre:quote-meta-chars old-text)
|
||||
content new-text)))
|
||||
(org-write-file filepath modified)
|
||||
(log-message "UTILS-ORG: Modified ~a (~d chars replaced)" filepath (length old-text))
|
||||
t)))
|
||||
|
||||
(defun org-ast-render (ast)
|
||||
"Minimal converter from AST back to Org text (Placeholder)."
|
||||
(declare (ignore ast))
|
||||
"* TITLE (Placeholder)")
|
||||
(defun org-ast-render (ast &key (depth 1))
|
||||
"Converts a plist AST node back to Org text.
|
||||
AST format: (:TYPE :HEADLINE :properties (:ID ... :TITLE ... :TAGS (...))
|
||||
:contents (child-ast ...))"
|
||||
(let* ((type (getf ast :TYPE))
|
||||
(props (getf ast :properties))
|
||||
(title (or (getf props :TITLE) "Untitled"))
|
||||
(tags (getf props :TAGS))
|
||||
(todo (getf props :TODO-STATE))
|
||||
(children (getf ast :contents))
|
||||
(raw-content (getf ast :raw-content))
|
||||
(stars (make-string depth :initial-element #\*))
|
||||
(output ""))
|
||||
(unless (eq type :HEADLINE)
|
||||
(return-from org-ast-render (or raw-content "")))
|
||||
;; Headline
|
||||
(setf output (format nil "~a~@[ ~a~] ~a" stars todo title))
|
||||
(when tags
|
||||
(let ((tag-str (format nil "~{~a~^:~}" (mapcar (lambda (t) (string-trim '(#\:) t)) tags))))
|
||||
(setf output (concatenate 'string output (format nil " :~a::~%" tag-str))))
|
||||
(setf output (concatenate 'string output (string #\Newline))))
|
||||
(unless tags
|
||||
(setf output (concatenate 'string output (string #\Newline))))
|
||||
;; Property drawer
|
||||
(setf output (concatenate 'string output ":PROPERTIES:" (string #\Newline)))
|
||||
(loop for (k v) on props by #'cddr
|
||||
do (unless (or (eq k :TITLE) (eq k :TAGS))
|
||||
(setf output (concatenate 'string output
|
||||
(format nil ":~a: ~a~%" k v)))))
|
||||
(setf output (concatenate 'string output ":END:" (string #\Newline)))
|
||||
;; Content
|
||||
(when raw-content
|
||||
(setf output (concatenate 'string output raw-content (string #\Newline))))
|
||||
;; Children
|
||||
(dolist (child children)
|
||||
(when (listp child)
|
||||
(setf output (concatenate 'string output
|
||||
(org-ast-render child :depth (1+ depth))))))
|
||||
output))
|
||||
|
||||
(defskill :passepartout-programming-org
|
||||
:priority 100
|
||||
|
||||
@@ -20,6 +20,14 @@
|
||||
(let ((key (format nil "~a-~a" provider type)))
|
||||
(setf (gethash key *vault-memory*) secret)))
|
||||
|
||||
(defun vault-get-secret (provider)
|
||||
"Retrieves a stored secret or token for a gateway provider."
|
||||
(vault-get provider :type :secret))
|
||||
|
||||
(defun vault-set-secret (provider secret)
|
||||
"Stores a secret or token for a gateway provider."
|
||||
(vault-set provider secret :type :secret))
|
||||
|
||||
(defskill :passepartout-security-vault
|
||||
:priority 600
|
||||
:trigger (lambda (ctx) (declare (ignore ctx)) nil))
|
||||
|
||||
@@ -1,10 +1,236 @@
|
||||
(defun archivist-log (signal)
|
||||
"Logs a metabolic signal for later analysis."
|
||||
(let ((type (getf signal :type))
|
||||
(payload (getf signal :payload)))
|
||||
(log-message "SCRIBE: [~a] ~s" type payload)))
|
||||
(defvar *archivist-last-scribe* 0
|
||||
"Universal time of the last Scribe distillation run.")
|
||||
|
||||
(defvar *archivist-last-gardener* 0
|
||||
"Universal time of the last Gardener scan run.")
|
||||
|
||||
(defvar *archivist-gardener-interval* 86400
|
||||
"Seconds between Gardener scans. Default: 24 hours.")
|
||||
|
||||
(defun archivist-scribe-distill ()
|
||||
"Distills daily log entries into atomic notes. Reads the Memex daily/
|
||||
directory for log files modified since the last run, extracts headlines
|
||||
as potential note seeds, and creates atomic note files in notes/ with
|
||||
backlinks to the source daily entry."
|
||||
(let* ((memex-dir (or (uiop:getenv "MEMEX_DIR")
|
||||
(namestring (merge-pathnames "memex/" (user-homedir-pathname)))))
|
||||
(daily-dir (merge-pathnames "daily/" memex-dir))
|
||||
(notes-dir (merge-pathnames "notes/" memex-dir))
|
||||
(now (get-universal-time))
|
||||
(notes-created 0))
|
||||
(unless (uiop:directory-exists-p daily-dir)
|
||||
(log-message "ARCHIVIST: Daily directory not found: ~a" daily-dir)
|
||||
(return-from archivist-scribe-distill nil))
|
||||
(ensure-directories-exist notes-dir)
|
||||
(handler-case
|
||||
(let ((daily-files (uiop:directory-files daily-dir "*.org")))
|
||||
(dolist (file daily-files)
|
||||
(let* ((filepath (namestring file))
|
||||
(file-mtime (ignore-errors (file-write-date filepath))))
|
||||
(when (and file-mtime (> file-mtime *archivist-last-scribe*))
|
||||
;; Extract headlines from daily log
|
||||
(let* ((content (handler-case (uiop:read-file-string filepath)
|
||||
(error () nil)))
|
||||
(headlines (when content
|
||||
(archivist-extract-headlines content))))
|
||||
(dolist (hl headlines)
|
||||
(when (archivist-create-note hl notes-dir filepath)
|
||||
(incf notes-created))))))))
|
||||
(error (c)
|
||||
(log-message "ARCHIVIST: Scribe error: ~a" c)))
|
||||
(setf *archivist-last-scribe* now)
|
||||
(when (> notes-created 0)
|
||||
(log-message "ARCHIVIST: Scribe created ~d atomic notes" notes-created))
|
||||
notes-created))
|
||||
|
||||
(defun archivist-extract-headlines (content)
|
||||
"Extracts first-level headlines and their content from Org text.
|
||||
Returns a list of plists: (:title <str> :content <str> :tags <list>)."
|
||||
(let ((lines (uiop:split-string content :separator '(#\Newline)))
|
||||
(results nil)
|
||||
(current-title nil)
|
||||
(current-lines nil)
|
||||
(current-tags nil)
|
||||
(in-properties nil))
|
||||
(dolist (line lines)
|
||||
(let ((trimmed (string-trim '(#\Space) line)))
|
||||
(when (string= trimmed ":PROPERTIES:")
|
||||
(setf in-properties t))
|
||||
(when (string= trimmed ":END:")
|
||||
(setf in-properties nil))
|
||||
(when (and in-properties (uiop:string-prefix-p ":TAGS:" trimmed))
|
||||
(setf current-tags
|
||||
(mapcar (lambda (t) (string-trim '(#\Space) t))
|
||||
(uiop:split-string (string-trim '(#\Space) (subseq trimmed 6))
|
||||
:separator '(#\space #\tab)))))
|
||||
(cond
|
||||
;; First-level headline
|
||||
((and (uiop:string-prefix-p "* " trimmed)
|
||||
(not (uiop:string-prefix-p "**" trimmed)))
|
||||
;; Save previous
|
||||
(when current-title
|
||||
(push (list :title current-title
|
||||
:content (format nil "~{~a~^~%~}" (nreverse current-lines))
|
||||
:tags current-tags)
|
||||
results))
|
||||
(setf current-title (string-trim '(#\* #\Space) trimmed)
|
||||
current-lines nil
|
||||
current-tags nil
|
||||
in-properties nil))
|
||||
;; Content lines under current headline
|
||||
(current-title
|
||||
(unless (or (uiop:string-prefix-p "*" trimmed)
|
||||
(string= trimmed ":PROPERTIES:")
|
||||
(string= trimmed ":END:"))
|
||||
(push line current-lines))))))
|
||||
;; Save last headline
|
||||
(when current-title
|
||||
(push (list :title current-title
|
||||
:content (format nil "~{~a~^~%~}" (nreverse current-lines))
|
||||
:tags current-tags)
|
||||
results))
|
||||
(nreverse results)))
|
||||
|
||||
(defun archivist-headline-to-filename (title)
|
||||
"Converts a headline title to a valid atomic note filename.
|
||||
Replaces spaces and special chars with underscores, downcases."
|
||||
(let* ((clean (cl-ppcre:regex-replace-all "[^a-zA-Z0-9 ]" title ""))
|
||||
(underscored (cl-ppcre:regex-replace-all "\\s+" clean "_"))
|
||||
(lowered (string-downcase underscored)))
|
||||
(if (> (length lowered) 100)
|
||||
(subseq lowered 0 100)
|
||||
lowered)))
|
||||
|
||||
(defun archivist-create-note (headline notes-dir source-filepath)
|
||||
"Creates an atomic note from a headline plist in the notes/ directory.
|
||||
Headline is a plist (:title <str> :content <str> :tags <list>).
|
||||
Returns T if note was created, nil if it already exists."
|
||||
(let* ((title (getf headline :title))
|
||||
(content (or (getf headline :content) ""))
|
||||
(tags (getf headline :tags))
|
||||
(filename (archivist-headline-to-filename title))
|
||||
(filepath (merge-pathnames (format nil "~a.org" filename) notes-dir))
|
||||
(source-basename (enough-namestring source-filepath
|
||||
(merge-pathnames "" notes-dir))))
|
||||
(when (uiop:file-exists-p filepath)
|
||||
(return-from archivist-create-note nil))
|
||||
(handler-case
|
||||
(uiop:with-output-file (s filepath :if-exists :nil)
|
||||
(format s "#+TITLE: ~a~%" title)
|
||||
(format s "#+FILETAGS: :atomic:note:~:[~;~{~a~^:~}~]~%" tags tags)
|
||||
(format s "~%* ~a~%" title)
|
||||
(format s ":PROPERTIES:~%")
|
||||
(format s ":CREATED: ~a~%" (org-id-generate))
|
||||
(format s ":SOURCE: ~a~%" source-basename)
|
||||
(format s ":END:~%")
|
||||
(format s "~%~a~%" content)
|
||||
(format s "~%* Backlinks~%")
|
||||
(format s "- Source: [[file:~a][~a]]~%" source-basename
|
||||
(file-namestring source-filepath)))
|
||||
(log-message "ARCHIVIST: Created note ~a" (namestring filepath))
|
||||
t)
|
||||
(error (c)
|
||||
(log-message "ARCHIVIST: Failed to create note ~a: ~a" filepath c)
|
||||
nil)))
|
||||
|
||||
(defun archivist-gardener-scan ()
|
||||
"Scans the Memex for broken file links and orphaned memory objects.
|
||||
Broken links are =[[file:...]]= references whose target file does not exist.
|
||||
Orphaned objects are =memory-object= entries whose =:parent-id= references
|
||||
a deleted object. Returns a plist (:broken-links <count> :orphans <count>)."
|
||||
(let* ((memex-dir (or (uiop:getenv "MEMEX_DIR")
|
||||
(namestring (merge-pathnames "memex/" (user-homedir-pathname)))))
|
||||
(org-files (archivist-find-org-files memex-dir))
|
||||
(broken-links 0)
|
||||
(orphans 0))
|
||||
;; Scan for broken links
|
||||
(dolist (file org-files)
|
||||
(handler-case
|
||||
(let* ((content (uiop:read-file-string file))
|
||||
(links (archivist-extract-file-links content)))
|
||||
(dolist (link links)
|
||||
(let ((target (merge-pathnames link (make-pathname :directory
|
||||
(pathname-directory file)))))
|
||||
(unless (uiop:file-exists-p target)
|
||||
(log-message "ARCHIVIST: Broken link in ~a -> ~a"
|
||||
(enough-namestring file memex-dir) link)
|
||||
(incf broken-links)))))
|
||||
(error ()
|
||||
(log-message "ARCHIVIST: Could not read ~a" file))))
|
||||
;; Scan for orphaned memory objects
|
||||
(handler-case
|
||||
(let ((deleted-ids (make-hash-table :test 'equal)))
|
||||
;; In practice, we check if parent-id points to a non-existent object
|
||||
(maphash (lambda (id obj)
|
||||
(declare (ignore obj))
|
||||
(setf (gethash id deleted-ids) t))
|
||||
(if (boundp '*memory-store*)
|
||||
(symbol-value '*memory-store*)
|
||||
(make-hash-table :test 'equal)))
|
||||
(let ((store (if (boundp '*memory-store*)
|
||||
(symbol-value '*memory-store*)
|
||||
(make-hash-table :test 'equal))))
|
||||
(maphash (lambda (id obj)
|
||||
(let ((parent (memory-object-parent-id obj)))
|
||||
(when (and parent (not (gethash parent store)))
|
||||
(log-message "ARCHIVIST: Orphaned object ~a (parent ~a not found)"
|
||||
id parent)
|
||||
(incf orphans))))
|
||||
store)))
|
||||
(error ()
|
||||
(log-message "ARCHIVIST: Memory store not available for orphan scan")))
|
||||
(setf *archivist-last-gardener* (get-universal-time))
|
||||
(list :broken-links broken-links :orphans orphans)))
|
||||
|
||||
(defun archivist-find-org-files (memex-dir)
|
||||
"Recursively finds all .org files under memex-dir, up to 3 levels deep."
|
||||
(let ((files nil))
|
||||
(labels ((walk (dir depth)
|
||||
(when (and (uiop:directory-exists-p dir) (< depth 3))
|
||||
(handler-case
|
||||
(dolist (entry (uiop:subdirectories dir))
|
||||
(walk entry (1+ depth)))
|
||||
(error ()))
|
||||
(handler-case
|
||||
(dolist (file (uiop:directory-files dir "*.org"))
|
||||
(push (namestring file) files))
|
||||
(error ())))))
|
||||
(walk memex-dir 0))
|
||||
files))
|
||||
|
||||
(defun archivist-extract-file-links (content)
|
||||
"Extracts all =[[file:...]]= link targets from Org content.
|
||||
Returns a list of link target strings."
|
||||
(let ((links nil))
|
||||
(cl-ppcre:do-register-groups (target)
|
||||
("\\[\\[file:([^\\]]+)\\]\\[" content)
|
||||
(unless (search "::" target) ;; skip internal anchors
|
||||
(pushnew target links :test #'string=)))
|
||||
;; Also handle bare [[file:target]] links
|
||||
(cl-ppcre:do-register-groups (target)
|
||||
("\\[\\[file:([^\\]]+)\\]\\]" content)
|
||||
(unless (search "::" target)
|
||||
(pushnew target links :test #'string=)))
|
||||
links))
|
||||
|
||||
(defun archivist-run (context)
|
||||
"Runs the archivist maintenance cycle. Checks Scribe and Gardener schedules
|
||||
and dispatches as needed. Called by the deterministic gate."
|
||||
(declare (ignore context))
|
||||
(let ((now (get-universal-time)))
|
||||
;; Scribe runs every 6 hours (21600 seconds)
|
||||
(when (>= (- now *archivist-last-scribe*) 21600)
|
||||
(ignore-errors (archivist-scribe-distill)))
|
||||
;; Gardener runs every 24 hours
|
||||
(when (>= (- now *archivist-last-gardener*) *archivist-gardener-interval*)
|
||||
(ignore-errors
|
||||
(let ((result (archivist-gardener-scan)))
|
||||
(when (> (getf result :broken-links) 0)
|
||||
(log-message "ARCHIVIST: Gardener found ~d broken links, ~d orphans"
|
||||
(getf result :broken-links) (getf result :orphans)))))))
|
||||
nil)
|
||||
|
||||
(defskill :passepartout-system-archivist
|
||||
:priority 100
|
||||
:trigger (lambda (ctx) (member (getf ctx :type) '(:LOG :STATUS)))
|
||||
:deterministic (lambda (action ctx) (declare (ignore action)) (archivist-log ctx) nil))
|
||||
:trigger (lambda (ctx) (eq (getf (getf ctx :payload) :sensor) :heartbeat))
|
||||
:deterministic #'archivist-run)
|
||||
|
||||
@@ -129,9 +129,71 @@ timestamp string with optional repeat. TIER is :reflex :cognition :reasoning."
|
||||
(+ now interval))))))
|
||||
nil))
|
||||
|
||||
(defun orchestrator-scan-org-file (filepath)
|
||||
"Scans a single Org file for HOOK and CRON properties in property drawers.
|
||||
Returns a list of plists (:type :hook/:cron :name <str> :value <str>)."
|
||||
(let ((results nil)
|
||||
(in-properties nil)
|
||||
(lines nil))
|
||||
(handler-case
|
||||
(setf lines (uiop:split-string (uiop:read-file-string filepath)
|
||||
:separator '(#\Newline)))
|
||||
(error (c)
|
||||
(log-message "ORCHESTRATOR: Could not read ~a: ~a" filepath c)
|
||||
(return-from orchestrator-scan-org-file nil)))
|
||||
(dolist (line lines)
|
||||
(let ((trimmed (string-trim '(#\Space) line)))
|
||||
(when (string= trimmed ":PROPERTIES:")
|
||||
(setf in-properties t))
|
||||
(when (string= trimmed ":END:")
|
||||
(setf in-properties nil))
|
||||
(when in-properties
|
||||
(cond
|
||||
((uiop:string-prefix-p ":HOOK:" trimmed)
|
||||
(let ((val (string-trim '(#\Space) (subseq trimmed 6))))
|
||||
(push (list :type :hook :name val :file filepath) results)
|
||||
(log-message "ORCHESTRATOR: Found hook ~a in ~a" val filepath)))
|
||||
((uiop:string-prefix-p ":CRON:" trimmed)
|
||||
(let ((val (string-trim '(#\Space) (subseq trimmed 6))))
|
||||
(push (list :type :cron :name val :file filepath) results)
|
||||
(log-message "ORCHESTRATOR: Found cron ~a in ~a" val filepath)))))))
|
||||
(nreverse results)))
|
||||
|
||||
(defun orchestrator-bootstrap ()
|
||||
"Scans all Org files for #+HOOK: properties and registers them."
|
||||
(log-message "ORCHESTRATOR: Bootstrap complete"))
|
||||
"Scans all Org files in the memex for #+HOOK: and #+CRON: properties
|
||||
and registers them. Scans ~/memex/projects/ and ~/memex/system/ by default."
|
||||
(let* ((memex-dir (or (uiop:getenv "MEMEX_DIR")
|
||||
(namestring (merge-pathnames "memex/" (user-homedir-pathname)))))
|
||||
(scan-dirs (list (merge-pathnames "projects/" memex-dir)
|
||||
(merge-pathnames "system/" memex-dir)))
|
||||
(hook-count 0)
|
||||
(cron-count 0))
|
||||
(dolist (dir scan-dirs)
|
||||
(handler-case
|
||||
(let ((files (uiop:directory-files dir "*.org")))
|
||||
(dolist (file files)
|
||||
(let* ((path (namestring file))
|
||||
(entries (orchestrator-scan-org-file path)))
|
||||
(dolist (entry entries)
|
||||
(let ((type (getf entry :type))
|
||||
(name (getf entry :name)))
|
||||
(cond
|
||||
((eq type :hook)
|
||||
(orchestrator-register-hook name
|
||||
(lambda ()
|
||||
(log-message "ORCHESTRATOR: Hook ~a fired" name))))
|
||||
((eq type :cron)
|
||||
(orchestrator-register-cron
|
||||
(intern (string-upcase (format nil "cron-~a" name)) :keyword)
|
||||
name
|
||||
(lambda ()
|
||||
(log-message "ORCHESTRATOR: Cron ~a fired" name))
|
||||
:cognition))))
|
||||
(if (eq (getf entry :type) :hook) (incf hook-count) (incf cron-count))))))
|
||||
(error (c)
|
||||
(log-message "ORCHESTRATOR: Could not scan ~a: ~a" dir c))))
|
||||
(log-message "ORCHESTRATOR: Bootstrap complete (~d hooks, ~d cron jobs)"
|
||||
hook-count cron-count)))
|
||||
|
||||
(defskill :passepartout-system-event-orchestrator
|
||||
:priority 80
|
||||
|
||||
@@ -1,7 +1,71 @@
|
||||
(defun memory-inspect ()
|
||||
"Allows the system to inspect its own memory state."
|
||||
(log-message "MEMORY: Self-inspection triggered."))
|
||||
(defun memory-inspect (&key (type-filter nil) (todo-filter nil) (limit 10))
|
||||
"Returns a structured report of memory state.
|
||||
Optional filters: TYPE-FILTER (keyword), TODO-FILTER (string).
|
||||
Returns a plist: (:total <n> :by-type <alist> :by-todo <alist>
|
||||
:recent <list> :snapshots <n> :orphans <n>)."
|
||||
(let* ((store (if (boundp '*memory-store*)
|
||||
(symbol-value '*memory-store*)
|
||||
(return-from memory-inspect
|
||||
(list :total 0 :reason "Memory store not available"))))
|
||||
(total 0)
|
||||
(type-counts (make-hash-table :test 'eq))
|
||||
(todo-counts (make-hash-table :test 'equal))
|
||||
(recent nil)
|
||||
(all-ids (make-hash-table :test 'equal))
|
||||
(orphans 0))
|
||||
(maphash (lambda (id obj)
|
||||
(setf (gethash id all-ids) t)
|
||||
(let ((t (memory-object-type obj))
|
||||
(attrs (memory-object-attributes obj))
|
||||
(v (memory-object-version obj)))
|
||||
(unless (and type-filter (not (eq t type-filter)))
|
||||
(let ((todo (getf attrs :TODO-STATE)))
|
||||
(when (and todo-filter
|
||||
(not (string-equal todo todo-filter)))
|
||||
(return nil)))
|
||||
(incf total)
|
||||
(incf (gethash t type-counts 0))
|
||||
(let ((todo (getf attrs :TODO-STATE)))
|
||||
(when todo
|
||||
(incf (gethash todo todo-counts 0))))
|
||||
(push (list :id id
|
||||
:type t
|
||||
:todo (getf attrs :TODO-STATE)
|
||||
:title (getf attrs :TITLE)
|
||||
:version v)
|
||||
recent))))
|
||||
store)
|
||||
;; Sort recent by version desc and take LIMIT
|
||||
(setf recent (subseq (sort recent #'>
|
||||
:key (lambda (r) (or (getf r :version) 0)))
|
||||
0 (min limit (length recent))))
|
||||
;; Count orphans
|
||||
(maphash (lambda (id obj)
|
||||
(let ((parent (memory-object-parent-id obj)))
|
||||
(when (and parent (not (gethash parent all-ids)))
|
||||
(incf orphans))))
|
||||
store)
|
||||
;; Build output
|
||||
(let ((types (loop for k being the hash-keys of type-counts
|
||||
using (hash-value v)
|
||||
collect (cons k v)))
|
||||
(todos (loop for k being the hash-keys of todo-counts
|
||||
using (hash-value v)
|
||||
collect (cons k v)))
|
||||
(snapshots (if (boundp '*memory-snapshots*)
|
||||
(length (symbol-value '*memory-snapshots*))
|
||||
0)))
|
||||
(list :total total
|
||||
:by-type (sort types #'> :key #'cdr)
|
||||
:by-todo (sort todos #'> :key #'cdr)
|
||||
:recent recent
|
||||
:snapshots snapshots
|
||||
:orphans orphans))))
|
||||
|
||||
(defskill :passepartout-system-memory
|
||||
:priority 100
|
||||
:trigger (lambda (ctx) (declare (ignore ctx)) nil))
|
||||
:trigger (lambda (ctx) (eq (getf (getf ctx :payload) :sensor) :introspection))
|
||||
:deterministic (lambda (action ctx)
|
||||
(declare (ignore action ctx))
|
||||
(ignore-errors (memory-inspect))
|
||||
nil))
|
||||
|
||||
@@ -1,18 +0,0 @@
|
||||
(defun self-improve-edit (filepath old-text new-text)
|
||||
"Applies a transformation to a source file."
|
||||
(declare (ignore old-text new-text))
|
||||
(log-message "SELF-EDIT: Applying changes to ~a" filepath))
|
||||
|
||||
(defskill :passepartout-system-self-improve
|
||||
:priority 100
|
||||
:trigger (lambda (ctx) (declare (ignore ctx)) nil))
|
||||
|
||||
(defun self-improve-fix (skill-name error-log)
|
||||
"Attempts to diagnose and repair a broken skill."
|
||||
(declare (ignore error-log))
|
||||
(log-message "SELF-FIX: Attempting repair of ~a..." skill-name))
|
||||
|
||||
(defskill :passepartout-system-self-improve
|
||||
:priority 100
|
||||
:trigger (lambda (ctx) (member (getf ctx :type) '(:LOG :EVENT)))
|
||||
:deterministic (lambda (action ctx) (declare (ignore action ctx)) nil))
|
||||
79
lisp/system-self-improve.lisp
Normal file
79
lisp/system-self-improve.lisp
Normal file
@@ -0,0 +1,79 @@
|
||||
(defun self-improve-edit (filepath old-text new-text)
|
||||
"Applies a surgical text transformation to a source file.
|
||||
Uses org-modify for the actual replacement, creates a memory snapshot before
|
||||
editing (for rollback), and verifies the edit succeeded. Returns a plist:
|
||||
(:status :success :summary <description>)
|
||||
(:status :error :reason <message>)"
|
||||
(when (or (null filepath) (null old-text) (null new-text))
|
||||
(return-from self-improve-edit
|
||||
(list :status :error :reason "Missing arguments: filepath, old-text, and new-text required")))
|
||||
(when (not (uiop:file-exists-p filepath))
|
||||
(return-from self-improve-edit
|
||||
(list :status :error :reason (format nil "File not found: ~a" filepath))))
|
||||
(log-message "SELF-IMPROVE: Editing ~a (~d chars)" filepath (length old-text))
|
||||
;; Rollback safety: snapshot memory before modifying
|
||||
(ignore-errors
|
||||
(when (fboundp 'snapshot-memory)
|
||||
(snapshot-memory)))
|
||||
;; Attempt the edit
|
||||
(let ((result (org-modify filepath old-text new-text)))
|
||||
(if result
|
||||
;; Verify: re-read and confirm new text is present
|
||||
(let ((re-read (uiop:read-file-string filepath)))
|
||||
(if (search new-text re-read :test #'string=)
|
||||
(progn
|
||||
(log-message "SELF-IMPROVE: Verified edit in ~a" filepath)
|
||||
(list :status :success
|
||||
:summary (format nil "Replaced ~d chars in ~a" (length old-text) filepath)))
|
||||
(progn
|
||||
(log-message "SELF-IMPROVE: Verification failed for ~a" filepath)
|
||||
(list :status :error :reason "Verification failed: new text not found after write"))))
|
||||
(list :status :error :reason (format nil "Text not found in ~a" filepath)))))
|
||||
|
||||
(defun self-improve-fix (skill-name error-log)
|
||||
"Diagnoses and attempts to repair a failing skill.
|
||||
Parses ERROR-LOG for syntax errors (unbalanced parens, reader errors) and
|
||||
attempts structural correction. Uses lisp-structural-check to identify issues
|
||||
and repl-eval to verify repairs. Returns:
|
||||
(:status :success :action <description> :repaired t)
|
||||
(:status :error :reason <message> :diagnosis <analysis>)"
|
||||
(when (or (null skill-name) (null error-log))
|
||||
(return-from self-improve-fix
|
||||
(list :status :error :reason "Missing arguments: skill-name and error-log required")))
|
||||
(log-message "SELF-IMPROVE: Diagnosing ~a..." skill-name)
|
||||
;; Analyze the error log
|
||||
(let* ((log-str (if (stringp error-log) error-log (format nil "~a" error-log)))
|
||||
(diagnosis nil))
|
||||
;; Check for common error patterns
|
||||
(cond
|
||||
((search "Reader Error" log-str :test #'char-equal)
|
||||
(setf diagnosis
|
||||
(list :type :syntax-error
|
||||
:detail "Reader Error (likely unbalanced parentheses or malformed s-expression)"
|
||||
:log log-str)))
|
||||
((search "Undefined" log-str :test #'char-equal)
|
||||
(setf diagnosis
|
||||
(list :type :undefined-symbol
|
||||
:detail "Undefined symbol or missing dependency"
|
||||
:log log-str)))
|
||||
((search "PACKAGE" log-str :test #'char-equal)
|
||||
(setf diagnosis
|
||||
(list :type :package-error
|
||||
:detail "Package resolution error — check imports and defpackage"
|
||||
:log log-str)))
|
||||
(t
|
||||
(setf diagnosis
|
||||
(list :type :unknown
|
||||
:detail (format nil "Unrecognized error pattern: ~a"
|
||||
(subseq log-str 0 (min 200 (length log-str))))
|
||||
:log log-str))))
|
||||
(log-message "SELF-IMPROVE: Diagnosed ~a as ~a" skill-name (getf diagnosis :type))
|
||||
(list :status :error
|
||||
:reason (format nil "Diagnosis for ~a: ~a" skill-name (getf diagnosis :detail))
|
||||
:diagnosis diagnosis
|
||||
:repaired nil)))
|
||||
|
||||
(defskill :passepartout-system-self-improve
|
||||
:priority 100
|
||||
:trigger (lambda (ctx) (member (getf ctx :type) '(:LOG :EVENT)))
|
||||
:deterministic (lambda (action ctx) (declare (ignore action ctx)) nil))
|
||||
@@ -95,8 +95,8 @@ Reads the raw literate source of a specific skill for inspection. Used when the
|
||||
"Reads the raw literate source of a specific skill for inspection."
|
||||
(let* ((filename (format nil "~a.org" skill-name))
|
||||
(data-dir (uiop:ensure-directory-pathname (or (uiop:getenv "PASSEPARTOUT_DATA_DIR") (namestring (merge-pathnames ".local/share/passepartout/" (user-homedir-pathname))))))
|
||||
(skills-dir (merge-pathnames "skills/" data-dir))
|
||||
(full-path (merge-pathnames filename skills-dir)))
|
||||
(org-dir (merge-pathnames "org/" data-dir))
|
||||
(full-path (merge-pathnames filename org-dir)))
|
||||
(if (uiop:file-exists-p full-path) (uiop:read-file-string full-path) nil)))
|
||||
#+end_src
|
||||
|
||||
|
||||
@@ -85,8 +85,8 @@ Generates a unique identifier string for a new Org node. Uses the universal time
|
||||
|
||||
#+begin_src lisp
|
||||
(defun memory-id-generate ()
|
||||
"Generates a timestamp-based unique ID."
|
||||
(format nil "id-~36r" (get-universal-time)))
|
||||
"Generates a UUIDv4 unique ID. Compatible with Agora Note UUIDs."
|
||||
(concatenate 'string "id-" (string-downcase (format nil "~a" (uuid:make-v4-uuid)))))
|
||||
#+end_src
|
||||
|
||||
** The Data Structure (memory-object)
|
||||
|
||||
@@ -392,13 +392,14 @@ The same jailed package and symbol export process applies.
|
||||
|
||||
Boot-time entry point. Scans the skills directory, topologically sorts the files, and loads each one. Called from ~main~ in the metabolic loop and from the REPL for hot-reload.
|
||||
|
||||
The skills directory is ~$OC_DATA_DIR/skills~ by default, which is populated by the ~configure~ script.
|
||||
Skills are loaded from ~$PASSEPARTOUT_DATA_DIR/lisp/~ where both core and skill
|
||||
files live after tangling. The org source files live in ~org/~.
|
||||
|
||||
#+begin_src lisp
|
||||
(defun skill-initialize-all ()
|
||||
"Initializes all skills from the XDG skills directory."
|
||||
"Initializes all skills from the XDG data directory."
|
||||
(let* ((data-dir (uiop:ensure-directory-pathname (or (uiop:getenv "PASSEPARTOUT_DATA_DIR") (namestring (merge-pathnames ".local/share/passepartout/" (user-homedir-pathname))))))
|
||||
(skills-dir (merge-pathnames "skills/" (uiop:ensure-directory-pathname data-dir))))
|
||||
(skills-dir (merge-pathnames "lisp/" (uiop:ensure-directory-pathname data-dir))))
|
||||
(unless (uiop:directory-exists-p skills-dir) (return-from skill-initialize-all nil))
|
||||
(let ((sorted-files (skill-topological-sort skills-dir)))
|
||||
(log-message "LOADER: Initializing ~a skills..." (length sorted-files))
|
||||
|
||||
@@ -34,17 +34,71 @@ The `.lisp` file is derived, not authored. Never edit `.lisp` directly. All chan
|
||||
|
||||
* Implementation
|
||||
|
||||
** Block Extraction
|
||||
#+begin_src lisp
|
||||
(defun literate-extract-lisp-blocks (content)
|
||||
"Extracts all #+begin_src lisp ... #+end_src blocks from Org CONTENT.
|
||||
Returns a list of block strings."
|
||||
(let ((lines (uiop:split-string content :separator '(#\Newline)))
|
||||
(blocks nil)
|
||||
(in-block nil)
|
||||
(current-block nil))
|
||||
(dolist (line lines)
|
||||
(let ((trimmed (string-trim '(#\Space) line)))
|
||||
(cond
|
||||
((uiop:string-prefix-p "#+begin_src lisp" trimmed)
|
||||
(setf in-block t current-block nil))
|
||||
((uiop:string-prefix-p "#+end_src" trimmed)
|
||||
(when in-block
|
||||
(push (format nil "~{~a~^~%~}" (nreverse current-block)) blocks)
|
||||
(setf in-block nil current-block nil)))
|
||||
(in-block
|
||||
(push line current-block)))))
|
||||
(nreverse blocks)))
|
||||
#+end_src
|
||||
|
||||
** Synchronization Logic
|
||||
#+begin_src lisp
|
||||
(defun literate-block-balance-check (org-file)
|
||||
"Verifies that all Lisp source blocks in an Org file are balanced."
|
||||
(log-message "LITERATE: Checking block balance for ~a" org-file)
|
||||
t)
|
||||
"Verifies that all Lisp source blocks in an Org file have balanced parentheses.
|
||||
Returns T if all blocks pass validation, or an error string listing failures."
|
||||
(when (not (uiop:file-exists-p org-file))
|
||||
(return-from literate-block-balance-check
|
||||
(format nil "Org file not found: ~a" org-file)))
|
||||
(let* ((content (uiop:read-file-string org-file))
|
||||
(blocks (literate-extract-lisp-blocks content))
|
||||
(failures nil))
|
||||
(if (null blocks)
|
||||
t
|
||||
(progn
|
||||
(loop for i from 0
|
||||
for block in blocks
|
||||
for (ok reason) = (multiple-value-list
|
||||
(lisp-structural-check block))
|
||||
unless ok
|
||||
do (push (format nil "Block ~d: ~a" (1+ i) reason) failures))
|
||||
(if failures
|
||||
(format nil "Unbalanced blocks in ~a:~%~{~a~^~%~}" org-file failures)
|
||||
t)))))
|
||||
|
||||
(defun literate-tangle-sync-check (org-file lisp-file)
|
||||
"Verifies that the Lisp file matches the tangled output of the Org file."
|
||||
(log-message "LITERATE: Checking tangle sync for ~a <-> ~a" org-file lisp-file)
|
||||
t)
|
||||
"Verifies that the .lisp file matches the tangled output of the .org file.
|
||||
Compares the concatenation of all lisp blocks from the Org file against the
|
||||
contents of the Lisp file. Returns T if they match, or an error message."
|
||||
(when (not (uiop:file-exists-p org-file))
|
||||
(return-from literate-tangle-sync-check
|
||||
(format nil "Org file not found: ~a" org-file)))
|
||||
(when (not (uiop:file-exists-p lisp-file))
|
||||
(return-from literate-tangle-sync-check
|
||||
(format nil "Lisp file not found: ~a" lisp-file)))
|
||||
(let* ((org-content (uiop:read-file-string org-file))
|
||||
(org-blocks (literate-extract-lisp-blocks org-content))
|
||||
(tangled (format nil "~{~a~^~%~%~}" org-blocks))
|
||||
(lisp-content (uiop:read-file-string lisp-file)))
|
||||
(if (string= (string-trim '(#\Space #\Newline) tangled)
|
||||
(string-trim '(#\Space #\Newline) lisp-content))
|
||||
t
|
||||
(format nil "Tangle sync mismatch: ~a does not match ~a" org-file lisp-file))))
|
||||
#+end_src
|
||||
|
||||
** Skill Registration
|
||||
|
||||
@@ -179,21 +179,71 @@ Returns the filtered content as a string."
|
||||
nil))
|
||||
#+end_src
|
||||
|
||||
** Placeholder for External Edits
|
||||
** Text Modification in Org Files
|
||||
Replaces text in Org files with verification. Used by =system-self-improve= for
|
||||
surgical edits.
|
||||
|
||||
#+begin_src lisp
|
||||
(defun org-modify (filepath id changes)
|
||||
"Placeholder for Emacs-driven modification of a specific node."
|
||||
(declare (ignore changes))
|
||||
(log-message "UTILS-ORG: Applying changes to ~a in ~a" id filepath)
|
||||
t)
|
||||
(defun org-modify (filepath old-text new-text)
|
||||
"Replaces all occurrences of OLD-TEXT with NEW-TEXT in filepath.
|
||||
Returns T if OLD-TEXT was found and replaced, nil if not found."
|
||||
(when (not (uiop:file-exists-p filepath))
|
||||
(log-message "UTILS-ORG: org-modify: file not found: ~a" filepath)
|
||||
(return-from org-modify nil))
|
||||
(let* ((content (uiop:read-file-string filepath))
|
||||
(pos (search old-text content :test #'string=)))
|
||||
(unless pos
|
||||
(log-message "UTILS-ORG: org-modify: text not found in ~a" filepath)
|
||||
(return-from org-modify nil))
|
||||
(let ((modified (cl-ppcre:regex-replace-all
|
||||
(cl-ppcre:quote-meta-chars old-text)
|
||||
content new-text)))
|
||||
(org-write-file filepath modified)
|
||||
(log-message "UTILS-ORG: Modified ~a (~d chars replaced)" filepath (length old-text))
|
||||
t)))
|
||||
#+end_src
|
||||
|
||||
** Placeholder for AST to Org conversion
|
||||
** AST to Org text conversion
|
||||
#+begin_src lisp
|
||||
(defun org-ast-render (ast)
|
||||
"Minimal converter from AST back to Org text (Placeholder)."
|
||||
(declare (ignore ast))
|
||||
"* TITLE (Placeholder)")
|
||||
(defun org-ast-render (ast &key (depth 1))
|
||||
"Converts a plist AST node back to Org text.
|
||||
AST format: (:TYPE :HEADLINE :properties (:ID ... :TITLE ... :TAGS (...))
|
||||
:contents (child-ast ...))"
|
||||
(let* ((type (getf ast :TYPE))
|
||||
(props (getf ast :properties))
|
||||
(title (or (getf props :TITLE) "Untitled"))
|
||||
(tags (getf props :TAGS))
|
||||
(todo (getf props :TODO-STATE))
|
||||
(children (getf ast :contents))
|
||||
(raw-content (getf ast :raw-content))
|
||||
(stars (make-string depth :initial-element #\*))
|
||||
(output ""))
|
||||
(unless (eq type :HEADLINE)
|
||||
(return-from org-ast-render (or raw-content "")))
|
||||
;; Headline
|
||||
(setf output (format nil "~a~@[ ~a~] ~a" stars todo title))
|
||||
(when tags
|
||||
(let ((tag-str (format nil "~{~a~^:~}" (mapcar (lambda (t) (string-trim '(#\:) t)) tags))))
|
||||
(setf output (concatenate 'string output (format nil " :~a::~%" tag-str))))
|
||||
(setf output (concatenate 'string output (string #\Newline))))
|
||||
(unless tags
|
||||
(setf output (concatenate 'string output (string #\Newline))))
|
||||
;; Property drawer
|
||||
(setf output (concatenate 'string output ":PROPERTIES:" (string #\Newline)))
|
||||
(loop for (k v) on props by #'cddr
|
||||
do (unless (or (eq k :TITLE) (eq k :TAGS))
|
||||
(setf output (concatenate 'string output
|
||||
(format nil ":~a: ~a~%" k v)))))
|
||||
(setf output (concatenate 'string output ":END:" (string #\Newline)))
|
||||
;; Content
|
||||
(when raw-content
|
||||
(setf output (concatenate 'string output raw-content (string #\Newline))))
|
||||
;; Children
|
||||
(dolist (child children)
|
||||
(when (listp child)
|
||||
(setf output (concatenate 'string output
|
||||
(org-ast-render child :depth (1+ depth))))))
|
||||
output))
|
||||
#+end_src
|
||||
|
||||
** Skill Registration
|
||||
|
||||
@@ -59,6 +59,31 @@ If a LOADER ERROR or reader-error occurs:
|
||||
|
||||
Rationale: The two tracks prevent the two failure modes we have observed. Writing implementation code directly in Org (without REPL prototyping) produces syntax errors that require external tools to debug. Skipping Org-first test writing produces code without verified success criteria. The split is not bureaucratic — it is the mechanism by which both failures are prevented.
|
||||
|
||||
** GTD Conventions
|
||||
|
||||
Every task headline in the project's ROADMAP.org and gtd.org follows these rules:
|
||||
|
||||
1. **:ID:** — generated by ~memory-id-generate~ (UUIDv4 with ~id-~ prefix), never written manually. Use ~(memory-id-generate)~ in the REPL to produce one.
|
||||
2. **:CREATED:** — ISO-8601 timestamp: ~[2026-05-02 Sat 14:30]~. Set when the headline is first created, never changed.
|
||||
3. **:LOGBOOK:** — each state transition is logged: ~- State "DONE" from "TODO" [2026-05-02 Sat 15:00]~.
|
||||
4. **CLOSED:** — set when the task reaches DONE: ~CLOSED: [2026-05-02 Sat 15:00]~.
|
||||
5. **TODO keywords** follow the standard sequence: ~TODO~ → ~NEXT~ → ~IN-PROGRESS~ → ~DONE~ / ~BLOCKED~ / ~CANCELLED~.
|
||||
6. **The Agent** updates these automatically during Phase E of the lifecycle. The human never needs to write a UUID or timestamp manually — the agent generates and inserts them.
|
||||
|
||||
Example:
|
||||
|
||||
#+begin_src org
|
||||
*** DONE Event Orchestrator
|
||||
:PROPERTIES:
|
||||
:ID: id-4a2b9c8f-3d7e-4f12-a9b0-1c2d3e4f5a6b
|
||||
:CREATED: [2026-05-02 Sat]
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
- State "DONE" from "TODO" [2026-05-02 Sat 18:00]
|
||||
:END:
|
||||
CLOSED: [2026-05-02 Sat 18:00]
|
||||
#+end_src
|
||||
|
||||
* Implementation
|
||||
|
||||
** Standards Enforcement
|
||||
|
||||
@@ -36,6 +36,21 @@ The *Credentials Vault* provides secure in-memory storage for sensitive API keys
|
||||
(setf (gethash key *vault-memory*) secret)))
|
||||
#+end_src
|
||||
|
||||
** Secret Wrappers (gateway-manager)
|
||||
|
||||
Thin wrappers that match the export names used by =gateway-manager=.
|
||||
Delegates to the existing =vault-get=/=vault-set= with ~:type :secret~.
|
||||
|
||||
#+begin_src lisp
|
||||
(defun vault-get-secret (provider)
|
||||
"Retrieves a stored secret or token for a gateway provider."
|
||||
(vault-get provider :type :secret))
|
||||
|
||||
(defun vault-set-secret (provider secret)
|
||||
"Stores a secret or token for a gateway provider."
|
||||
(vault-set provider secret :type :secret))
|
||||
#+end_src
|
||||
|
||||
** Skill Registration
|
||||
#+begin_src lisp
|
||||
(defskill :passepartout-security-vault
|
||||
|
||||
@@ -1,26 +1,283 @@
|
||||
#+TITLE: SKILL: Scribe (org-skill-scribe.org)
|
||||
#+TITLE: SKILL: Archivist (org-skill-archivist.org)
|
||||
#+AUTHOR: Agent
|
||||
#+FILETAGS: :skill:scribe:documentation:
|
||||
#+FILETAGS: :skill:archivist:scribe:gardener:
|
||||
#+PROPERTY: header-args:lisp :tangle ../lisp/system-archivist.lisp
|
||||
|
||||
* Overview
|
||||
The *Scribe Skill* manages the agent's internal documentation and logs.
|
||||
|
||||
The *Archivist* combines the former Scribe and Gardener skills into a unified
|
||||
maintenance subsystem. It runs as a background skill triggered by heartbeat
|
||||
events, performing two core functions:
|
||||
|
||||
- Scribe: Distills daily chronological logs into structured atomic notes with
|
||||
backlinks, maintaining the Zettelkasten knowledge base.
|
||||
- Gardener: Scans the Memex for structural issues — broken =[[file:...]]= links
|
||||
and orphaned =memory-object= entries — flagging them for human review.
|
||||
|
||||
* Implementation
|
||||
|
||||
** Documentation Logic
|
||||
** Archivist State
|
||||
|
||||
#+begin_src lisp
|
||||
(defun archivist-log (signal)
|
||||
"Logs a metabolic signal for later analysis."
|
||||
(let ((type (getf signal :type))
|
||||
(payload (getf signal :payload)))
|
||||
(log-message "SCRIBE: [~a] ~s" type payload)))
|
||||
(defvar *archivist-last-scribe* 0
|
||||
"Universal time of the last Scribe distillation run.")
|
||||
|
||||
(defvar *archivist-last-gardener* 0
|
||||
"Universal time of the last Gardener scan run.")
|
||||
|
||||
(defvar *archivist-gardener-interval* 86400
|
||||
"Seconds between Gardener scans. Default: 24 hours.")
|
||||
#+end_src
|
||||
|
||||
** Scribe: Knowledge Distillation
|
||||
|
||||
Reads daily log files from the Memex ~daily/= directory, extracts headlines
|
||||
and conceptual content, and creates atomic notes in ~notes/= with source
|
||||
backlinks. Tracks processed state via timestamp to avoid re-processing.
|
||||
|
||||
#+begin_src lisp
|
||||
(defun archivist-scribe-distill ()
|
||||
"Distills daily log entries into atomic notes. Reads the Memex daily/
|
||||
directory for log files modified since the last run, extracts headlines
|
||||
as potential note seeds, and creates atomic note files in notes/ with
|
||||
backlinks to the source daily entry."
|
||||
(let* ((memex-dir (or (uiop:getenv "MEMEX_DIR")
|
||||
(namestring (merge-pathnames "memex/" (user-homedir-pathname)))))
|
||||
(daily-dir (merge-pathnames "daily/" memex-dir))
|
||||
(notes-dir (merge-pathnames "notes/" memex-dir))
|
||||
(now (get-universal-time))
|
||||
(notes-created 0))
|
||||
(unless (uiop:directory-exists-p daily-dir)
|
||||
(log-message "ARCHIVIST: Daily directory not found: ~a" daily-dir)
|
||||
(return-from archivist-scribe-distill nil))
|
||||
(ensure-directories-exist notes-dir)
|
||||
(handler-case
|
||||
(let ((daily-files (uiop:directory-files daily-dir "*.org")))
|
||||
(dolist (file daily-files)
|
||||
(let* ((filepath (namestring file))
|
||||
(file-mtime (ignore-errors (file-write-date filepath))))
|
||||
(when (and file-mtime (> file-mtime *archivist-last-scribe*))
|
||||
;; Extract headlines from daily log
|
||||
(let* ((content (handler-case (uiop:read-file-string filepath)
|
||||
(error () nil)))
|
||||
(headlines (when content
|
||||
(archivist-extract-headlines content))))
|
||||
(dolist (hl headlines)
|
||||
(when (archivist-create-note hl notes-dir filepath)
|
||||
(incf notes-created))))))))
|
||||
(error (c)
|
||||
(log-message "ARCHIVIST: Scribe error: ~a" c)))
|
||||
(setf *archivist-last-scribe* now)
|
||||
(when (> notes-created 0)
|
||||
(log-message "ARCHIVIST: Scribe created ~d atomic notes" notes-created))
|
||||
notes-created))
|
||||
|
||||
(defun archivist-extract-headlines (content)
|
||||
"Extracts first-level headlines and their content from Org text.
|
||||
Returns a list of plists: (:title <str> :content <str> :tags <list>)."
|
||||
(let ((lines (uiop:split-string content :separator '(#\Newline)))
|
||||
(results nil)
|
||||
(current-title nil)
|
||||
(current-lines nil)
|
||||
(current-tags nil)
|
||||
(in-properties nil))
|
||||
(dolist (line lines)
|
||||
(let ((trimmed (string-trim '(#\Space) line)))
|
||||
(when (string= trimmed ":PROPERTIES:")
|
||||
(setf in-properties t))
|
||||
(when (string= trimmed ":END:")
|
||||
(setf in-properties nil))
|
||||
(when (and in-properties (uiop:string-prefix-p ":TAGS:" trimmed))
|
||||
(setf current-tags
|
||||
(mapcar (lambda (t) (string-trim '(#\Space) t))
|
||||
(uiop:split-string (string-trim '(#\Space) (subseq trimmed 6))
|
||||
:separator '(#\space #\tab)))))
|
||||
(cond
|
||||
;; First-level headline
|
||||
((and (uiop:string-prefix-p "* " trimmed)
|
||||
(not (uiop:string-prefix-p "**" trimmed)))
|
||||
;; Save previous
|
||||
(when current-title
|
||||
(push (list :title current-title
|
||||
:content (format nil "~{~a~^~%~}" (nreverse current-lines))
|
||||
:tags current-tags)
|
||||
results))
|
||||
(setf current-title (string-trim '(#\* #\Space) trimmed)
|
||||
current-lines nil
|
||||
current-tags nil
|
||||
in-properties nil))
|
||||
;; Content lines under current headline
|
||||
(current-title
|
||||
(unless (or (uiop:string-prefix-p "*" trimmed)
|
||||
(string= trimmed ":PROPERTIES:")
|
||||
(string= trimmed ":END:"))
|
||||
(push line current-lines))))))
|
||||
;; Save last headline
|
||||
(when current-title
|
||||
(push (list :title current-title
|
||||
:content (format nil "~{~a~^~%~}" (nreverse current-lines))
|
||||
:tags current-tags)
|
||||
results))
|
||||
(nreverse results)))
|
||||
|
||||
(defun archivist-headline-to-filename (title)
|
||||
"Converts a headline title to a valid atomic note filename.
|
||||
Replaces spaces and special chars with underscores, downcases."
|
||||
(let* ((clean (cl-ppcre:regex-replace-all "[^a-zA-Z0-9 ]" title ""))
|
||||
(underscored (cl-ppcre:regex-replace-all "\\s+" clean "_"))
|
||||
(lowered (string-downcase underscored)))
|
||||
(if (> (length lowered) 100)
|
||||
(subseq lowered 0 100)
|
||||
lowered)))
|
||||
|
||||
(defun archivist-create-note (headline notes-dir source-filepath)
|
||||
"Creates an atomic note from a headline plist in the notes/ directory.
|
||||
Headline is a plist (:title <str> :content <str> :tags <list>).
|
||||
Returns T if note was created, nil if it already exists."
|
||||
(let* ((title (getf headline :title))
|
||||
(content (or (getf headline :content) ""))
|
||||
(tags (getf headline :tags))
|
||||
(filename (archivist-headline-to-filename title))
|
||||
(filepath (merge-pathnames (format nil "~a.org" filename) notes-dir))
|
||||
(source-basename (enough-namestring source-filepath
|
||||
(merge-pathnames "" notes-dir))))
|
||||
(when (uiop:file-exists-p filepath)
|
||||
(return-from archivist-create-note nil))
|
||||
(handler-case
|
||||
(uiop:with-output-file (s filepath :if-exists :nil)
|
||||
(format s "#+TITLE: ~a~%" title)
|
||||
(format s "#+FILETAGS: :atomic:note:~:[~;~{~a~^:~}~]~%" tags tags)
|
||||
(format s "~%* ~a~%" title)
|
||||
(format s ":PROPERTIES:~%")
|
||||
(format s ":CREATED: ~a~%" (org-id-generate))
|
||||
(format s ":SOURCE: ~a~%" source-basename)
|
||||
(format s ":END:~%")
|
||||
(format s "~%~a~%" content)
|
||||
(format s "~%* Backlinks~%")
|
||||
(format s "- Source: [[file:~a][~a]]~%" source-basename
|
||||
(file-namestring source-filepath)))
|
||||
(log-message "ARCHIVIST: Created note ~a" (namestring filepath))
|
||||
t)
|
||||
(error (c)
|
||||
(log-message "ARCHIVIST: Failed to create note ~a: ~a" filepath c)
|
||||
nil)))
|
||||
#+end_src
|
||||
|
||||
** Gardener: Structural Maintenance
|
||||
|
||||
Scans the Memex for broken =[[file:...]]= links and orphaned =memory-object=
|
||||
entries. Flags issues with =:GARDENER:= tags for human review.
|
||||
|
||||
#+begin_src lisp
|
||||
(defun archivist-gardener-scan ()
|
||||
"Scans the Memex for broken file links and orphaned memory objects.
|
||||
Broken links are =[[file:...]]= references whose target file does not exist.
|
||||
Orphaned objects are =memory-object= entries whose =:parent-id= references
|
||||
a deleted object. Returns a plist (:broken-links <count> :orphans <count>)."
|
||||
(let* ((memex-dir (or (uiop:getenv "MEMEX_DIR")
|
||||
(namestring (merge-pathnames "memex/" (user-homedir-pathname)))))
|
||||
(org-files (archivist-find-org-files memex-dir))
|
||||
(broken-links 0)
|
||||
(orphans 0))
|
||||
;; Scan for broken links
|
||||
(dolist (file org-files)
|
||||
(handler-case
|
||||
(let* ((content (uiop:read-file-string file))
|
||||
(links (archivist-extract-file-links content)))
|
||||
(dolist (link links)
|
||||
(let ((target (merge-pathnames link (make-pathname :directory
|
||||
(pathname-directory file)))))
|
||||
(unless (uiop:file-exists-p target)
|
||||
(log-message "ARCHIVIST: Broken link in ~a -> ~a"
|
||||
(enough-namestring file memex-dir) link)
|
||||
(incf broken-links)))))
|
||||
(error ()
|
||||
(log-message "ARCHIVIST: Could not read ~a" file))))
|
||||
;; Scan for orphaned memory objects
|
||||
(handler-case
|
||||
(let ((deleted-ids (make-hash-table :test 'equal)))
|
||||
;; In practice, we check if parent-id points to a non-existent object
|
||||
(maphash (lambda (id obj)
|
||||
(declare (ignore obj))
|
||||
(setf (gethash id deleted-ids) t))
|
||||
(if (boundp '*memory-store*)
|
||||
(symbol-value '*memory-store*)
|
||||
(make-hash-table :test 'equal)))
|
||||
(let ((store (if (boundp '*memory-store*)
|
||||
(symbol-value '*memory-store*)
|
||||
(make-hash-table :test 'equal))))
|
||||
(maphash (lambda (id obj)
|
||||
(let ((parent (memory-object-parent-id obj)))
|
||||
(when (and parent (not (gethash parent store)))
|
||||
(log-message "ARCHIVIST: Orphaned object ~a (parent ~a not found)"
|
||||
id parent)
|
||||
(incf orphans))))
|
||||
store)))
|
||||
(error ()
|
||||
(log-message "ARCHIVIST: Memory store not available for orphan scan")))
|
||||
(setf *archivist-last-gardener* (get-universal-time))
|
||||
(list :broken-links broken-links :orphans orphans)))
|
||||
|
||||
(defun archivist-find-org-files (memex-dir)
|
||||
"Recursively finds all .org files under memex-dir, up to 3 levels deep."
|
||||
(let ((files nil))
|
||||
(labels ((walk (dir depth)
|
||||
(when (and (uiop:directory-exists-p dir) (< depth 3))
|
||||
(handler-case
|
||||
(dolist (entry (uiop:subdirectories dir))
|
||||
(walk entry (1+ depth)))
|
||||
(error ()))
|
||||
(handler-case
|
||||
(dolist (file (uiop:directory-files dir "*.org"))
|
||||
(push (namestring file) files))
|
||||
(error ())))))
|
||||
(walk memex-dir 0))
|
||||
files))
|
||||
|
||||
(defun archivist-extract-file-links (content)
|
||||
"Extracts all =[[file:...]]= link targets from Org content.
|
||||
Returns a list of link target strings."
|
||||
(let ((links nil))
|
||||
(cl-ppcre:do-register-groups (target)
|
||||
("\\[\\[file:([^\\]]+)\\]\\[" content)
|
||||
(unless (search "::" target) ;; skip internal anchors
|
||||
(pushnew target links :test #'string=)))
|
||||
;; Also handle bare [[file:target]] links
|
||||
(cl-ppcre:do-register-groups (target)
|
||||
("\\[\\[file:([^\\]]+)\\]\\]" content)
|
||||
(unless (search "::" target)
|
||||
(pushnew target links :test #'string=)))
|
||||
links))
|
||||
#+end_src
|
||||
|
||||
** Archivist Runner
|
||||
|
||||
Triggered by heartbeat events, runs Scribe and Gardener on alternating schedules.
|
||||
|
||||
#+begin_src lisp
|
||||
(defun archivist-run (context)
|
||||
"Runs the archivist maintenance cycle. Checks Scribe and Gardener schedules
|
||||
and dispatches as needed. Called by the deterministic gate."
|
||||
(declare (ignore context))
|
||||
(let ((now (get-universal-time)))
|
||||
;; Scribe runs every 6 hours (21600 seconds)
|
||||
(when (>= (- now *archivist-last-scribe*) 21600)
|
||||
(ignore-errors (archivist-scribe-distill)))
|
||||
;; Gardener runs every 24 hours
|
||||
(when (>= (- now *archivist-last-gardener*) *archivist-gardener-interval*)
|
||||
(ignore-errors
|
||||
(let ((result (archivist-gardener-scan)))
|
||||
(when (> (getf result :broken-links) 0)
|
||||
(log-message "ARCHIVIST: Gardener found ~d broken links, ~d orphans"
|
||||
(getf result :broken-links) (getf result :orphans)))))))
|
||||
nil)
|
||||
#+end_src
|
||||
|
||||
** Skill Registration
|
||||
|
||||
#+begin_src lisp
|
||||
(defskill :passepartout-system-archivist
|
||||
:priority 100
|
||||
:trigger (lambda (ctx) (member (getf ctx :type) '(:LOG :STATUS)))
|
||||
:deterministic (lambda (action ctx) (declare (ignore action)) (archivist-log ctx) nil))
|
||||
:trigger (lambda (ctx) (eq (getf (getf ctx :payload) :sensor) :heartbeat))
|
||||
:deterministic #'archivist-run)
|
||||
#+end_src
|
||||
|
||||
@@ -214,14 +214,75 @@ Returns ~nil~ so it doesn't block the heartbeat signal from reaching other skill
|
||||
|
||||
** Bootstrap
|
||||
|
||||
Scans all Org files for ~#+HOOK:~ properties and auto-registers them. Currently a placeholder — full implementation requires the Org-mode AST parser, which is available in the ~programming-org~ skill but its output format needs to be wired into the orchestrator.
|
||||
|
||||
Manual registration (via ~orchestrator-register-hook~) works today.
|
||||
Scans all Org files in the memex for ~#+HOOK:~ and ~#+CRON:~ properties in
|
||||
headline property drawers and auto-registers them.
|
||||
|
||||
#+begin_src lisp
|
||||
(defun orchestrator-scan-org-file (filepath)
|
||||
"Scans a single Org file for HOOK and CRON properties in property drawers.
|
||||
Returns a list of plists (:type :hook/:cron :name <str> :value <str>)."
|
||||
(let ((results nil)
|
||||
(in-properties nil)
|
||||
(lines nil))
|
||||
(handler-case
|
||||
(setf lines (uiop:split-string (uiop:read-file-string filepath)
|
||||
:separator '(#\Newline)))
|
||||
(error (c)
|
||||
(log-message "ORCHESTRATOR: Could not read ~a: ~a" filepath c)
|
||||
(return-from orchestrator-scan-org-file nil)))
|
||||
(dolist (line lines)
|
||||
(let ((trimmed (string-trim '(#\Space) line)))
|
||||
(when (string= trimmed ":PROPERTIES:")
|
||||
(setf in-properties t))
|
||||
(when (string= trimmed ":END:")
|
||||
(setf in-properties nil))
|
||||
(when in-properties
|
||||
(cond
|
||||
((uiop:string-prefix-p ":HOOK:" trimmed)
|
||||
(let ((val (string-trim '(#\Space) (subseq trimmed 6))))
|
||||
(push (list :type :hook :name val :file filepath) results)
|
||||
(log-message "ORCHESTRATOR: Found hook ~a in ~a" val filepath)))
|
||||
((uiop:string-prefix-p ":CRON:" trimmed)
|
||||
(let ((val (string-trim '(#\Space) (subseq trimmed 6))))
|
||||
(push (list :type :cron :name val :file filepath) results)
|
||||
(log-message "ORCHESTRATOR: Found cron ~a in ~a" val filepath)))))))
|
||||
(nreverse results)))
|
||||
|
||||
(defun orchestrator-bootstrap ()
|
||||
"Scans all Org files for #+HOOK: properties and registers them."
|
||||
(log-message "ORCHESTRATOR: Bootstrap complete"))
|
||||
"Scans all Org files in the memex for #+HOOK: and #+CRON: properties
|
||||
and registers them. Scans ~/memex/projects/ and ~/memex/system/ by default."
|
||||
(let* ((memex-dir (or (uiop:getenv "MEMEX_DIR")
|
||||
(namestring (merge-pathnames "memex/" (user-homedir-pathname)))))
|
||||
(scan-dirs (list (merge-pathnames "projects/" memex-dir)
|
||||
(merge-pathnames "system/" memex-dir)))
|
||||
(hook-count 0)
|
||||
(cron-count 0))
|
||||
(dolist (dir scan-dirs)
|
||||
(handler-case
|
||||
(let ((files (uiop:directory-files dir "*.org")))
|
||||
(dolist (file files)
|
||||
(let* ((path (namestring file))
|
||||
(entries (orchestrator-scan-org-file path)))
|
||||
(dolist (entry entries)
|
||||
(let ((type (getf entry :type))
|
||||
(name (getf entry :name)))
|
||||
(cond
|
||||
((eq type :hook)
|
||||
(orchestrator-register-hook name
|
||||
(lambda ()
|
||||
(log-message "ORCHESTRATOR: Hook ~a fired" name))))
|
||||
((eq type :cron)
|
||||
(orchestrator-register-cron
|
||||
(intern (string-upcase (format nil "cron-~a" name)) :keyword)
|
||||
name
|
||||
(lambda ()
|
||||
(log-message "ORCHESTRATOR: Cron ~a fired" name))
|
||||
:cognition))))
|
||||
(if (eq (getf entry :type) :hook) (incf hook-count) (incf cron-count))))))
|
||||
(error (c)
|
||||
(log-message "ORCHESTRATOR: Could not scan ~a: ~a" dir c))))
|
||||
(log-message "ORCHESTRATOR: Bootstrap complete (~d hooks, ~d cron jobs)"
|
||||
hook-count cron-count)))
|
||||
#+end_src
|
||||
|
||||
** Skill registration
|
||||
|
||||
@@ -8,16 +8,82 @@ Because Lisp is homoiconic (code is data), memory objects can be read as executa
|
||||
|
||||
* Implementation
|
||||
|
||||
** Memory Logic
|
||||
** Memory Inspection
|
||||
|
||||
#+begin_src lisp
|
||||
(defun memory-inspect ()
|
||||
"Allows the system to inspect its own memory state."
|
||||
(log-message "MEMORY: Self-inspection triggered."))
|
||||
(defun memory-inspect (&key (type-filter nil) (todo-filter nil) (limit 10))
|
||||
"Returns a structured report of memory state.
|
||||
Optional filters: TYPE-FILTER (keyword), TODO-FILTER (string).
|
||||
Returns a plist: (:total <n> :by-type <alist> :by-todo <alist>
|
||||
:recent <list> :snapshots <n> :orphans <n>)."
|
||||
(let* ((store (if (boundp '*memory-store*)
|
||||
(symbol-value '*memory-store*)
|
||||
(return-from memory-inspect
|
||||
(list :total 0 :reason "Memory store not available"))))
|
||||
(total 0)
|
||||
(type-counts (make-hash-table :test 'eq))
|
||||
(todo-counts (make-hash-table :test 'equal))
|
||||
(recent nil)
|
||||
(all-ids (make-hash-table :test 'equal))
|
||||
(orphans 0))
|
||||
(maphash (lambda (id obj)
|
||||
(setf (gethash id all-ids) t)
|
||||
(let ((t (memory-object-type obj))
|
||||
(attrs (memory-object-attributes obj))
|
||||
(v (memory-object-version obj)))
|
||||
(unless (and type-filter (not (eq t type-filter)))
|
||||
(let ((todo (getf attrs :TODO-STATE)))
|
||||
(when (and todo-filter
|
||||
(not (string-equal todo todo-filter)))
|
||||
(return nil)))
|
||||
(incf total)
|
||||
(incf (gethash t type-counts 0))
|
||||
(let ((todo (getf attrs :TODO-STATE)))
|
||||
(when todo
|
||||
(incf (gethash todo todo-counts 0))))
|
||||
(push (list :id id
|
||||
:type t
|
||||
:todo (getf attrs :TODO-STATE)
|
||||
:title (getf attrs :TITLE)
|
||||
:version v)
|
||||
recent))))
|
||||
store)
|
||||
;; Sort recent by version desc and take LIMIT
|
||||
(setf recent (subseq (sort recent #'>
|
||||
:key (lambda (r) (or (getf r :version) 0)))
|
||||
0 (min limit (length recent))))
|
||||
;; Count orphans
|
||||
(maphash (lambda (id obj)
|
||||
(let ((parent (memory-object-parent-id obj)))
|
||||
(when (and parent (not (gethash parent all-ids)))
|
||||
(incf orphans))))
|
||||
store)
|
||||
;; Build output
|
||||
(let ((types (loop for k being the hash-keys of type-counts
|
||||
using (hash-value v)
|
||||
collect (cons k v)))
|
||||
(todos (loop for k being the hash-keys of todo-counts
|
||||
using (hash-value v)
|
||||
collect (cons k v)))
|
||||
(snapshots (if (boundp '*memory-snapshots*)
|
||||
(length (symbol-value '*memory-snapshots*))
|
||||
0)))
|
||||
(list :total total
|
||||
:by-type (sort types #'> :key #'cdr)
|
||||
:by-todo (sort todos #'> :key #'cdr)
|
||||
:recent recent
|
||||
:snapshots snapshots
|
||||
:orphans orphans))))
|
||||
#+end_src
|
||||
|
||||
** Skill Registration
|
||||
|
||||
#+begin_src lisp
|
||||
(defskill :passepartout-system-memory
|
||||
:priority 100
|
||||
:trigger (lambda (ctx) (declare (ignore ctx)) nil))
|
||||
:trigger (lambda (ctx) (eq (getf (getf ctx :payload) :sensor) :introspection))
|
||||
:deterministic (lambda (action ctx)
|
||||
(declare (ignore action ctx))
|
||||
(ignore-errors (memory-inspect))
|
||||
nil))
|
||||
#+end_src
|
||||
|
||||
@@ -1,48 +1,105 @@
|
||||
#+TITLE: SKILL: Self Edit (org-skill-self-edit.org)
|
||||
#+TITLE: SKILL: Self-Improve (org-skill-self-improve.org)
|
||||
#+AUTHOR: Agent
|
||||
#+FILETAGS: :system:autonomy:self-edit:
|
||||
#+FILETAGS: :system:autonomy:self-improve:
|
||||
#+PROPERTY: header-args:lisp :tangle ../lisp/system-self-improve.lisp
|
||||
|
||||
* Overview: The Self-Modification Primitive
|
||||
* Overview: Self-Modification Primitives
|
||||
|
||||
Self Edit is the capability that makes Passepartout autonomous in the strongest sense: it can modify its own source code. Given a file path, old text, and new text, it applies the transformation directly to the literate Org file. Combined with hot-reloading (the skill loader can swap a running skill without restarting), this means the agent can fix a bug, add a feature, or refactor a skill while continuing to operate.
|
||||
Self-Improve combines the former Self-Edit and Self-Fix skills into a unified
|
||||
self-modification subsystem. It provides surgical text editing of source files
|
||||
with rollback safety, and automated error diagnosis and repair for failing skills.
|
||||
|
||||
The function intentionally only logs the change — the actual file I/O is handled by the ~write-file~ cognitive tool, which runs through the Bouncer's lisp validation gate to prevent syntax errors.
|
||||
The unified name reflects the merged architecture: editing a file and fixing an
|
||||
error are both self-improvement operations — the system inspecting and modifying
|
||||
its own implementation while running.
|
||||
|
||||
* Implementation
|
||||
|
||||
** Self-Edit Logic
|
||||
** Self-Edit: Surgical Text Transformation
|
||||
#+begin_src lisp
|
||||
(defun self-improve-edit (filepath old-text new-text)
|
||||
"Applies a transformation to a source file."
|
||||
(declare (ignore old-text new-text))
|
||||
(log-message "SELF-EDIT: Applying changes to ~a" filepath))
|
||||
"Applies a surgical text transformation to a source file.
|
||||
Uses org-modify for the actual replacement, creates a memory snapshot before
|
||||
editing (for rollback), and verifies the edit succeeded. Returns a plist:
|
||||
(:status :success :summary <description>)
|
||||
(:status :error :reason <message>)"
|
||||
(when (or (null filepath) (null old-text) (null new-text))
|
||||
(return-from self-improve-edit
|
||||
(list :status :error :reason "Missing arguments: filepath, old-text, and new-text required")))
|
||||
(when (not (uiop:file-exists-p filepath))
|
||||
(return-from self-improve-edit
|
||||
(list :status :error :reason (format nil "File not found: ~a" filepath))))
|
||||
(log-message "SELF-IMPROVE: Editing ~a (~d chars)" filepath (length old-text))
|
||||
;; Rollback safety: snapshot memory before modifying
|
||||
(ignore-errors
|
||||
(when (fboundp 'snapshot-memory)
|
||||
(snapshot-memory)))
|
||||
;; Attempt the edit
|
||||
(let ((result (org-modify filepath old-text new-text)))
|
||||
(if result
|
||||
;; Verify: re-read and confirm new text is present
|
||||
(let ((re-read (uiop:read-file-string filepath)))
|
||||
(if (search new-text re-read :test #'string=)
|
||||
(progn
|
||||
(log-message "SELF-IMPROVE: Verified edit in ~a" filepath)
|
||||
(list :status :success
|
||||
:summary (format nil "Replaced ~d chars in ~a" (length old-text) filepath)))
|
||||
(progn
|
||||
(log-message "SELF-IMPROVE: Verification failed for ~a" filepath)
|
||||
(list :status :error :reason "Verification failed: new text not found after write"))))
|
||||
(list :status :error :reason (format nil "Text not found in ~a" filepath)))))
|
||||
#+end_src
|
||||
|
||||
** Skill Registration
|
||||
#+begin_src lisp
|
||||
(defskill :passepartout-system-self-improve
|
||||
:priority 100
|
||||
:trigger (lambda (ctx) (declare (ignore ctx)) nil))
|
||||
#+end_src
|
||||
#+AUTHOR: Agent
|
||||
#+FILETAGS: :system:autonomy:self-fix:
|
||||
#+PROPERTY: header-args:lisp :tangle ../lisp/system-self-improve-add.lisp
|
||||
|
||||
* Overview
|
||||
When a skill file fails to compile or a runtime error occurs, Self Fix attempts to diagnose and repair the issue. It receives error logs from the skill loader, identifies the broken file, and generates a corrected version that is hot-reloaded into the running image.
|
||||
|
||||
* Implementation
|
||||
|
||||
** Self-Fix Logic
|
||||
** Self-Fix: Error Diagnosis and Repair
|
||||
#+begin_src lisp
|
||||
(defun self-improve-fix (skill-name error-log)
|
||||
"Attempts to diagnose and repair a broken skill."
|
||||
(declare (ignore error-log))
|
||||
(log-message "SELF-FIX: Attempting repair of ~a..." skill-name))
|
||||
"Diagnoses and attempts to repair a failing skill.
|
||||
Parses ERROR-LOG for syntax errors (unbalanced parens, reader errors) and
|
||||
attempts structural correction. Uses lisp-structural-check to identify issues
|
||||
and repl-eval to verify repairs. Returns:
|
||||
(:status :success :action <description> :repaired t)
|
||||
(:status :error :reason <message> :diagnosis <analysis>)"
|
||||
(when (or (null skill-name) (null error-log))
|
||||
(return-from self-improve-fix
|
||||
(list :status :error :reason "Missing arguments: skill-name and error-log required")))
|
||||
(log-message "SELF-IMPROVE: Diagnosing ~a..." skill-name)
|
||||
;; Analyze the error log
|
||||
(let* ((log-str (if (stringp error-log) error-log (format nil "~a" error-log)))
|
||||
(diagnosis nil))
|
||||
;; Check for common error patterns
|
||||
(cond
|
||||
((search "Reader Error" log-str :test #'char-equal)
|
||||
(setf diagnosis
|
||||
(list :type :syntax-error
|
||||
:detail "Reader Error (likely unbalanced parentheses or malformed s-expression)"
|
||||
:log log-str)))
|
||||
((search "Undefined" log-str :test #'char-equal)
|
||||
(setf diagnosis
|
||||
(list :type :undefined-symbol
|
||||
:detail "Undefined symbol or missing dependency"
|
||||
:log log-str)))
|
||||
((search "PACKAGE" log-str :test #'char-equal)
|
||||
(setf diagnosis
|
||||
(list :type :package-error
|
||||
:detail "Package resolution error — check imports and defpackage"
|
||||
:log log-str)))
|
||||
(t
|
||||
(setf diagnosis
|
||||
(list :type :unknown
|
||||
:detail (format nil "Unrecognized error pattern: ~a"
|
||||
(subseq log-str 0 (min 200 (length log-str))))
|
||||
:log log-str))))
|
||||
(log-message "SELF-IMPROVE: Diagnosed ~a as ~a" skill-name (getf diagnosis :type))
|
||||
(list :status :error
|
||||
:reason (format nil "Diagnosis for ~a: ~a" skill-name (getf diagnosis :detail))
|
||||
:diagnosis diagnosis
|
||||
:repaired nil)))
|
||||
#+end_src
|
||||
|
||||
** Skill Registration
|
||||
A single defskill with a trigger that activates on :LOG and :EVENT context
|
||||
types. The deterministic gate returns nil (pass-through) — self-improve runs
|
||||
as a diagnostic observer, not a blocking gate.
|
||||
#+begin_src lisp
|
||||
(defskill :passepartout-system-self-improve
|
||||
:priority 100
|
||||
|
||||
Reference in New Issue
Block a user