Files
passepartout/docs/ROADMAP.org
Amr Gharbeia ec882f87fb
Some checks failed
Deploy (Gitea) / deploy (push) Failing after 2s
docs: mark P0 and P1 remediation items as DONE
2026-05-03 10:44:25 -04:00

539 lines
18 KiB
Org Mode

#+TITLE: Passepartout Evolutionary Roadmap
#+STARTUP: content
#+FILETAGS: :docs:roadmap:
* The Evolutionary Roadmap
The roadmap is designed working backwards from SOTA parity (v1.0.0), guiding each version toward a fully autonomous, self-editing agent. Each version builds on the previous, with features designed to be implemented in pure Common Lisp + Org-mode.
The TODO states in each version's Tasks section are the authoritative task tracker. The feature tables describe what each version delivers.
** Non-Negotiable Identity
- Pure Common Lisp + Org-mode. No JSON. No YAML. No external databases.
- Single-address-space memory (Lisp hash tables in RAM — the agent IS the memory).
- "Thin harness, fat skills" — complexity lives at the edges, not the kernel.
- One agent composed of many skills. Concurrency via bordeaux-threads (shared memory).
- Plists everywhere — homoiconic communication between all components.
** Version Roadmap
*** v0.1.0: The Autonomous Foundation — RELEASED 2026-04-20
The secure, auditable Lisp kernel. All core infrastructure in place.
**** DONE Perceive-Reason-Act pipeline
:PROPERTIES:
:ID: id-06f10b9a-4054-4dea-a927-b0935fbdcd2f
:CREATED: [2026-03-22 Sun]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-04-20 Mon]
:END:
**** DONE Skills engine with jailed loading
:PROPERTIES:
:ID: id-dc83944f-3923-4142-b324-c317dacd6b0b
:CREATED: [2026-03-22 Sun]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-04-20 Mon]
:END:
**** DONE Policy skill (6 invariants)
:PROPERTIES:
:ID: id-929c84b7-d6ae-42b9-a8b5-d9df962db826
:CREATED: [2026-03-22 Sun]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-04-20 Mon]
:END:
**** DONE Memory (memory-object + Merkle hashing)
:PROPERTIES:
:ID: id-3a96b384-cacf-4da0-8faa-1647739feba9
:CREATED: [2026-03-22 Sun]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-04-20 Mon]
:END:
**** DONE Scribe + Gardener background workers
:PROPERTIES:
:ID: id-3f618a38-ec23-4034-ba3c-ef272e212e2b
:CREATED: [2026-03-22 Sun]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-04-20 Mon]
:END:
**** DONE LLM gateway (OpenRouter, Ollama)
:PROPERTIES:
:ID: id-f5d870e2-cbd2-4c00-a8d4-174ab4118afc
:CREATED: [2026-04-11 Sat]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-04-20 Mon]
:END:
**** DONE Shell actuator, Emacs bridge, credentials vault
:PROPERTIES:
:ID: id-7ca3167f-8353-4bb7-8b97-c039017716b0
:CREATED: [2026-04-11 Sat]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-04-20 Mon]
:END:
**** DONE FiveAM test suite
:PROPERTIES:
:ID: id-925d4180-764b-4219-8bdc-8e1849572da1
:CREATED: [2026-04-11 Sat]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-04-20 Mon]
:END:
*** v0.2.0: Interactive Refinement — RELEASED 2026-04-29
The "Brain" meets the "Machine." Standardization and professionalization of the user interface and environment.
**** DONE Professional TUI (Croatoan-based, styled, scrollable)
:PROPERTIES:
:ID: id-57cef382-fe14-42e6-aade-03e05e3e920b
:CREATED: [2026-04-28 Tue]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-04-29 Wed]
:END:
**** DONE Self-editing (error detection, surgical fix, hot-reload)
:PROPERTIES:
:ID: id-459b8275-9979-4d0f-8d61-a9af883930d4
:CREATED: [2026-04-23 Wed]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-04-29 Wed]
:END:
**** DONE Enhanced utilities (structural Lisp/Org manipulation + REPL)
:PROPERTIES:
:ID: id-23f37c0d-4e77-4dc3-ab43-52a5987eb426
:CREATED: [2026-04-23 Wed]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-04-29 Wed]
:END:
**** DONE Onboarding wizard (modular Lisp setup for LLM providers)
:PROPERTIES:
:ID: id-bd497de7-3533-4056-b89f-2c992d2ea28b
:CREATED: [2026-04-28 Tue]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-04-29 Wed]
:END:
**** DONE Memory rollback (snapshot and restore)
:PROPERTIES:
:ID: id-fd2fb6e3-03e7-4e22-b9e9-a7eecfd06718
:CREATED: [2026-04-12 Sun]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-04-29 Wed]
:END:
**** DONE Secret Exposure Gate, Shell Safety, Lisp Validation
:PROPERTIES:
:ID: id-aa53c128-195b-42d4-9838-2def59faf7cf
:CREATED: [2026-05-02 Sat]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-05-02 Sat]
:END:
**** DONE Multi-distro deployment (Debian+Fedora, systemd, Docker)
:PROPERTIES:
:ID: id-783df999-f7fe-45c8-896d-2fd07c604d64
:CREATED: [2026-05-02 Sat]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-05-02 Sat]
:END:
**** DONE Project rename to Passepartout (files, packages, env vars)
:PROPERTIES:
:ID: id-91724874-aa0d-4804-9220-8bc5551f1366
:CREATED: [2026-05-02 Sat]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-05-02 Sat]
:END:
**** DONE 31 org files with full literate prose
:PROPERTIES:
:ID: id-597b2a92-aac6-481a-b2c4-4f9842ced97c
:CREATED: [2026-05-02 Sat]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-05-02 Sat]
:END:
*** v0.3.0: Event Orchestration + HITL
Unified control plane and Human-in-the-Loop state management.
** Tasks
*** Remediation: Backfill v0.1.0/v0.2.0 Gaps
These features were marked DONE in prior versions but are stubs, no-ops, or
missing. They must be completed before v0.3.0 feature work proceeds.
**** DONE P0: Add vault-get-secret / vault-set-secret wrappers :backfill:
CLOSED: [2026-05-03 Sun 10:42]
:PROPERTIES:
:ID: id-vault-secret-wrappers
:CREATED: [2026-05-03 Sun]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
:END:
=vault-get-secret= and =vault-set-secret= are exported from =core-defpackage=
and called from =gateway-manager.org= (lines 36, 86, 180) but never defined.
=gateway-link= crashes at runtime. Add one-line wrappers in =security-vault.org=
that delegate to the existing =vault-get=/=vault-set= with ~:type :secret~.
**** DONE P0: system-archivist — Scribe + Gardener :backfill:
CLOSED: [2026-05-03 Sun 10:42]
:PROPERTIES:
:ID: id-archivist-distillation
:CREATED: [2026-05-03 Sun]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
:END:
Scribe: distill daily Org logs into atomic Zettelkasten notes with backlinks.
Gardener: scan for broken =[[file:]]= links and orphaned =memory-object= entries.
Wire both as cron jobs via =system-event-orchestrator=.
Depends on: orchestrator bootstrap (P1 item below).
**** DONE P0: system-self-improve — surgical edit + error fix :backfill:
CLOSED: [2026-05-03 Sun 10:42]
:PROPERTIES:
:ID: id-self-improve-real
:CREATED: [2026-05-03 Sun]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
:END:
= self-improve-edit=: =org-read-file= → text replace → =snapshot-memory= →
=org-write-file= → =literate-block-balance-check= → tangle → reload.
=self-improve-fix=: parse error log → =lisp-structural-check= →
=lisp-extract= → surgical repair → =repl-eval= verify.
Remove the dead first =defskill= registration (trigger nil, overwritten by second).
Depends on: =programming-org=, =programming-literate= (P0 items below).
**** DONE P0: programming-org — fix org-modify + org-ast-render :backfill:
CLOSED: [2026-05-03 Sun 10:42]
:PROPERTIES:
:ID: id-org-modify-render
:CREATED: [2026-05-03 Sun]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
:END:
=org-modify(filepath, id, changes)= ignores ~changes~ and only logs. Should locate
node by ID in file and apply changes to its content.
=org-ast-render(ast)= returns a hardcoded placeholder. Should convert plist AST
back to Org text.
**** DONE P0: programming-literate — fix both stubs :backfill:
CLOSED: [2026-05-03 Sun 10:42]
:PROPERTIES:
:ID: id-literate-real
:CREATED: [2026-05-03 Sun]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
:END:
=literate-block-balance-check=: verify all =#+begin_src lisp= blocks in an Org file
have balanced parentheses. Returns T if all balanced, error message otherwise.
=literate-tangle-sync-check=: verify =.lisp= file matches tangled output of =.org= file.
**** DONE P1: system-event-orchestrator — bootstrap implementation :backfill:
CLOSED: [2026-05-03 Sun 10:42]
:PROPERTIES:
:ID: id-orchestrator-bootstrap
:CREATED: [2026-05-03 Sun]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
:END:
=orchestrator-bootstrap= currently only logs. Should scan Org files for =#+HOOK:=
and =#+CRON:= properties and register them via the existing registries.
Prerequisite for archivist cron jobs.
**** DONE P1: system-memory — memory introspection :backfill:
CLOSED: [2026-05-03 Sun 10:42]
:PROPERTIES:
:ID: id-memory-inspect
:CREATED: [2026-05-03 Sun]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
:END:
=memory-inspect= only logs. Should return structured statistics: object count
by type, TODO state distribution, orphan count, snapshot list. Trigger on
=:INTROSPECTION= sensor type.
**** DONE P1: Path relic — skills/ → lisp/ in skill-initialize-all :backfill:
CLOSED: [2026-05-03 Sun 10:42]
:PROPERTIES:
:ID: id-path-relic
:CREATED: [2026-05-03 Sun]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-05-03 Sun 10:42]
:END:
=skill-initialize-all= and =context-skill-source= resolve against =skills/=
under =$PASSEPARTOUT_DATA_DIR=. Core and skills were merged into =lisp/=.
Update both functions to point at =lisp/=.
**** TODO P2: core-context — semantic retrieval (embeddings) :backfill:
:PROPERTIES:
:ID: id-embeddings
:CREATED: [2026-05-03 Sun]
:END:
=org-object-vector= is never populated; all similarities are 0.0. Generate
embeddings via Ollama =nomic-embed-text= at ingest time. Store in
=memory-object.vector=. Fallback: TF-IDF bag-of-words.
**** TODO P2: core-context — subtree-based skill source loading :backfill:
:PROPERTIES:
:ID: id-skill-subtree
:CREATED: [2026-05-03 Sun]
:END:
=context-skill-source= reads entire Org files. Add =context-skill-subtree=
for targeted retrieval of specific function docs or test blocks by heading name.
**** TODO P3: Variable name drift normalization (out of scope for now) :backfill:
:PROPERTIES:
:ID: id-name-normalization
:CREATED: [2026-05-03 Sun]
:END:
=*memory*= (context) vs =*memory-store*= (memory). =*skills-registry*= with
underscore (reason/context) vs =*skill-registry*= with hyphen (defpackage).
Normalization pass across all modules. Touches every file — do after P0-P2
are stable. Do not mix with functional changes.
*** DONE Project Renaming (Bouncer → Dispatcher)
:PROPERTIES:
:ID: id-9e779580-287b-b3d1-37b9-bcefd750bf9e
:CREATED: [2026-05-01 Fri 15:40]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-05-02 Sat 22:00]
:END:
The Dispatcher's role has evolved beyond security guard. It is the seed of the deterministic engine — it learns to execute procedures without invoking the neural net.
*** DONE Event Orchestrator (unified hooks+cron+routing)
:PROPERTIES:
:ID: id-d35aea3d-2e5f-4a12-a9b0-1c2d3e4f5a6b
:CREATED: [2026-05-02 Sat 14:00]
:END:
:LOGBOOK:
- State "DONE" from "TODO" [2026-05-02 Sat 22:36]
:END:
Unified control plane for hooks, cron, and complexity-based routing.
- *hook-registry* + *cron-registry* + tier classifier
- Hooks via ~#+HOOK:~ Org-mode properties
- Three complexity tiers: ~:REFLEX~ (no LLM), ~:COGNITION~ (light LLM), ~:REASONING~ (full LLM)
- Hooked into heartbeat for cron processing
- Rule-based tier classifier (overrideable via ~*tier-classifier*~)
*** TODO Context Manager (project scoping)
:PROPERTIES:
:ID: id-a10ed34e-9f37-4a15-b499-46672c00d951
:CREATED: [2026-05-02 Sat 23:00]
:END:
Stack-based context with ~push-context~ / ~pop-context~.
Path resolution relative to current context.
Memory scope: ~:scope~ property on memory-objects (memex/session/project).
Implement lazy-loading proxies for large-scale memory traversal.
*** TODO Model-Tier Routing (cost optimization)
Extend ~*model-selector-fn*~ for complexity-based routing.
- Heartbeats → smallest model
- User input → medium model
- Complex reasoning → large model
*** TODO Memory Scope Segmentation
Extend memory-object with ~:scope~ property.
- ~:memex~ (permanent knowledge), ~:session~ (ephemeral), ~:project~ (current work)
- Scope-aware retrieval in memory layer
*** TODO Asynchronous Embedding Gateway
Provider-agnostic vector generation (Ollama, llama.cpp, OpenAI).
Edits mark nodes as ~:vector :pending~; background worker batches and updates Merkle tree.
*** TODO TUI Experience (Daily Driver Quality)
The TUI is a standalone Croatoan app in ~org/gateway-tui.org~.
None of these changes require daemon modifications — the protocol between TUI and
daemon (port 9105, framed plists) is stable.
- P0: Chat scrollback (Page Up/Down) — ~2h
- P0: Input history (up/down arrows) — ~1h
- P1: Status bar (daemon, model, time) — ~3h
- P1: Message rendering (timestamps, colors, wrapping) — ~2h
- P2: Command palette (/help redesign) — ~4h
- P2: Multi-line input (Shift+Enter) — ~3h
- P3: Background activity indicator — ~2h
- P4: Tab completion for / commands — ~3h
- P4: Configurable theme — ~4h
*** TODO Human-in-the-Loop (HITL)
Continuation-based interaction. The agent can suspend its cognitive loop to ask for
permission or clarification and resume precisely where it left off. Builds on the
dispatcher's existing Flight Plan mechanism.
*** v0.4.0: Long-Horizon Planning + Git Workflows
Structured tracking, failure handling, and course correction for multi-step engineering work.
** Tasks
*** TODO Long-Horizon Planning (task tree DAG)
Decompose complex tasks into Org-mode headline trees.
Terminal states: ~:todo~~:next-action~~:in-progress~~:done~ / ~:blocked~ / ~:stuck~.
Parent summarises child results.
Branch pruning when paths fail.
*** TODO Git Steward (version control integration)
Status, diff, commit, push, branch operations.
Policy enforces commit-before-modify gate.
Log commits to memory.
*** TODO TDD Runner Integration
Run FiveAM tests on file save.
Inject ~:test-failure~ event on red.
Hook into self-fix for auto-repair proposals.
*** TODO Deep Emacs Integration
Full org-agenda awareness: navigate, clock time, refile, archive.
Uses org-element + org-id.
*** v0.5.0: Interactive Actuation & Environment Stewardship
Interactive terminal sessions and autonomous dependency management.
** Tasks
*** TODO Interactive PTY Actuator
Stream long-running process output to the context window (e.g., ~npm run dev~, REPLs).
Async interrupt control (Ctrl+C emulation).
*** TODO The Environment Steward
Autonomously detect missing dependencies ("Command not found").
Propose installation command and retry the failed action.
*** v0.6.0: Concurrency + Creator + GTD
The agent bootstraps itself and manages parallel workstreams.
** Tasks
*** TODO Skill Creator (autonomous skill generation)
LLM drafts complete skill org-file from natural language.
Mandatory: syntax validation → jail-load → test → register.
*** TODO Architect Agent (PRD → PROTOCOL)
Scan ~:STATUS: FROZEN~ PRDs. Generate Phase B PROTOCOL from Phase A.
*** TODO GTD Integration (project tracking)
Full GTD cycle: capture, clarify, organize, reflect, engage.
org-gtd v4.0 DAG (~:TRIGGER:~, ~:BLOCKER:~).
*** TODO Consensus Loop (multi-model agreement)
Run multiple providers for critical decisions.
Compare results, detect disagreements.
Confidence scoring.
*** TODO Web Research (Playwright browsing)
Headless Chromium via Python bridge.
Text extraction, screenshots, Gemini Web UI automation.
*** TODO Memex Management (PARA lifecycle)
Archive DONE tasks, suggest refiling.
Detect orphaned nodes.
PARA/Zettelkasten maintenance.
*** v0.7.0: Visual Grounding & MCP Bridge
Multimodal visual interaction and ecosystem-wide tool compatibility.
** Tasks
*** TODO Computer Use / Vision
Allow the agent to request host OS or browser screenshots.
Analyze UI and issue precise X/Y coordinate click/type commands via X11/Wayland bridge.
*** TODO MCP Gateway Bridge
Lisp-native client for the Model Context Protocol.
Connect Passepartout to external tools and data sources.
*** v0.8.0: The Evaluation Harness
Automated benchmarking to mathematically prove the agent's reasoning capabilities.
** Tasks
*** TODO SWE-Bench Harness
Automated pipeline that clones repositories and feeds GitHub issues.
Track multi-step resolution trajectory, run tests, and score success.
*** v1.0.0: SOTA Parity
Feature-complete agent competitive with commercial agents. All features from v0.2.0 through v0.8.0 combined, verified, and tested end-to-end.
| Area | Parity Target |
|------|--------------|
| Self-improvement | Claude Code self-debug |
| Planning | ULTRAPLAN equivalent |
| Tool ecosystem | 10+ cognitive tools |
| Context window | Semantic search + scope segmentation |
| Safety | 6 Policy invariants + formal verification |
| Multi-step tasks | Task trees with terminal states |
| Code editing | Full file read/write via org manipulation |
| Memory | Vector recall in memory-object |
| Emacs integration | Full org-mode control (exceeds Claude Code) |
| Autonomy | 100% local capable (exceeds Claude Code) |
*** v2.0.0: Lisp Machine Emergence
From Lisp-using agent to true Lisp machine. Agent IS the Emacs process.
- Lish: Lisp editor — Org-mode as IDE. Org-babel for interactive evaluation. Full REPL in TUI.
- Lish: Shell replacement — Lisp-based shell that speaks plists. Org-mode buffers as file system.
*** v3.0.0: Neurosymbolic Maturity
Deterministic planner takes the wheel. LLM relegated to semantic translation.
- Deterministic planner: Pure Lisp task scheduler. No LLM needed for scheduling.
- Self-correcting gates: Gates learn from false positives (user override patterns).
*** v4.0.0: AI Stack Internalized
The agent understands its own weights. No external inference.
- Llama.cpp in Lisp: FFI binding. No Python subprocess. Pure Common Lisp inference.
- Weights as sexps: Neural weights as Lisp data structures. Homoiconic model introspection.
*** v5.0.0: True Agency
World models, temporal reasoning, goal persistence across restarts.
- World models: Predictive models of user behavior, project dynamics, system state.
- Temporal reasoning: Scheduling, deadlines, elapsed duration awareness.
- Goal persistence: Goals survive restarts. Long-term projects in memory-objects.