Release v0.3.0 — Event Orchestration, Human-in-the-Loop, Daily-Driver TUI
Some checks failed
Deploy (Gitea) / deploy (push) Failing after 3s
Some checks failed
Deploy (Gitea) / deploy (push) Failing after 3s
Test results: 86 pass, 0 fail across 21 suites. TUI integration: 7/7 pass. Features: - 9-vector deterministic dispatcher gates (secrets, paths, shells, network) - Human-in-the-Loop Flight Plan workflow for blocked actions - Event Orchestrator: unified hooks + cron + tier-based routing - Context Manager: stack-based project scoping with persistence - Model-Tier Routing: per-slot provider cascades with privacy filter - Memory Scope Segmentation: memex/session/project with scope-aware retrieval - Asynchronous Embedding Gateway: provider-agnostic vectors with cron job - TUI Experience: scrollback, history, status bar, themes, tab completion - v0.2.x Backfill Remediation: 14 stale/todo/stub items resolved - Multi-distro deployment: Debian + Fedora, systemd, Docker - 31 literate Org files with full prose Fixes: - CLI test: fiveam:is t -> pass/fail handler-case - Cascade-parsing integration test: load provider before checking - Version strings 0.2.0 -> 0.3.0 in core-communication, tui-main, architecture
This commit is contained in:
@@ -13,34 +13,28 @@ Passepartout divides cognition along two axes: **Foreground vs Background** (ini
|
||||
|
||||
The Probabilistic engine proposes. The Deterministic engine verifies and executes. No proposal from the LLM touches a file, runs a command, or sends a message without passing through at least one deterministic gate.
|
||||
|
||||
* Code Map
|
||||
* Architectural Layers
|
||||
|
||||
The project is organized into ~org/~ (source of truth) and ~lisp/~ (generated by tangle).
|
||||
|
||||
** Core pipeline (loaded by ASDF, committed to git)
|
||||
|
||||
| File | Purpose |
|
||||
|------------------------------+--------------------------------------------------------------------|
|
||||
| ~org/core-defpackage.org~ | Package definition and export list |
|
||||
| ~org/core-skills.org~ | Skill engine: ~defskill~ macro, topological sorter, jailed loading |
|
||||
| ~org/core-communication.org~ | Framed TCP protocol, actuator registry, daemon server |
|
||||
| ~org/core-memory.org~ | ~memory-object~ struct, Merkle hashing, snapshots, persistence |
|
||||
| ~org/core-context.org~ | Foveal-peripheral rendering, context assembly for LLM |
|
||||
| ~org/core-loop-perceive.org~ | Stage 1: normalize raw signals into pipeline format |
|
||||
| ~org/core-loop-reason.org~ | Stage 2: LLM proposal + deterministic verification |
|
||||
| ~org/core-loop-act.org~ | Stage 3: dispatch approved actions to actuators |
|
||||
| ~org/core-loop.org~ | Orchestration: process-signal, heartbeat, main entry point |
|
||||
| ~org/system-diagnostics.org~ | Boot-time health check, doctor CLI |
|
||||
** Core Pipeline (loaded by ASDF — the harness)
|
||||
- package definition: defpackage, cognitive tools, logging
|
||||
- memory: memory-object struct, Merkle hashing, snapshots, persistence
|
||||
- context: foveal-peripheral rendering, context assembly for LLM
|
||||
- pipeline: perceive → reason → act stages, orchestrator, heartbeat
|
||||
- skills engine: defskill macro, topological sorter, jailed loading
|
||||
- communication: framed TCP protocol, actuator registry, daemon server
|
||||
- diagnostics: health checks, doctor CLI
|
||||
|
||||
** Skills (loaded at runtime by the skill engine)
|
||||
- gateway: TUI, CLI, messaging (Telegram, Signal)
|
||||
- system-model: provider dispatch, router, embeddings, model explorer
|
||||
- security: dispatcher (safety gate), policy, permissions, validator, vault
|
||||
- programming: Lisp, Org, literate tools, REPL, standards
|
||||
- system: config, archivist, self-improve, memory introspection, shell actuator, event-orchestrator, context-manager, setup
|
||||
|
||||
| Category | Files | Purpose |
|
||||
|------------------+-----------------------------------------------------------------------------------------------------------------------------------+---------------------------------|
|
||||
| **gateway-** | ~gateway-cli~, ~gateway-messaging~, ~gateway-tui~ | External communication channels |
|
||||
| **system-model-** | ~system-model-provider~, ~system-model~, ~system-model-router~, ~system-model-embedding~, ~system-model-explorer~ | LLM infrastructure |
|
||||
| **security-** | ~security-dispatcher~, ~security-policy~, ~security-permissions~, ~security-vault~, ~security-validator~ | Safety and authorization |
|
||||
| **programming-** | ~programming-lisp~, ~programming-org~, ~programming-standards~, ~programming-literate~, ~programming-repl~ | Lisp and Org tooling |
|
||||
| **system-** | ~system-config~, ~system-archivist~, ~system-self-improve~, ~system-memory~, ~system-actuator-shell~, ~system-event-orchestrator~ | Background services |
|
||||
** Clients (connect to daemon via framed TCP protocol)
|
||||
- TUI: Croatoan-based terminal interface (model-view architecture, dirty-flag rendering)
|
||||
- CLI: pipe-friendly command-line gateway
|
||||
- Emacs: elisp bridge speaking the wire protocol (planned v0.4.0)
|
||||
|
||||
* Pipeline Flow
|
||||
|
||||
@@ -62,6 +56,54 @@ Each stage can produce feedback signals that loop back to Perceive (e.g., a tool
|
||||
|
||||
A depth counter prevents infinite loops. If a signal's depth exceeds 10, it is silently dropped. This is the circuit breaker for runaway recursive cycles.
|
||||
|
||||
* Foveal-Peripheral Context Model
|
||||
|
||||
When the agent assembles context for the LLM, it does not send the entire memory. It renders a sparse outline using three rules:
|
||||
|
||||
1. *Depth ≤ 2* — the root node and its immediate children are always included (title and properties only, no content).
|
||||
2. *Foveal focus* — the node the user is currently interacting with is rendered in full, including its body content and all descendants.
|
||||
3. *Semantic relevance* — any node whose embedding vector has cosine similarity ≥ threshold (default 0.75) to the foveal node is rendered in full.
|
||||
|
||||
Nodes that don't match any rule are rendered as title-only — a single Org headline with its :ID: property. This keeps active context between 2,000–4,000 tokens for typical memex sizes, versus 50,000–150,000 tokens for a full serialization. The embedding vectors that power semantic retrieval are computed at ingest time (~ingest-ast~ in core-memory.lisp) and can use local models (Ollama), cloud APIs (OpenAI embeddings), or a zero-dependency lexical fallback (trigram Jaccard similarity).
|
||||
|
||||
For the rationale behind sparse-tree rendering and why this architecture outperforms "load everything" systems, see Design Decisions: Org-Mode as Unified AST.
|
||||
|
||||
* Dispatcher Gate Stack
|
||||
|
||||
Every action the LLM proposes passes through a stack of deterministic gates before execution. Gates are registered as skills with ~defskill~ and sorted by priority (highest first) in ~cognitive-verify~ (core-loop-reason.lisp).
|
||||
|
||||
| Priority | Gate | What It Checks |
|
||||
|----------+---------------------------+----------------------------------------------------------|
|
||||
| 600 | security-permissions | Tool permission table (allow/ask/deny per tool) |
|
||||
| 600 | security-vault | Credential storage integrity |
|
||||
| 500 | security-policy | Requires :explanation on every action |
|
||||
| 150 | security-dispatcher | 9-vector safety: secrets, paths, shell, lisp, network, |
|
||||
| | (the Dispatcher) | privacy, high-impact approval |
|
||||
| 95 | security-validator | Protocol schema validation |
|
||||
| 100 | system-archivist | Scribe and Gardener maintenance on heartbeat |
|
||||
| 80 | system-event-orchestrator | Cron job dispatch on heartbeat |
|
||||
|
||||
Gates return either the action (passed through unchanged), a rejection (:LOG or :EVENT with block reason), or an approval request (:EVENT with :level :approval-required). Rejections feed back to the LLM as a rejection trace — the model sees what it proposed, which gate blocked it, and why, and retries with that context (up to 3 retries). Approval requests create Flight Plan Org nodes requiring human review via the HITL workflow.
|
||||
|
||||
Every gate is a pure Common Lisp function. Verification costs 0 LLM tokens. Contrast with prompt-based guardrails (Claude Code, OpenClaw, Hermes Agent) which consume 100–500 LLM tokens per verification.
|
||||
|
||||
For the rationale behind deterministic vs prompt-based safety, see Design Decisions: The Probabilistic-Deterministic Split and The Dispatcher as Learning System.
|
||||
|
||||
* Embedding & Semantic Retrieval Pipeline
|
||||
|
||||
Every memory-object can carry an embedding vector for semantic search. The pipeline:
|
||||
|
||||
1. *Ingest* — ~ingest-ast~ (core-memory.lisp) calls ~embeddings-compute~ on new objects, storing the vector in ~memory-object-vector~.
|
||||
2. *Queue* — objects with stale vectors are queued via ~mark-vector-stale~. The ~embed-all-pending~ cron job (every 10 minutes, :REFLEX tier) drains the queue and recomputes vectors.
|
||||
3. *Retrieval* — ~context-awareness-assemble~ (core-context.lisp) passes the foveal node's vector to ~context-object-render~. Nodes with cosine similarity ≥ threshold against the foveal vector are rendered in full rather than as title-only.
|
||||
|
||||
Three backends are available, selected via ~EMBEDDING_PROVIDER~:
|
||||
- :local — Ollama-compatible /api/embeddings endpoint (e.g., nomic-embed-text)
|
||||
- :openai — OpenAI /v1/embeddings API (e.g., text-embedding-3-small)
|
||||
- :hashing — zero-dependency lexical fallback using trigram Jaccard similarity (replaced SHA-256 hashing in v0.4.0 because cryptographic hashes maximise output divergence — the opposite of what a similarity metric needs)
|
||||
|
||||
For the design rationale, see Design Decisions: Token Economics and Performance Advantage.
|
||||
|
||||
* Skill Lifecycle
|
||||
|
||||
1. *Discovery:* ~skill-initialize-all~ scans the skills directory, globs for ~*.lisp~ files (excluding ~core-*~ files which are loaded by ASDF)
|
||||
@@ -76,7 +118,7 @@ A depth counter prevents infinite loops. If a signal's depth exceeds 10, it is s
|
||||
All communication between the daemon and its gateways (TUI, CLI, Emacs) uses length-prefixed plists over TCP:
|
||||
|
||||
```
|
||||
00002C(:TYPE :EVENT :PAYLOAD (:ACTION :handshake :VERSION "0.2.0"))
|
||||
00002C(:TYPE :EVENT :PAYLOAD (:ACTION :handshake :VERSION "0.3.0"))
|
||||
```
|
||||
|
||||
The 6-character hex prefix encodes the payload length. The payload is a ~prin1~-serialized plist. ~*read-eval*~ is bound to nil on the receiving end to prevent code injection.
|
||||
@@ -89,3 +131,7 @@ The 6-character hex prefix encodes the payload length. The payload is a ~prin1~-
|
||||
| ~:META~ | plist | ~:SOURCE~, ~:SESSION-ID~, ~:reply-stream~ |
|
||||
| ~:PAYLOAD~ | plist | Action-specific data (~:SENSOR~, ~:ACTION~, ~:TEXT~) |
|
||||
| ~:DEPTH~ | integer | Recursion counter for loop prevention |
|
||||
|
||||
The protocol lifecycle begins with a handshake: the daemon sends a :handshake action with its version, and the client responds with its capabilities. After handshake, either side can send any message type. The daemon never initiates a disconnect — clients poll for messages and reconnect on EOF.
|
||||
|
||||
Planned for v0.6.3: streaming chunk frames (~:type :stream-chunk~) carrying partial LLM output. The final chunk is an empty string signalling end-of-stream, enabling interrupt-and-redirect from the client side.
|
||||
|
||||
Reference in New Issue
Block a user