docs: finalize architecture and philosophy with Mermaid diagrams and PSF mandates
This commit is contained in:
226
docs/README.org
226
docs/README.org
@@ -1,183 +1,133 @@
|
||||
#+TITLE: org-agent: The Neurosymbolic Kernel
|
||||
#+AUTHOR: User
|
||||
#+CREATED: [2026-03-17 Tue]
|
||||
#+UPDATED: [2026-03-24 Tue]
|
||||
#+UPDATED: [2026-03-31 Tue]
|
||||
|
||||
A hyper-minimalist, self-editing, proactive AI agent framework. `org-agent` acts as the "executive soul" of a personal OS, using Org-mode as its native memory and Common Lisp as its deterministic reasoning engine.
|
||||
|
||||
* The Philosophy
|
||||
* The Philosophy: The PSF Mandates
|
||||
|
||||
** Mandate 1: Strictly Org-mode and Common Lisp
|
||||
The system is built on a "No Legacy" policy. Markdown (.md) and JSON are strictly prohibited for internal system logic, planning, and memory. Org-mode is the native Abstract Syntax Tree (AST) for both human and machine, and Common Lisp (SBCL) is the deterministic reasoning engine.
|
||||
The system is governed by three non-negotiable mandates from the **Personal Software Foundry (PSF)** that ensure long-term sustainability and human-agent alignment:
|
||||
|
||||
** Mandate 2: Minimalist Core, Skill-Based Extension
|
||||
The `org-agent` kernel (the Daemon) MUST remain a minimalist microkernel. It handles only the cognitive loop, the persistent Object-Store, and the communication protocol. All business logic, LLM provider connectors, and task-management rules MUST be implemented as hot-reloadable **Skills** living in the user's Memex.
|
||||
** 1. Lisp Machine Sovereignty
|
||||
The agent is a living Lisp image (SBCL). Code is updated via late-binding, hot-reloading into the active image. Manual "restarts" are considered a failure of architectural integrity. The system is a "Living Organism" rather than a static script.
|
||||
|
||||
** Why Org-mode? (Homoiconic Memory)
|
||||
Most agent frameworks rely on a messy combination of Python scripts, JSON states, and Markdown prompts. This breaks the human-agent interface. JSON is for machines; Markdown is for humans.
|
||||
** 2. Homoiconic Memory (The Org Mandate)
|
||||
Every document, plan, PRD, and skill in the system MUST be written in **Org-mode**. Markdown (.md) and JSON/YAML/XML are strictly prohibited for internal system logic. Org-mode provides a rigorous, hierarchical AST for both machine and human, ensuring they share the exact same mental model.
|
||||
|
||||
*Org-mode is for both.* It provides a rigorous, hierarchical Abstract Syntax Tree (AST) that a machine can navigate deterministically, while remaining a perfectly ergonomic, human-readable text document. In this system, your notes, your tasks, your prompts, and your agent's code all live in the exact same format.
|
||||
** 3. Literate Programming (Knuth-Sovereign Principle)
|
||||
The PSF treats code not as instructions for a computer, but as a "work of literature" for humans that happens to be executable. All system logic is implemented as **Literate Org files**. Documentation (the "Why") and Implementation (the "How") are unified, ensuring the system's **Institutional Memory** is preserved.
|
||||
|
||||
** Why Common Lisp? (The Kernel vs. The Actuators)
|
||||
The `org-agent` kernel is built in Common Lisp to provide a persistent, high-performance background process (SBCL) that maintains a live, threaded Object Store in RAM. It performs heavy neurosymbolic reasoning asynchronously, decoupled from any single user interface.
|
||||
* The Architecture: The Neurosymbolic Cognitive Loop
|
||||
|
||||
This architecture treats all interfaces as external **Actuators** and **Sensors**:
|
||||
- **Editor Actuator (Emacs):** A sensor array that detects file changes and executes structural refactoring.
|
||||
- **Messaging Actuator (Signal/Telegram/Discord):** A delivery channel for proactive alerts and human-in-the-loop decisions.
|
||||
- **Web Actuator (Dashboard):** A visual telemetry interface for monitoring the live kernel state.
|
||||
The kernel operates on a deterministic pipeline (OODA) that bridges intuitive "Neural" thinking with rigorous "Symbolic" logic.
|
||||
|
||||
* The Architecture: The Cognitive Loop
|
||||
#+begin_src mermaid
|
||||
graph TD
|
||||
subgraph Sensors
|
||||
Emacs[Emacs Bridge]
|
||||
Cron[Cron Heartbeat]
|
||||
Signal[Signal Inbound]
|
||||
end
|
||||
|
||||
The core engine is agnostic to both business logic and communication channels. It routes data through a strict four-stage cognitive pipeline:
|
||||
subgraph Kernel [The Neurosymbolic Kernel]
|
||||
Perceive[Perceive: Update Object Store]
|
||||
Think[Think: System 1 Neural Proposal]
|
||||
Decide[Decide: System 2 Symbolic Gate]
|
||||
Act[Act: Dispatch to Actuators]
|
||||
end
|
||||
|
||||
1. **Perceive:** Sensors (Emacs, Webhooks, CRON) send updates over the Org-Agent Communication Protocol (OACP). The kernel updates its live Object Store.
|
||||
2. **Think (System 1):** The `neuro.lisp` module queries an LLM (e.g., Gemini, OpenAI, or local models) based on the context, asking for an intuitive, pattern-matched suggestion. It returns an *unverified* proposed action.
|
||||
3. **Decide (System 2):** The `symbolic.lisp` module is the absolute gatekeeper. It takes the LLM's proposal and runs it through strict Lisp constraints (e.g., "A parent task cannot be marked DONE if it has active TODO children"). If the logic fails, the LLM is overruled.
|
||||
4. **Act:** Verified commands are dispatched to the appropriate Actuators (refactoring a buffer, sending a Signal message, or updating a database).
|
||||
subgraph Actuators
|
||||
Refactor[Org Buffer Refactor]
|
||||
Shell[Shell/Sandbox Actuator]
|
||||
Notify[External Delivery]
|
||||
end
|
||||
|
||||
* Extensibility: The Org-Native Skill Standard
|
||||
Sensors --> Perceive
|
||||
Perceive --> Think
|
||||
Think --> Decide
|
||||
Decide --> Act
|
||||
Act --> Actuators
|
||||
|
||||
%% Feedback Loop
|
||||
Actuators -.-> Sensors
|
||||
#+end_src
|
||||
|
||||
To keep the core microkernel minimal, all capabilities (API connectors, GTD logic, Atomic Notes (Zettelkasten) memory management) are abstracted into **Skills**.
|
||||
- **System 1 (Neural):** Uses LLMs (Gemini, Ollama) for pattern matching and creative problem-solving. It is fast but unreliable.
|
||||
- **System 2 (Symbolic):** Uses Common Lisp and formal logic to verify LLM proposals against security invariants and structural rules. It is the absolute gatekeeper.
|
||||
|
||||
Adhering to the Lisp Machine Mandate (Code is Data), a skill is not a Python folder. A skill is a single `.org` file located in the Atomic Notes (Zettelkasten) directory.
|
||||
* Extensibility: The Skill Graph
|
||||
|
||||
The kernel parses these `.org` files at startup, extracts the `#+begin_src lisp` blocks, and hot-loads them into the live system. You can define a System 1 Prompt and a System 2 Verification Rule entirely within your personal notes.
|
||||
Capabilities are not hardcoded. They live as nodes in a **Skill Graph**. Each skill is an Org file that can declare dependencies on others, forming a network of recursive, networked intelligence.
|
||||
|
||||
#+begin_src mermaid
|
||||
graph LR
|
||||
Kernel[Daemon Core] --> Router[:skill-router]
|
||||
Router --> AtomicNotes[:skill-atomic-notes]
|
||||
Router --> WebResearch[:skill-web-research]
|
||||
|
||||
ProjectFoundry[:skill-project-foundry] --> Shell[:skill-shell-actuator]
|
||||
ProjectFoundry --> GTD[:skill-gtd]
|
||||
|
||||
Groomer[:skill-groomer] --> TDD[:skill-tdd-runner]
|
||||
Groomer --> Scientist[:skill-scientist]
|
||||
|
||||
Scientist --> SelfFix[:skill-self-fix]
|
||||
SelfFix --> Shell
|
||||
|
||||
%% Capabilities
|
||||
AtomicNotes --> OS[CLOSOS Object Store]
|
||||
Shell --> Hardware[Hardware / Sandbox]
|
||||
#+end_src
|
||||
|
||||
* Security & Isolation
|
||||
|
||||
Using `eval` on text generated by LLMs or extracted from text files is fundamentally dangerous. `org-agent` implements strict defense-in-depth:
|
||||
`org-agent` implements strict defense-in-depth to safely handle LLM-generated code:
|
||||
|
||||
** Layer 1: Lisp-Level Sandboxing
|
||||
- **Reader Safety:** `*read-eval*` is strictly disabled during AST parsing, completely neutralizing reader macro injection attacks (`#.(uiop:run-program ...)`).
|
||||
- **Package Jailing:** Every Org-Native skill is dynamically compiled into its own isolated Lisp package (`:org-agent.skills.<name>`). Skills cannot accidentally (or maliciously) overwrite the core System 2 gatekeeper or collide with other skills.
|
||||
- **Reader Safety:** `*read-eval*` is strictly disabled during AST parsing.
|
||||
- **Package Jailing:** Every Org-Native skill is dynamically compiled into its own isolated Lisp package (`:org-agent.skills.<name>`).
|
||||
- **Formal Verification Gate:** System 2 uses an SMT-based gate to prove that proposed shell commands do not violate security invariants.
|
||||
|
||||
** Layer 2: OS-Level Containerization
|
||||
The entire Common Lisp kernel can be isolated within a "Hardware Compartment" to protect the host OS.
|
||||
** Layer 2: Hardware Compartmentalization
|
||||
The entire kernel can be isolated within a "Hardware Compartment" (Docker, MicroVM, or Bare Metal) to protect the host OS.
|
||||
|
||||
* Documentation
|
||||
* The Evolution: Path to Order 5
|
||||
|
||||
Detailed specifications and planning documents are located in the [[file:docs/][docs/]] directory:
|
||||
- [[file:docs/PRD.org][Product Requirements Document (PRD)]]
|
||||
- [[file:docs/PROTOCOL.org][Communication Protocol (OACP)]]
|
||||
- [[file:docs/PHASE_2_ROADMAP.org][Phase 2 Roadmap]]
|
||||
- Specialized PRDs for [[file:docs/PRD_PROJECT_FOUNDRY.org][Project Foundry]], [[file:docs/PRD_ORG_DELIVERY.org][Org Delivery]], [[file:docs/PRD_LLM_CASCADE.org][LLM Cascade]], and more.
|
||||
The development of `org-agent` follows five orders of autonomy:
|
||||
|
||||
* Hardware Compartments (Deployment)
|
||||
|
||||
`org-agent` supports multiple levels of isolation. Choose the compartment that fits your security and performance needs. See the `deploy/` directory for templates.
|
||||
|
||||
** 1. Bare Metal
|
||||
Run directly on your host CPU for maximum performance. Best for development.
|
||||
|
||||
** 2. Docker (Standard)
|
||||
The default containerized experience.
|
||||
|
||||
** 3. LXC / Systemd-nspawn
|
||||
Lightweight Linux containers with lower overhead than Docker.
|
||||
|
||||
** 4. Virtual Machines (Debian / Fedora)
|
||||
Strong isolation using Vagrant/VirtualBox. Ensures zero-leakage from the Lisp machine.
|
||||
|
||||
** 5. Functional Deployment (Guix)
|
||||
Reproducible, declarative environment management.
|
||||
1. **Order 1: Intelligent Assistant:** Manages notes, code, and daily ingress drudgery.
|
||||
2. **Order 2: Sovereign Architect (Current):** Achieves **Recursive Self-Maintenance** (Grooming, Self-Repair, Hardware Inhabitation).
|
||||
3. **Order 3: Federated Swarm:** Multi-agent coordination across distributed hardware.
|
||||
4. **Order 4: Physical Actuator:** Orchestrating the material world (Home automation, SDR, Energy).
|
||||
5. **Order 5: The 100-Year Memex:** Absolute convergence where Agent and Hardware dissolve into a single, sovereign persistent ancestor.
|
||||
|
||||
* Installation & Setup Guide
|
||||
|
||||
This guide covers the standard distributed deployment: running the `org-agent` daemon on a remote Docker server, connecting to it from your local Emacs instance, and configuring dynamic LLMs (like OpenRouter).
|
||||
|
||||
** Step 1: Server Setup (Global Docker Compose)
|
||||
`org-agent` is designed to fit into a professional multi-app Docker environment.
|
||||
`org-agent` fits into a professional multi-app Docker environment.
|
||||
|
||||
1. **Clone the repository on your server:**
|
||||
#+begin_src bash
|
||||
git clone http://10.10.10.43:3000/amr/memex-amero.git /home/amr/memex
|
||||
git clone http://<gitea-ip>:3000/amr/memex-amero.git /home/amr/memex
|
||||
#+end_src
|
||||
|
||||
2. **Configure your Environment (.env):**
|
||||
Place your `.env` file in `/docker/compose/` alongside your master `docker-compose.yml`.
|
||||
#+begin_src bash
|
||||
# Create /docker/compose/.env with your keys:
|
||||
# OPENROUTER_API_KEY=your_key_here
|
||||
# ORG_AGENT_DAEMON_PORT=9105
|
||||
# ORG_AGENT_WEB_PORT=8080
|
||||
# MEMEX_DIR=/memex
|
||||
#+end_src
|
||||
Place your `.env` file in `/docker/compose/` with your keys.
|
||||
|
||||
3. **Integrate into Global Compose:**
|
||||
Add the following service fragment to your master file at `/docker/compose/docker-compose.yml`:
|
||||
#+begin_src yaml
|
||||
services:
|
||||
org-agent:
|
||||
build:
|
||||
context: /home/amr/memex/projects/org-agent
|
||||
dockerfile: deploy/docker/Dockerfile
|
||||
container_name: org-agent
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "9105:9105"
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
- /docker/memex:/memex
|
||||
env_file:
|
||||
- .env
|
||||
#+end_src
|
||||
|
||||
4. **Start the Service:**
|
||||
3. **Start the Service:**
|
||||
#+begin_src bash
|
||||
cd /docker/compose
|
||||
docker-compose up -d org-agent
|
||||
#+end_src
|
||||
|
||||
** Step 2: Local Emacs Setup (The Actuator)
|
||||
Your laptop acts as the sensor/actuator array.
|
||||
** Step 2: Local Emacs Setup
|
||||
1. Load `org-agent.el` in your local Emacs.
|
||||
2. Configure `org-agent-host` and `org-agent-port`.
|
||||
3. Run `M-x org-agent-connect` to establish the OACP socket.
|
||||
|
||||
1. **Load the Emacs Package:**
|
||||
Evaluate the `org-agent.el` file in your local Emacs.
|
||||
#+begin_src elisp
|
||||
(add-to-list 'load-path "/path/to/local/org-agent/src")
|
||||
(require 'org-agent)
|
||||
#+end_src
|
||||
* Documentation
|
||||
|
||||
2. **Configure the Connection:**
|
||||
Tell Emacs where your Docker server is located.
|
||||
#+begin_src elisp
|
||||
(setq org-agent-host "10.0.0.5") ;; Replace with your server's IP
|
||||
(setq org-agent-port 9105)
|
||||
#+end_src
|
||||
|
||||
3. **Connect to the Brain:**
|
||||
Run the interactive command to establish the OACP socket.
|
||||
`M-x org-agent-connect`
|
||||
|
||||
** Step 3: Dynamic Model Configuration (Homoiconic Setup)
|
||||
`org-agent` does not use external JSON config files for its behavior. You configure the agent directly within your Org-mode Memex.
|
||||
|
||||
1. Open any `.org` file in your Memex (e.g., `settings.org`).
|
||||
2. Add the following property to define your preferred model:
|
||||
#+begin_src org
|
||||
* Agent Settings
|
||||
:PROPERTIES:
|
||||
:LLM_MODEL_OPENROUTER: google/gemini-pro-1.5
|
||||
:END:
|
||||
#+end_src
|
||||
3. **Save the buffer.** The agent instantly detects the change via Emacs, updates its internal Object Store, and routes all future neural thoughts through the selected model.
|
||||
|
||||
To see all available models, simply type `@agent list models` in any Org buffer and save.
|
||||
|
||||
* Current State: Phase 3 (The Self-Editing Kernel) Achieved
|
||||
|
||||
- DONE Core Lisp microkernel (Cognitive Loop: Perceive -> Think -> Decide -> Act)
|
||||
- DONE OACP (Swank/Socket communication protocol) implemented
|
||||
- DONE Org AST-to-Lisp conversion logic & Object Store integration
|
||||
- DONE System 2 Safety Gating (The Harness) established
|
||||
- DONE Org-Native Skill parsing and loading
|
||||
- DONE Secure Docker containerization
|
||||
- DONE Skill Graph & Recursive Dependencies (Ars Contexta)
|
||||
- DONE Multi-Provider LLM Failover Cascade
|
||||
- DONE Context API (Peripheral Vision)
|
||||
- DONE Heartbeat Loop (Proactive Awareness)
|
||||
- DONE Immune System (Autonomous Self-Repair)
|
||||
- DONE Web Dashboard (Visual Telemetry)
|
||||
- DONE Org-Native Multi-modal Delivery (Signal/Telegram/Discord)
|
||||
- DONE Project Foundry (Autonomous Scaffolding & Git Stewardship)
|
||||
- DONE Strictly Org-mode Mandate (.md purge)
|
||||
- [[file:docs/PHASE_2_ROADMAP.org][Phase 2 Roadmap]]
|
||||
- [[file:docs/PRD_PROJECT_FOUNDRY.org][PRD: Project Foundry]]
|
||||
- [[file:docs/PROTOCOL_model_discovery.org][Protocol: Model Discovery]]
|
||||
- [[file:docs/reports/grooming-report-2026-03-31.org][Latest Grooming Report]]
|
||||
|
||||
50
docs/reports/grooming-report-2026-03-31.org
Normal file
50
docs/reports/grooming-report-2026-03-31.org
Normal file
@@ -0,0 +1,50 @@
|
||||
#+TITLE: Technical Debt Report: org-agent Kernel Grooming (2026-03-31)
|
||||
#+AUTHOR: Gemini CLI (Autonomous Groomer Agent)
|
||||
#+FILETAGS: :grooming:audit:technical-debt:psf:
|
||||
|
||||
* 1. Executive Summary
|
||||
The `org-agent` kernel is largely aligned with the "Neurosymbolic" and "Org-Native" mandates. However, several "just-in-case" features and hardcoded dependencies on non-sovereign LLM providers have been identified. Redundant function definitions in the core event bus indicate a need for surgical consolidation.
|
||||
|
||||
* 2. Audit Findings
|
||||
|
||||
** A. Redundancy in `core.lisp`
|
||||
`dispatch-action` is defined twice (lines 92 and 159). The second definition references an undefined variable `*actuators*`, which likely causes runtime errors if triggered.
|
||||
|
||||
** B. Hardcoded Proprietary Dependencies
|
||||
`embedding.lisp` is tightly coupled to Google's Gemini API (`text-embedding-004`). This violates the **Sovereignty Above All** invariant.
|
||||
|
||||
** C. Business Logic Leakage
|
||||
The `:auth-google-code` handler in `execute-system-action` (kernel core) is specific to a single skill/provider. This breaks the microkernel abstraction.
|
||||
|
||||
** D. Semantic Bloat (Just-in-case Computing)
|
||||
Every ingested Org element triggers a synchronous embedding call in `object-store.lisp`. This is inefficient and introduces high latency/cost for large files, violating the **Zero-Bloat Mandate**.
|
||||
|
||||
** E. Aspirational Code
|
||||
The "Distillation Loop" and "Swarm" logic are present in the kernel but appear incomplete or unused in the primary cognitive loop.
|
||||
|
||||
* 3. Proposed Surgical Refactors
|
||||
|
||||
** Refactor 1: Consolidate the Actuator Registry
|
||||
- **Action**: Remove the duplicate `dispatch-action` in `core.lisp`.
|
||||
- **Action**: Standardize on `*actuator-registry*`.
|
||||
- **Action**: Move the `:auth-google-code` logic out of the kernel and into a dedicated `auth` skill hook.
|
||||
- **Alignment**: **Zero-Bloat Mandate** (Invariant 3).
|
||||
|
||||
** Refactor 2: Pluggable & Lazy Embeddings
|
||||
- **Action**: Abstract the embedding logic into a provider-agnostic interface similar to `ask-neuro`.
|
||||
- **Action**: Allow the user to specify a local (e.g., Llama.cpp or Ollama) embedding model.
|
||||
- **Action**: Make embedding generation **lazy** or **opt-in** via Org properties (e.g., `:EMBED: t`) to avoid "Semantic Bloat".
|
||||
- **Alignment**: **Sovereignty Above All** (Invariant 1) & **Zero-Bloat** (Invariant 3).
|
||||
|
||||
** Refactor 3: Decouple "Peripheral Vision" (Context API)
|
||||
- **Action**: Move `context-query-store` and other search helpers from `object-store.lisp` into a `context-utils.lisp` or a core skill.
|
||||
- **Action**: Simplify the `org-object` struct to remove the `vector` field by default, storing vectors in a separate side-car hash table only when needed.
|
||||
- **Alignment**: **Minimalist Core Mandate** (Philosophy Mandate 2).
|
||||
|
||||
* 4. Verification (SOUL.org Alignment)
|
||||
|
||||
| Refactor | Invariant Alignment | Rationale |
|
||||
|----------+---------------------+-----------|
|
||||
| 1 | Radical Transparency | Removes confusing duplicate logic and prevents runtime panics. |
|
||||
| 2 | Sovereignty / Sustainability | Enables offline-first operation and reduces cloud dependency/cost. |
|
||||
| 3 | Zero-Bloat Mandate | Keeps the microkernel focused strictly on the cognitive OODA loop. |
|
||||
Reference in New Issue
Block a user