feat(kernel): upgrade neurosymbolic kernel to Order 2 (recursive self-maintenance, SOTA upgrades)
This commit is contained in:
29
docs/PRD.org
29
docs/PRD.org
@@ -1,29 +0,0 @@
|
||||
#+TITLE: PRD: org-agent Cognitive Core & Configuration
|
||||
#+AUTHOR: PSF Requirements Definer
|
||||
#+CREATED: [2026-03-23 Mon]
|
||||
#+STATUS: FROZEN
|
||||
|
||||
* 0. Core Mandates
|
||||
The `org-agent` project MUST adhere to these foundational mandates:
|
||||
- **Mandate 1: Strict Homoiconic Memory.** All documentation, planning, and system logic MUST be authored in Org-mode (.org) and Common Lisp. Markdown (.md) and JSON are strictly prohibited for internal use.
|
||||
- **Mandate 2: Minimalist Microkernel.** The core daemon MUST remain minimalist, handling only the cognitive loop, the persistent Object-Store, and the communication protocol. All domain-specific features and LLM provider logic MUST be implemented as hot-reloadable **Skills** living in the user's Memex.
|
||||
|
||||
* 1. Purpose
|
||||
The `org-agent` must transition from a monolithic prototype to a generalized neurosymbolic kernel.
|
||||
|
||||
* 2. Functional Requirements
|
||||
** 2.1. Cognitive Loop (PTA Refactor)
|
||||
- The daemon MUST implement a 4-stage pipeline: Perceive -> Think -> Decide -> Act.
|
||||
- System 1 (Neural) MUST be restricted to the 'Think' stage.
|
||||
- System 2 (Symbolic) MUST have absolute authority in the 'Decide' stage to block or modify neural proposals.
|
||||
- The I/O protocol (OACP) MUST be encapsulated in 'Perceive' and 'Act'.
|
||||
|
||||
** 2.2. Externalized Configuration (.env)
|
||||
- All secrets (API keys) and environment-specific settings (ports, paths) MUST live in a `.env` file.
|
||||
- The system MUST automatically load `.env` upon system initialization.
|
||||
- Secrets MUST NOT be hardcoded or checked into source control.
|
||||
|
||||
* 3. Success Criteria
|
||||
- TODO Daemon starts and loads LLM_API_KEY from .env.
|
||||
- TODO `cognitive-loop` successfully routes a `:buffer-update` event through all 4 stages.
|
||||
- TODO System 2 (`decide`) successfully blocks an `:eval` request containing "shell-command".
|
||||
@@ -1,137 +0,0 @@
|
||||
#+TITLE: org-agent Communication Protocol (OACP)
|
||||
#+AUTHOR: Agent
|
||||
#+DATE: 2026-03-22
|
||||
#+ID: org-agent-protocol
|
||||
#+STARTUP: content
|
||||
|
||||
* Core Mandates
|
||||
The OACP and the `org-agent` project MUST adhere to these foundational mandates:
|
||||
- **Mandate 1: Strict Homoiconic Memory.** All documentation, planning, and system logic MUST be authored in Org-mode (.org) and Common Lisp. Markdown (.md) and JSON are strictly prohibited for internal use.
|
||||
- **Mandate 2: Minimalist Microkernel.** The core daemon MUST remain minimalist, handling only the cognitive loop, the persistent Object-Store, and the communication protocol. All domain-specific features and LLM provider logic MUST be implemented as hot-reloadable **Skills** living in the user's Memex.
|
||||
|
||||
* Overview
|
||||
|
||||
OACP (org-agent Communication Protocol) defines the "nervous system" of the Neurosymbolic Lisp Machine. It facilitates bidirectional, asynchronous communication between the **Common Lisp Core** (The Soul/Daemon) and the **Emacs Interface** (The Actuator/Terminal).
|
||||
|
||||
The protocol is designed to be:
|
||||
- **Homoiconic:** Messages are native Lisp S-expressions (plists).
|
||||
- **Asynchronous:** Neither side should block waiting for the other.
|
||||
- **Extensible:** New sensors and actuators can be added by defining new plist keys.
|
||||
|
||||
* Framing & Transport
|
||||
|
||||
- **Transport:** TCP Socket (default port: 9105).
|
||||
- **Framing:** Each message is prefixed by a 6-character hexadecimal length string (e.g., `00002a` for a 42-byte message), followed by the S-expression string encoded in UTF-8.
|
||||
|
||||
* Message Structure
|
||||
|
||||
A standard message is a flat property list (plist):
|
||||
|
||||
#+begin_src lisp
|
||||
(:type TYPE :id ID :payload PAYLOAD)
|
||||
#+end_src
|
||||
|
||||
- **:type:** One of `:EVENT`, `:REQUEST`, `:RESPONSE`, or `:LOG`.
|
||||
- **:id:** A unique integer or string for correlation (mandatory for `:REQUEST` and `:RESPONSE`).
|
||||
- **:payload:** A nested plist containing the specific data or command.
|
||||
|
||||
* Perception (Emacs -> Core)
|
||||
|
||||
Emacs acts as the sensor array, pushing events to the Core daemon.
|
||||
|
||||
** :BUFFER-MODIFIED
|
||||
Sent when an Org buffer is changed or saved.
|
||||
#+begin_src lisp
|
||||
(:type :EVENT :payload (:sensor :buffer-update :file "/home/amr/org/todo.org" :state :saved))
|
||||
#+end_src
|
||||
|
||||
** :USER-INPUT
|
||||
Sent when the user explicitly triggers an agent action (e.g., `M-x org-agent-ask`).
|
||||
#+begin_src lisp
|
||||
(:type :EVENT :payload (:sensor :user-prompt :text "Refactor this subtree" :context :current-subtree))
|
||||
#+end_src
|
||||
|
||||
** :CURSOR-MOVED
|
||||
Sent when the agent needs to track the user's focus.
|
||||
#+begin_src lisp
|
||||
(:type :EVENT :payload (:sensor :focus :buffer "init.el" :line 42 :column 0))
|
||||
#+end_src
|
||||
|
||||
* Action (Core -> Emacs)
|
||||
|
||||
The Core daemon sends requests to Emacs to perform physical actions in the environment.
|
||||
|
||||
** :EXECUTE-ELISP
|
||||
The primary "actuator."
|
||||
#+begin_src lisp
|
||||
(:type :REQUEST :id 101 :payload (:action :eval :code "(org-todo \"NEXT\")"))
|
||||
#+end_src
|
||||
|
||||
** :UI-NOTIFY
|
||||
Display info to the user without interrupting focus.
|
||||
#+begin_src lisp
|
||||
(:type :REQUEST :id 102 :payload (:action :message :text "I have identified a contradiction in your GTD.org" :level :warning))
|
||||
#+end_src
|
||||
|
||||
** :PROVIDE-COMPLETION
|
||||
Provide LLM-driven completions for the current point.
|
||||
#+begin_src lisp
|
||||
(:type :REQUEST :id 103 :payload (:action :complete :prefix "defun" :suggestions ("defun-agent" "defun-percieve")))
|
||||
#+end_src
|
||||
|
||||
** :ORG-DELIVERY
|
||||
Enqueue external messages.
|
||||
#+begin_src lisp
|
||||
(:type :REQUEST :target :delivery :payload (:channel :signal :to "+1..." :text "Hello"))
|
||||
#+end_src
|
||||
|
||||
** :PROJECT-SCAFFOLD
|
||||
Create new project folders and headings.
|
||||
#+begin_src lisp
|
||||
(:type :REQUEST :target :foundry :payload (:action :scaffold :name "Project-X" :type "Lisp"))
|
||||
#+end_src
|
||||
|
||||
** :SYSTEM-SELF-EDIT
|
||||
Kernel-level self-modification.
|
||||
#+begin_src lisp
|
||||
(:type :REQUEST :target :system :payload (:action :create-skill :filename "skill-x.org" :content "..."))
|
||||
#+end_src
|
||||
|
||||
* Metadata & Handshake
|
||||
|
||||
** :HELLO
|
||||
Sent by both sides upon connection.
|
||||
#+begin_src lisp
|
||||
(:type :EVENT :payload (:action :handshake :version "0.1.0" :capabilities (:auth :swank :org-ast)))
|
||||
#+end_src
|
||||
|
||||
* Cognitive Loop Internal API (Foundry Standard)
|
||||
|
||||
To ensure neurosymbolic sovereignty, the Core Daemon must strictly separate stages.
|
||||
|
||||
** 1. Perceive
|
||||
- **Signature:** `(perceive raw-stimulus) -> context-plist`
|
||||
- **Responsibility:** Protocol decoding, Object-Store synchronization, context augmentation.
|
||||
|
||||
** 2. Think (System 1)
|
||||
- **Signature:** `(think context-plist) -> proposed-action-plist`
|
||||
- **Responsibility:** Neural pattern matching, associative retrieval, unverified proposal generation.
|
||||
|
||||
** 3. Decide (System 2)
|
||||
- **Signature:** `(decide proposed-action context) -> approved-action-plist`
|
||||
- **Responsibility:** Symbolic verification, safety gating, deterministic rule application (e.g., GTD logic).
|
||||
|
||||
** 4. Act (Dispatch)
|
||||
- **Signature:** `(act stream approved-action)` / `(dispatch-action approved-action)`
|
||||
- **Responsibility:** Actuator lookup and transmission. Targets: `:emacs`, `:delivery`, `:system`, `:shell`.
|
||||
|
||||
** 5. Context API (System 1 Peripheral Vision)
|
||||
- **(context-list-all-skills):** Introspect the brain hierarchy.
|
||||
- **(context-get-skill-source name):** Read the logic graph.
|
||||
- **(context-resolve-path path):** Environment-relative path expansion.
|
||||
- **(context-get-system-logs limit):** Perception of kernel failures.
|
||||
|
||||
* Success Metrics
|
||||
- Latency between `:EVENT` and `:RESPONSE` < 100ms for UI actions.
|
||||
- 100% fidelity in Org AST preservation across the bridge.
|
||||
- Zero UI blocking in Emacs during heavy CL reasoning loops.
|
||||
183
docs/README.org
Normal file
183
docs/README.org
Normal file
@@ -0,0 +1,183 @@
|
||||
#+TITLE: org-agent: The Neurosymbolic Kernel
|
||||
#+AUTHOR: User
|
||||
#+CREATED: [2026-03-17 Tue]
|
||||
#+UPDATED: [2026-03-24 Tue]
|
||||
|
||||
A hyper-minimalist, self-editing, proactive AI agent framework. `org-agent` acts as the "executive soul" of a personal OS, using Org-mode as its native memory and Common Lisp as its deterministic reasoning engine.
|
||||
|
||||
* The Philosophy
|
||||
|
||||
** Mandate 1: Strictly Org-mode and Common Lisp
|
||||
The system is built on a "No Legacy" policy. Markdown (.md) and JSON are strictly prohibited for internal system logic, planning, and memory. Org-mode is the native Abstract Syntax Tree (AST) for both human and machine, and Common Lisp (SBCL) is the deterministic reasoning engine.
|
||||
|
||||
** Mandate 2: Minimalist Core, Skill-Based Extension
|
||||
The `org-agent` kernel (the Daemon) MUST remain a minimalist microkernel. It handles only the cognitive loop, the persistent Object-Store, and the communication protocol. All business logic, LLM provider connectors, and task-management rules MUST be implemented as hot-reloadable **Skills** living in the user's Memex.
|
||||
|
||||
** Why Org-mode? (Homoiconic Memory)
|
||||
Most agent frameworks rely on a messy combination of Python scripts, JSON states, and Markdown prompts. This breaks the human-agent interface. JSON is for machines; Markdown is for humans.
|
||||
|
||||
*Org-mode is for both.* It provides a rigorous, hierarchical Abstract Syntax Tree (AST) that a machine can navigate deterministically, while remaining a perfectly ergonomic, human-readable text document. In this system, your notes, your tasks, your prompts, and your agent's code all live in the exact same format.
|
||||
|
||||
** Why Common Lisp? (The Kernel vs. The Actuators)
|
||||
The `org-agent` kernel is built in Common Lisp to provide a persistent, high-performance background process (SBCL) that maintains a live, threaded Object Store in RAM. It performs heavy neurosymbolic reasoning asynchronously, decoupled from any single user interface.
|
||||
|
||||
This architecture treats all interfaces as external **Actuators** and **Sensors**:
|
||||
- **Editor Actuator (Emacs):** A sensor array that detects file changes and executes structural refactoring.
|
||||
- **Messaging Actuator (Signal/Telegram/Discord):** A delivery channel for proactive alerts and human-in-the-loop decisions.
|
||||
- **Web Actuator (Dashboard):** A visual telemetry interface for monitoring the live kernel state.
|
||||
|
||||
* The Architecture: The Cognitive Loop
|
||||
|
||||
The core engine is agnostic to both business logic and communication channels. It routes data through a strict four-stage cognitive pipeline:
|
||||
|
||||
1. **Perceive:** Sensors (Emacs, Webhooks, CRON) send updates over the Org-Agent Communication Protocol (OACP). The kernel updates its live Object Store.
|
||||
2. **Think (System 1):** The `neuro.lisp` module queries an LLM (e.g., Gemini, OpenAI, or local models) based on the context, asking for an intuitive, pattern-matched suggestion. It returns an *unverified* proposed action.
|
||||
3. **Decide (System 2):** The `symbolic.lisp` module is the absolute gatekeeper. It takes the LLM's proposal and runs it through strict Lisp constraints (e.g., "A parent task cannot be marked DONE if it has active TODO children"). If the logic fails, the LLM is overruled.
|
||||
4. **Act:** Verified commands are dispatched to the appropriate Actuators (refactoring a buffer, sending a Signal message, or updating a database).
|
||||
|
||||
* Extensibility: The Org-Native Skill Standard
|
||||
|
||||
To keep the core microkernel minimal, all capabilities (API connectors, GTD logic, Atomic Notes (Zettelkasten) memory management) are abstracted into **Skills**.
|
||||
|
||||
Adhering to the Lisp Machine Mandate (Code is Data), a skill is not a Python folder. A skill is a single `.org` file located in the Atomic Notes (Zettelkasten) directory.
|
||||
|
||||
The kernel parses these `.org` files at startup, extracts the `#+begin_src lisp` blocks, and hot-loads them into the live system. You can define a System 1 Prompt and a System 2 Verification Rule entirely within your personal notes.
|
||||
|
||||
* Security & Isolation
|
||||
|
||||
Using `eval` on text generated by LLMs or extracted from text files is fundamentally dangerous. `org-agent` implements strict defense-in-depth:
|
||||
|
||||
** Layer 1: Lisp-Level Sandboxing
|
||||
- **Reader Safety:** `*read-eval*` is strictly disabled during AST parsing, completely neutralizing reader macro injection attacks (`#.(uiop:run-program ...)`).
|
||||
- **Package Jailing:** Every Org-Native skill is dynamically compiled into its own isolated Lisp package (`:org-agent.skills.<name>`). Skills cannot accidentally (or maliciously) overwrite the core System 2 gatekeeper or collide with other skills.
|
||||
|
||||
** Layer 2: OS-Level Containerization
|
||||
The entire Common Lisp kernel can be isolated within a "Hardware Compartment" to protect the host OS.
|
||||
|
||||
* Documentation
|
||||
|
||||
Detailed specifications and planning documents are located in the [[file:docs/][docs/]] directory:
|
||||
- [[file:docs/PRD.org][Product Requirements Document (PRD)]]
|
||||
- [[file:docs/PROTOCOL.org][Communication Protocol (OACP)]]
|
||||
- [[file:docs/PHASE_2_ROADMAP.org][Phase 2 Roadmap]]
|
||||
- Specialized PRDs for [[file:docs/PRD_PROJECT_FOUNDRY.org][Project Foundry]], [[file:docs/PRD_ORG_DELIVERY.org][Org Delivery]], [[file:docs/PRD_LLM_CASCADE.org][LLM Cascade]], and more.
|
||||
|
||||
* Hardware Compartments (Deployment)
|
||||
|
||||
`org-agent` supports multiple levels of isolation. Choose the compartment that fits your security and performance needs. See the `deploy/` directory for templates.
|
||||
|
||||
** 1. Bare Metal
|
||||
Run directly on your host CPU for maximum performance. Best for development.
|
||||
|
||||
** 2. Docker (Standard)
|
||||
The default containerized experience.
|
||||
|
||||
** 3. LXC / Systemd-nspawn
|
||||
Lightweight Linux containers with lower overhead than Docker.
|
||||
|
||||
** 4. Virtual Machines (Debian / Fedora)
|
||||
Strong isolation using Vagrant/VirtualBox. Ensures zero-leakage from the Lisp machine.
|
||||
|
||||
** 5. Functional Deployment (Guix)
|
||||
Reproducible, declarative environment management.
|
||||
|
||||
* Installation & Setup Guide
|
||||
|
||||
This guide covers the standard distributed deployment: running the `org-agent` daemon on a remote Docker server, connecting to it from your local Emacs instance, and configuring dynamic LLMs (like OpenRouter).
|
||||
|
||||
** Step 1: Server Setup (Global Docker Compose)
|
||||
`org-agent` is designed to fit into a professional multi-app Docker environment.
|
||||
|
||||
1. **Clone the repository on your server:**
|
||||
#+begin_src bash
|
||||
git clone http://10.10.10.43:3000/amr/memex-amero.git /home/amr/memex
|
||||
#+end_src
|
||||
|
||||
2. **Configure your Environment (.env):**
|
||||
Place your `.env` file in `/docker/compose/` alongside your master `docker-compose.yml`.
|
||||
#+begin_src bash
|
||||
# Create /docker/compose/.env with your keys:
|
||||
# OPENROUTER_API_KEY=your_key_here
|
||||
# ORG_AGENT_DAEMON_PORT=9105
|
||||
# ORG_AGENT_WEB_PORT=8080
|
||||
# MEMEX_DIR=/memex
|
||||
#+end_src
|
||||
|
||||
3. **Integrate into Global Compose:**
|
||||
Add the following service fragment to your master file at `/docker/compose/docker-compose.yml`:
|
||||
#+begin_src yaml
|
||||
services:
|
||||
org-agent:
|
||||
build:
|
||||
context: /home/amr/memex/projects/org-agent
|
||||
dockerfile: deploy/docker/Dockerfile
|
||||
container_name: org-agent
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "9105:9105"
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
- /docker/memex:/memex
|
||||
env_file:
|
||||
- .env
|
||||
#+end_src
|
||||
|
||||
4. **Start the Service:**
|
||||
#+begin_src bash
|
||||
cd /docker/compose
|
||||
docker-compose up -d org-agent
|
||||
#+end_src
|
||||
|
||||
** Step 2: Local Emacs Setup (The Actuator)
|
||||
Your laptop acts as the sensor/actuator array.
|
||||
|
||||
1. **Load the Emacs Package:**
|
||||
Evaluate the `org-agent.el` file in your local Emacs.
|
||||
#+begin_src elisp
|
||||
(add-to-list 'load-path "/path/to/local/org-agent/src")
|
||||
(require 'org-agent)
|
||||
#+end_src
|
||||
|
||||
2. **Configure the Connection:**
|
||||
Tell Emacs where your Docker server is located.
|
||||
#+begin_src elisp
|
||||
(setq org-agent-host "10.0.0.5") ;; Replace with your server's IP
|
||||
(setq org-agent-port 9105)
|
||||
#+end_src
|
||||
|
||||
3. **Connect to the Brain:**
|
||||
Run the interactive command to establish the OACP socket.
|
||||
`M-x org-agent-connect`
|
||||
|
||||
** Step 3: Dynamic Model Configuration (Homoiconic Setup)
|
||||
`org-agent` does not use external JSON config files for its behavior. You configure the agent directly within your Org-mode Memex.
|
||||
|
||||
1. Open any `.org` file in your Memex (e.g., `settings.org`).
|
||||
2. Add the following property to define your preferred model:
|
||||
#+begin_src org
|
||||
* Agent Settings
|
||||
:PROPERTIES:
|
||||
:LLM_MODEL_OPENROUTER: google/gemini-pro-1.5
|
||||
:END:
|
||||
#+end_src
|
||||
3. **Save the buffer.** The agent instantly detects the change via Emacs, updates its internal Object Store, and routes all future neural thoughts through the selected model.
|
||||
|
||||
To see all available models, simply type `@agent list models` in any Org buffer and save.
|
||||
|
||||
* Current State: Phase 3 (The Self-Editing Kernel) Achieved
|
||||
|
||||
- DONE Core Lisp microkernel (Cognitive Loop: Perceive -> Think -> Decide -> Act)
|
||||
- DONE OACP (Swank/Socket communication protocol) implemented
|
||||
- DONE Org AST-to-Lisp conversion logic & Object Store integration
|
||||
- DONE System 2 Safety Gating (The Harness) established
|
||||
- DONE Org-Native Skill parsing and loading
|
||||
- DONE Secure Docker containerization
|
||||
- DONE Skill Graph & Recursive Dependencies (Ars Contexta)
|
||||
- DONE Multi-Provider LLM Failover Cascade
|
||||
- DONE Context API (Peripheral Vision)
|
||||
- DONE Heartbeat Loop (Proactive Awareness)
|
||||
- DONE Immune System (Autonomous Self-Repair)
|
||||
- DONE Web Dashboard (Visual Telemetry)
|
||||
- DONE Org-Native Multi-modal Delivery (Signal/Telegram/Discord)
|
||||
- DONE Project Foundry (Autonomous Scaffolding & Git Stewardship)
|
||||
- DONE Strictly Org-mode Mandate (.md purge)
|
||||
Reference in New Issue
Block a user