amr/passepartout
1
0
Fork 0
Code Issues 15 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Revert "docs: finalize architecture and philosophy with Mermaid diagrams and PSF mandates"

Browse Source 
This reverts commit 6c17619492.
This commit is contained in:
Amr Gharbeia
2026-04-01 16:14:20 -04:00
parent 6c17619492
commit 0eb9e3773f
8 changed files with 349 additions and 336 deletions
Download Patch File Download Diff File Expand all files Collapse all files

258
docs/README.org
View File

@@ -1,133 +1,183 @@
#+TITLE: org-agent: The Neurosymbolic Kernel
#+AUTHOR: User
#+CREATED: [2026-03-17 Tue]
#+UPDATED: [2026-03-31 Tue]
#+UPDATED: [2026-03-24 Tue]
A hyper-minimalist, self-editing, proactive AI agent framework. `org-agent` acts as the "executive soul" of a personal OS, using Org-mode as its native memory and Common Lisp as its deterministic reasoning engine.
* The Philosophy: The PSF Mandates
* The Philosophy
The system is governed by three non-negotiable mandates from the **Personal Software Foundry (PSF)** that ensure long-term sustainability and human-agent alignment:
** Mandate 1: Strictly Org-mode and Common Lisp
The system is built on a "No Legacy" policy. Markdown (.md) and JSON are strictly prohibited for internal system logic, planning, and memory. Org-mode is the native Abstract Syntax Tree (AST) for both human and machine, and Common Lisp (SBCL) is the deterministic reasoning engine.
** 1. Lisp Machine Sovereignty
The agent is a living Lisp image (SBCL). Code is updated via late-binding, hot-reloading into the active image. Manual "restarts" are considered a failure of architectural integrity. The system is a "Living Organism" rather than a static script.
** Mandate 2: Minimalist Core, Skill-Based Extension
The `org-agent` kernel (the Daemon) MUST remain a minimalist microkernel. It handles only the cognitive loop, the persistent Object-Store, and the communication protocol. All business logic, LLM provider connectors, and task-management rules MUST be implemented as hot-reloadable **Skills** living in the user's Memex.
** 2. Homoiconic Memory (The Org Mandate)
Every document, plan, PRD, and skill in the system MUST be written in **Org-mode**. Markdown (.md) and JSON/YAML/XML are strictly prohibited for internal system logic. Org-mode provides a rigorous, hierarchical AST for both machine and human, ensuring they share the exact same mental model.
** Why Org-mode? (Homoiconic Memory)
Most agent frameworks rely on a messy combination of Python scripts, JSON states, and Markdown prompts. This breaks the human-agent interface. JSON is for machines; Markdown is for humans.
** 3. Literate Programming (Knuth-Sovereign Principle)
The PSF treats code not as instructions for a computer, but as a "work of literature" for humans that happens to be executable. All system logic is implemented as **Literate Org files**. Documentation (the "Why") and Implementation (the "How") are unified, ensuring the system's **Institutional Memory** is preserved.
*Org-mode is for both.* It provides a rigorous, hierarchical Abstract Syntax Tree (AST) that a machine can navigate deterministically, while remaining a perfectly ergonomic, human-readable text document. In this system, your notes, your tasks, your prompts, and your agent's code all live in the exact same format.
* The Architecture: The Neurosymbolic Cognitive Loop
** Why Common Lisp? (The Kernel vs. The Actuators)
The `org-agent` kernel is built in Common Lisp to provide a persistent, high-performance background process (SBCL) that maintains a live, threaded Object Store in RAM. It performs heavy neurosymbolic reasoning asynchronously, decoupled from any single user interface.
The kernel operates on a deterministic pipeline (OODA) that bridges intuitive "Neural" thinking with rigorous "Symbolic" logic.
This architecture treats all interfaces as external **Actuators** and **Sensors**:
- **Editor Actuator (Emacs):** A sensor array that detects file changes and executes structural refactoring.
- **Messaging Actuator (Signal/Telegram/Discord):** A delivery channel for proactive alerts and human-in-the-loop decisions.
- **Web Actuator (Dashboard):** A visual telemetry interface for monitoring the live kernel state.
#+begin_src mermaid
graph TD
subgraph Sensors
Emacs[Emacs Bridge]
Cron[Cron Heartbeat]
Signal[Signal Inbound]
end
* The Architecture: The Cognitive Loop
subgraph Kernel [The Neurosymbolic Kernel]
Perceive[Perceive: Update Object Store]
Think[Think: System 1 Neural Proposal]
Decide[Decide: System 2 Symbolic Gate]
Act[Act: Dispatch to Actuators]
end
The core engine is agnostic to both business logic and communication channels. It routes data through a strict four-stage cognitive pipeline:
subgraph Actuators
Refactor[Org Buffer Refactor]
Shell[Shell/Sandbox Actuator]
Notify[External Delivery]
end
1. **Perceive:** Sensors (Emacs, Webhooks, CRON) send updates over the Org-Agent Communication Protocol (OACP). The kernel updates its live Object Store.
2. **Think (System 1):** The `neuro.lisp` module queries an LLM (e.g., Gemini, OpenAI, or local models) based on the context, asking for an intuitive, pattern-matched suggestion. It returns an *unverified* proposed action.
3. **Decide (System 2):** The `symbolic.lisp` module is the absolute gatekeeper. It takes the LLM's proposal and runs it through strict Lisp constraints (e.g., "A parent task cannot be marked DONE if it has active TODO children"). If the logic fails, the LLM is overruled.
4. **Act:** Verified commands are dispatched to the appropriate Actuators (refactoring a buffer, sending a Signal message, or updating a database).
Sensors --> Perceive
Perceive --> Think
Think --> Decide
Decide --> Act
Act --> Actuators
%% Feedback Loop
Actuators -.-> Sensors
#+end_src
* Extensibility: The Org-Native Skill Standard
- **System 1 (Neural):** Uses LLMs (Gemini, Ollama) for pattern matching and creative problem-solving. It is fast but unreliable.
- **System 2 (Symbolic):** Uses Common Lisp and formal logic to verify LLM proposals against security invariants and structural rules. It is the absolute gatekeeper.
To keep the core microkernel minimal, all capabilities (API connectors, GTD logic, Atomic Notes (Zettelkasten) memory management) are abstracted into **Skills**.
* Extensibility: The Skill Graph
Adhering to the Lisp Machine Mandate (Code is Data), a skill is not a Python folder. A skill is a single `.org` file located in the Atomic Notes (Zettelkasten) directory.
Capabilities are not hardcoded. They live as nodes in a **Skill Graph**. Each skill is an Org file that can declare dependencies on others, forming a network of recursive, networked intelligence.
#+begin_src mermaid
graph LR
Kernel[Daemon Core] --> Router[:skill-router]
Router --> AtomicNotes[:skill-atomic-notes]
Router --> WebResearch[:skill-web-research]
ProjectFoundry[:skill-project-foundry] --> Shell[:skill-shell-actuator]
ProjectFoundry --> GTD[:skill-gtd]
Groomer[:skill-groomer] --> TDD[:skill-tdd-runner]
Groomer --> Scientist[:skill-scientist]
Scientist --> SelfFix[:skill-self-fix]
SelfFix --> Shell
%% Capabilities
AtomicNotes --> OS[CLOSOS Object Store]
Shell --> Hardware[Hardware / Sandbox]
#+end_src
The kernel parses these `.org` files at startup, extracts the `#+begin_src lisp` blocks, and hot-loads them into the live system. You can define a System 1 Prompt and a System 2 Verification Rule entirely within your personal notes.
* Security & Isolation
`org-agent` implements strict defense-in-depth to safely handle LLM-generated code:
Using `eval` on text generated by LLMs or extracted from text files is fundamentally dangerous. `org-agent` implements strict defense-in-depth:
** Layer 1: Lisp-Level Sandboxing
- **Reader Safety:** `*read-eval*` is strictly disabled during AST parsing.
- **Package Jailing:** Every Org-Native skill is dynamically compiled into its own isolated Lisp package (`:org-agent.skills.<name>`).
- **Formal Verification Gate:** System 2 uses an SMT-based gate to prove that proposed shell commands do not violate security invariants.
- **Reader Safety:** `*read-eval*` is strictly disabled during AST parsing, completely neutralizing reader macro injection attacks (`#.(uiop:run-program ...)`).
- **Package Jailing:** Every Org-Native skill is dynamically compiled into its own isolated Lisp package (`:org-agent.skills.<name>`). Skills cannot accidentally (or maliciously) overwrite the core System 2 gatekeeper or collide with other skills.
** Layer 2: Hardware Compartmentalization
The entire kernel can be isolated within a "Hardware Compartment" (Docker, MicroVM, or Bare Metal) to protect the host OS.
* The Evolution: Path to Order 5
The development of `org-agent` follows five orders of autonomy:
1. **Order 1: Intelligent Assistant:** Manages notes, code, and daily ingress drudgery.
2. **Order 2: Sovereign Architect (Current):** Achieves **Recursive Self-Maintenance** (Grooming, Self-Repair, Hardware Inhabitation).
3. **Order 3: Federated Swarm:** Multi-agent coordination across distributed hardware.
4. **Order 4: Physical Actuator:** Orchestrating the material world (Home automation, SDR, Energy).
5. **Order 5: The 100-Year Memex:** Absolute convergence where Agent and Hardware dissolve into a single, sovereign persistent ancestor.
* Installation & Setup Guide
** Step 1: Server Setup (Global Docker Compose)
`org-agent` fits into a professional multi-app Docker environment.
1. **Clone the repository on your server:**
#+begin_src bash
git clone http://<gitea-ip>:3000/amr/memex-amero.git /home/amr/memex
#+end_src
2. **Configure your Environment (.env):**
Place your `.env` file in `/docker/compose/` with your keys.
3. **Start the Service:**
#+begin_src bash
docker-compose up -d org-agent
#+end_src
** Step 2: Local Emacs Setup
1. Load `org-agent.el` in your local Emacs.
2. Configure `org-agent-host` and `org-agent-port`.
3. Run `M-x org-agent-connect` to establish the OACP socket.
** Layer 2: OS-Level Containerization
The entire Common Lisp kernel can be isolated within a "Hardware Compartment" to protect the host OS.
* Documentation
Detailed specifications and planning documents are located in the [[file:docs/][docs/]] directory:
- [[file:docs/PRD.org][Product Requirements Document (PRD)]]
- [[file:docs/PROTOCOL.org][Communication Protocol (OACP)]]
- [[file:docs/PHASE_2_ROADMAP.org][Phase 2 Roadmap]]
- [[file:docs/PRD_PROJECT_FOUNDRY.org][PRD: Project Foundry]]
- [[file:docs/PROTOCOL_model_discovery.org][Protocol: Model Discovery]]
- [[file:docs/reports/grooming-report-2026-03-31.org][Latest Grooming Report]]
- Specialized PRDs for [[file:docs/PRD_PROJECT_FOUNDRY.org][Project Foundry]], [[file:docs/PRD_ORG_DELIVERY.org][Org Delivery]], [[file:docs/PRD_LLM_CASCADE.org][LLM Cascade]], and more.
* Hardware Compartments (Deployment)
`org-agent` supports multiple levels of isolation. Choose the compartment that fits your security and performance needs. See the `deploy/` directory for templates.
** 1. Bare Metal
Run directly on your host CPU for maximum performance. Best for development.
** 2. Docker (Standard)
The default containerized experience.
** 3. LXC / Systemd-nspawn
Lightweight Linux containers with lower overhead than Docker.
** 4. Virtual Machines (Debian / Fedora)
Strong isolation using Vagrant/VirtualBox. Ensures zero-leakage from the Lisp machine.
** 5. Functional Deployment (Guix)
Reproducible, declarative environment management.
* Installation & Setup Guide
This guide covers the standard distributed deployment: running the `org-agent` daemon on a remote Docker server, connecting to it from your local Emacs instance, and configuring dynamic LLMs (like OpenRouter).
** Step 1: Server Setup (Global Docker Compose)
`org-agent` is designed to fit into a professional multi-app Docker environment.
1. **Clone the repository on your server:**
#+begin_src bash
git clone http://10.10.10.43:3000/amr/memex-amero.git /home/amr/memex
#+end_src
2. **Configure your Environment (.env):**
Place your `.env` file in `/docker/compose/` alongside your master `docker-compose.yml`.
#+begin_src bash
# Create /docker/compose/.env with your keys:
# OPENROUTER_API_KEY=your_key_here
# ORG_AGENT_DAEMON_PORT=9105
# ORG_AGENT_WEB_PORT=8080
# MEMEX_DIR=/memex
#+end_src
3. **Integrate into Global Compose:**
Add the following service fragment to your master file at `/docker/compose/docker-compose.yml`:
#+begin_src yaml
services:
org-agent:
build:
context: /home/amr/memex/projects/org-agent
dockerfile: deploy/docker/Dockerfile
container_name: org-agent
restart: unless-stopped
ports:
- "9105:9105"
- "8080:8080"
volumes:
- /docker/memex:/memex
env_file:
- .env
#+end_src
4. **Start the Service:**
#+begin_src bash
cd /docker/compose
docker-compose up -d org-agent
#+end_src
** Step 2: Local Emacs Setup (The Actuator)
Your laptop acts as the sensor/actuator array.
1. **Load the Emacs Package:**
Evaluate the `org-agent.el` file in your local Emacs.
#+begin_src elisp
(add-to-list 'load-path "/path/to/local/org-agent/src")
(require 'org-agent)
#+end_src
2. **Configure the Connection:**
Tell Emacs where your Docker server is located.
#+begin_src elisp
(setq org-agent-host "10.0.0.5") ;; Replace with your server's IP
(setq org-agent-port 9105)
#+end_src
3. **Connect to the Brain:**
Run the interactive command to establish the OACP socket.
`M-x org-agent-connect`
** Step 3: Dynamic Model Configuration (Homoiconic Setup)
`org-agent` does not use external JSON config files for its behavior. You configure the agent directly within your Org-mode Memex.
1. Open any `.org` file in your Memex (e.g., `settings.org`).
2. Add the following property to define your preferred model:
#+begin_src org
* Agent Settings
:PROPERTIES:
:LLM_MODEL_OPENROUTER: google/gemini-pro-1.5
:END:
#+end_src
3. **Save the buffer.** The agent instantly detects the change via Emacs, updates its internal Object Store, and routes all future neural thoughts through the selected model.
To see all available models, simply type `@agent list models` in any Org buffer and save.
* Current State: Phase 3 (The Self-Editing Kernel) Achieved
- DONE Core Lisp microkernel (Cognitive Loop: Perceive -> Think -> Decide -> Act)
- DONE OACP (Swank/Socket communication protocol) implemented
- DONE Org AST-to-Lisp conversion logic & Object Store integration
- DONE System 2 Safety Gating (The Harness) established
- DONE Org-Native Skill parsing and loading
- DONE Secure Docker containerization
- DONE Skill Graph & Recursive Dependencies (Ars Contexta)
- DONE Multi-Provider LLM Failover Cascade
- DONE Context API (Peripheral Vision)
- DONE Heartbeat Loop (Proactive Awareness)
- DONE Immune System (Autonomous Self-Repair)
- DONE Web Dashboard (Visual Telemetry)
- DONE Org-Native Multi-modal Delivery (Signal/Telegram/Discord)
- DONE Project Foundry (Autonomous Scaffolding & Git Stewardship)
- DONE Strictly Org-mode Mandate (.md purge)

50
docs/reports/grooming-report-2026-03-31.org
View File

@@ -1,50 +0,0 @@
#+TITLE: Technical Debt Report: org-agent Kernel Grooming (2026-03-31)
#+AUTHOR: Gemini CLI (Autonomous Groomer Agent)
#+FILETAGS: :grooming:audit:technical-debt:psf:
* 1. Executive Summary
The `org-agent` kernel is largely aligned with the "Neurosymbolic" and "Org-Native" mandates. However, several "just-in-case" features and hardcoded dependencies on non-sovereign LLM providers have been identified. Redundant function definitions in the core event bus indicate a need for surgical consolidation.
* 2. Audit Findings
** A. Redundancy in `core.lisp`
`dispatch-action` is defined twice (lines 92 and 159). The second definition references an undefined variable `*actuators*`, which likely causes runtime errors if triggered.
** B. Hardcoded Proprietary Dependencies
`embedding.lisp` is tightly coupled to Google's Gemini API (`text-embedding-004`). This violates the **Sovereignty Above All** invariant.
** C. Business Logic Leakage
The `:auth-google-code` handler in `execute-system-action` (kernel core) is specific to a single skill/provider. This breaks the microkernel abstraction.
** D. Semantic Bloat (Just-in-case Computing)
Every ingested Org element triggers a synchronous embedding call in `object-store.lisp`. This is inefficient and introduces high latency/cost for large files, violating the **Zero-Bloat Mandate**.
** E. Aspirational Code
The "Distillation Loop" and "Swarm" logic are present in the kernel but appear incomplete or unused in the primary cognitive loop.
* 3. Proposed Surgical Refactors
** Refactor 1: Consolidate the Actuator Registry
- **Action**: Remove the duplicate `dispatch-action` in `core.lisp`.
- **Action**: Standardize on `*actuator-registry*`.
- **Action**: Move the `:auth-google-code` logic out of the kernel and into a dedicated `auth` skill hook.
- **Alignment**: **Zero-Bloat Mandate** (Invariant 3).
** Refactor 2: Pluggable & Lazy Embeddings
- **Action**: Abstract the embedding logic into a provider-agnostic interface similar to `ask-neuro`.
- **Action**: Allow the user to specify a local (e.g., Llama.cpp or Ollama) embedding model.
- **Action**: Make embedding generation **lazy** or **opt-in** via Org properties (e.g., `:EMBED: t`) to avoid "Semantic Bloat".
- **Alignment**: **Sovereignty Above All** (Invariant 1) & **Zero-Bloat** (Invariant 3).
** Refactor 3: Decouple "Peripheral Vision" (Context API)
- **Action**: Move `context-query-store` and other search helpers from `object-store.lisp` into a `context-utils.lisp` or a core skill.
- **Action**: Simplify the `org-object` struct to remove the `vector` field by default, storing vectors in a separate side-car hash table only when needed.
- **Alignment**: **Minimalist Core Mandate** (Philosophy Mandate 2).
* 4. Verification (SOUL.org Alignment)
| Refactor | Invariant Alignment | Rationale |
|----------+---------------------+-----------|
| 1 | Radical Transparency | Removes confusing duplicate logic and prevents runtime panics. |
| 2 | Sovereignty / Sustainability | Enables offline-first operation and reduces cloud dependency/cost. |
| 3 | Zero-Bloat Mandate | Keeps the microkernel focused strictly on the cognitive OODA loop. |

1
org-agent.asd
View File

@@ -10,7 +10,6 @@
:components ((:file "package")
(:file "protocol")
(:file "object-store")
(:file "context")
(:file "embedding")
(:file "skills")
(:file "neuro")

134
src/context.lisp
View File

@@ -1,134 +0,0 @@
(in-package :org-agent)
;;; ============================================================================
;;; Context API (System 1 Peripheral Vision)
;;; ============================================================================
;;; These functions provide the 'peripheral vision' for the LLM.
;;; When building a prompt, a skill can call these functions to gather
;;; relevant facts from the Object Store, preventing 'tunnel vision'.
(defun context-query-store (&key tag todo-state type)
"A high-level search engine for the Object Store.
TAG: String to search for in the :TAGS property.
TODO-STATE: The string state (e.g., 'TODO', 'DONE', 'WAITING').
TYPE: The keyword type (e.g., :HEADLINE).
Returns a list of org-object structs that satisfy ALL provided criteria."
(let ((results nil))
(maphash (lambda (id obj)
(declare (ignore id))
(let* ((attrs (org-object-attributes obj))
(obj-type (org-object-type obj))
(tags (getf attrs :TAGS))
(state (getf attrs :TODO-STATE))
(match t))
;; Filter by Type
(when (and type (not (eq obj-type type))) (setf match nil))
;; Filter by Tag (Org tags are often stored as a colon-delimited string like ':work:urgent:')
(when tag
(let ((tags-str (format nil "~a" tags)))
(unless (search tag tags-str :test #'string-equal)
(setf match nil))))
;; Filter by TODO State
(when (and todo-state (not (equal state todo-state))) (setf match nil))
(when match (push obj results))))
*object-store*)
results))
(defun context-get-active-projects ()
"Retrieves all headlines tagged with 'project' that are not yet complete.
This allows the agent to understand what the user is currently working on."
(let ((projects (context-query-store :tag "project" :type :HEADLINE)))
(remove-if (lambda (obj) (equal (getf (org-object-attributes obj) :TODO-STATE) "DONE"))
projects)))
(defun context-get-recent-completed-tasks ()
"Retrieves tasks that have been successfully finished.
Used to give the LLM context about the user's 'momentum' and recent wins."
(context-query-store :todo-state "DONE" :type :HEADLINE))
;;; ============================================================================
;;; Introspection API (Self-Awareness)
;;; ============================================================================
;;; These functions allow the agent to see its own internal configuration,
;;; such as its skill priorities and source code. This is critical for
;;; Phase 3 (Self-Editing) and autonomous priority negotiation.
(defun context-list-all-skills ()
"Returns a list of plists for all currently registered skills.
Each plist contains :name, :priority, and :dependencies.
This allows System 1 to understand the current 'Skill Graph'."
(let ((results nil))
(maphash (lambda (name skill)
(declare (ignore name))
(push (list :name (skill-name skill)
:priority (skill-priority skill)
:dependencies (skill-dependencies skill))
results))
*skills-registry*)
(sort results #'> :key (lambda (x) (getf x :priority)))))
(defun context-get-skill-source (skill-name)
"Reads the raw Org-mode source code of a specific skill.
Returns the file content as a string, or NIL if the file is missing."
(let* ((filename (format nil "~a.org" skill-name))
(skills-dir (merge-pathnames "skills/" (asdf:system-source-directory :org-agent)))
(full-path (merge-pathnames filename skills-dir)))
(if (uiop:file-exists-p full-path)
(uiop:read-file-string full-path)
nil)))
(defun context-get-system-logs (&optional (limit 20))
"Returns the most recent N lines from the kernel's execution history.
Allows the agent to 'perceive pain' (errors/rejections) and trigger self-repair."
(bt:with-lock-held (*logs-lock*)
(let ((count (min limit (length *system-logs*))))
(subseq *system-logs* 0 count))))
(defun context-get-skill-telemetry (skill-name)
"Returns performance metrics for a specific skill.
Returns a plist with :executions, :total-time, and :failures."
(bt:with-lock-held (*telemetry-lock*)
(gethash (string-downcase skill-name) *skill-telemetry*)))
(defun context-filter-sparse-tree (ast predicate)
"Recursively prunes an Org AST, keeping only nodes that match PREDICATE
and their parent hierarchies. Reduces token waste by removing noise."
(if (listp ast)
(let* ((type (getf ast :type))
(contents (getf ast :contents))
;; Recursively filter children
(filtered-contents
(remove-if #'null
(mapcar (lambda (c) (context-filter-sparse-tree c predicate))
contents))))
(if (or (funcall predicate ast)
(not (null filtered-contents)))
;; If this node matches OR has matching children, keep it
(let ((new-ast (copy-list ast)))
(setf (getf new-ast :contents) filtered-contents)
new-ast)
;; Otherwise, prune this entire branch
nil))
;; If it's a string (leaf content), keep it if the predicate says so,
;; but usually we keep it if the parent headline matches.
nil))
(defun context-resolve-path (path-string)
"Resolves environment variables in a path string (e.g., '$PROJECTS_DIR/my-proj').
This ensures project links remain valid even if base directories are moved."
(if (and (stringp path-string) (uiop:string-prefix-p "$" path-string))
(let* ((parts (uiop:split-string path-string :separator '(#\/)))
(var-name (subseq (car parts) 1)) ; Strip the '$'
(var-val (org-agent::get-env var-name))
(remaining (cl:reduce (lambda (a b) (format nil "~a/~a" a b)) (cdr parts))))
(if var-val
;; Strip any extra quotes that cl-dotenv might have preserved
(let ((clean-val (string-trim '(#\" #\Space) var-val)))
(format nil "~a/~a" (string-right-trim "/" clean-val) remaining))
path-string))
path-string))

49
src/core.lisp
View File

@@ -106,8 +106,9 @@
(defun dispatch-action (action)
"Routes an approved action intent to the correct physical actuator."
(when (and action (not (eq action :rejected)))
(when action
(let* ((payload (getf action :payload))
;; We default to :emacs for backward compatibility.
(target (or (getf action :target) :emacs))
(actuator-fn (gethash target *actuator-registry*)))
(if actuator-fn
@@ -147,9 +148,16 @@
(setf (skill-priority skill) val)
(kernel-log "ACTUATOR [System] - Set priority of ~a to ~a" name val))
(kernel-log "ACTUATOR [System] ERROR - Skill ~a not found" name))))
(:auth-google-code
(let ((code (getf payload :code)))
(kernel-log "ACTUATOR [System] - Received Google OAuth code. Exchanging...")
;; We call the function in the skill package.
;; Note: In a production kernel, we would use a more robust hook system.
(if (uiop:symbol-call :org-agent.skills.org-skill-auth-google-oauth :auth-google-receive-code code)
(kernel-log "ACTUATOR [System] - Google OAuth exchange successful.")
(kernel-log "ACTUATOR [System] - Google OAuth exchange FAILED."))))
(t (kernel-log "ACTUATOR [System] - Unknown command ~a" cmd)))))
;;; ============================================================================
;;; The Cognitive Loop (OODA)
;;; ============================================================================
@@ -178,7 +186,42 @@
(dispatch-action approved-action))))
;;; ================= ===========================================================
(defun perceive (raw-message)
"Updates the Object Store based on incoming stimulus and returns the context."
(let ((type (getf raw-message :type))
(payload (getf raw-message :payload)))
(kernel-log "PERCEIVE: ~a (~a)" type (or (getf payload :sensor) "no-sensor"))
(cond
((eq type :EVENT)
(let ((sensor (getf payload :sensor)))
(case sensor
(:buffer-update
(let ((ast (getf payload :ast)))
(when ast (ingest-ast ast))))
(:point-update
(let ((element (getf payload :element)))
(when element (ingest-ast element))))
;; Ensure we don't return NIL for these
(:user-command t)
(:heartbeat t)
(:chat-message t))))
((eq type :RESPONSE)
(kernel-log "ACT RESULT: ~a" (getf payload :status))))
;; ALWAYS return the raw message as the context base
raw-message))
(defun dispatch-action (action)
"Sends an approved action to the appropriate actuator."
(when (and action (not (eq action :rejected)))
(let ((target (getf action :target)))
(kernel-log "DISPATCH: Target ~a" target)
(let ((actuator (gethash target *actuators*)))
(if actuator
(funcall actuator action)
(kernel-log "ERROR: No actuator registered for ~a" target))))))
;;; ============================================================================
;;; Daemon Lifecycle Management
;;; ============================================================================

34
src/embedding.lisp
View File

@@ -4,32 +4,15 @@
;;; Vector Embedding and Math
;;; ============================================================================
(defvar *embedding-registry* (make-hash-table :test 'equal)
"Registry of embedding provider functions.")
(defvar *embedding-provider* :gemini
"The active embedding provider.")
(defun register-embedding-provider (name fn)
"Registers a function to handle vector embedding requests."
(setf (gethash name *embedding-registry*) fn))
(defun get-embedding (text)
"Fetches the vector embedding for a given text string using the active provider."
(let ((provider-fn (gethash *embedding-provider* *embedding-registry*)))
(if provider-fn
(funcall provider-fn text)
(progn
(kernel-log "EMBEDDING ERROR: No provider registered for ~a" *embedding-provider*)
nil))))
;;; --- Default Provider: Gemini ---
(defun embed-gemini (text)
"Fetches the vector embedding for a given text string from Gemini's embedding-004 model."
(let* ((auth (get-provider-auth :gemini))
(api-key (getf auth :api-key))
(endpoint "https://generativelanguage.googleapis.com/v1beta/models/text-embedding-004:embedContent"))
(unless api-key (return-from embed-gemini nil))
(unless api-key
(return-from get-embedding nil))
(let* ((url (format nil "~a?key=~a" endpoint api-key))
(headers `(("Content-Type" . "application/json")))
(body (cl-json:encode-json-to-string
@@ -38,15 +21,12 @@
(handler-case
(let* ((response (dex:post url :headers headers :content body))
(json (cl-json:decode-json-from-string response)))
;; Path: embedding.values
(cdr (assoc :values (cdr (assoc :embedding json)))))
(error (c)
(kernel-log "EMBEDDING FAILURE [Gemini]: ~a" c)
(kernel-log "EMBEDDING FAILURE: ~a" c)
nil)))))
(register-embedding-provider :gemini #'embed-gemini)
;;; --- Math Utilities ---
(defun dot-product (v1 v2)
(reduce #'+ (mapcar #'* v1 v2)))

18
src/neuro.lisp
View File

@@ -104,10 +104,6 @@
;; Read the LLM string back into a native Lisp data structure.
(suggestion (ignore-errors (read-from-string thought))))
(kernel-log "SYSTEM 1 Suggestion: ~a~%" thought)
;; SOTA: Store the successful prompt/result for future distillation
(when (and suggestion (not (eq suggestion :rejected)))
(setf (getf (gethash (skill-name active-skill) *skill-telemetry*) :last-successful-thought)
(list :prompt prompt :result thought)))
suggestion)
;; If the skill has no neuro-prompt, it's a 'Deterministic Skill' (Symbolic-only).
'(:type :LOG :payload (:text "Skill triggered (Deterministic only)")))))
@@ -128,12 +124,8 @@
(defun distillation-loop ()
"Periodically reviews internal logs and distills prompts for active skills.
This is an autonomous self-improvement cycle."
(maphash (lambda (name telemetry)
(let ((thought (getf telemetry :last-successful-thought)))
(when thought
(kernel-log "NEURO [Evolution] - Distilling prompt for skill '~a'..." name)
(let ((distilled (distill-prompt (getf thought :prompt) (getf thought :result))))
;; In a full Order 2 implementation, we would now surgically update
;; the .org file with the #+DISTILLED_PROMPT: property.
(kernel-log "NEURO [Evolution] - Distilled prompt for ~a: ~a" name distilled)))))
*skill-telemetry*))
(let ((logs (context-get-system-logs 50)))
(dolist (log logs)
(when (search "Verified by skill" log)
;; Extract the skill name and attempt distillation
(kernel-log "NEURO - Triggering prompt distillation cycle...")))))

141
src/object-store.lisp
View File

@@ -42,12 +42,11 @@
(id (or (getf props :ID)
(format nil "temp-~a" (get-universal-time))))
(contents (getf ast :contents))
;; Lazy Embedding: Only embed if the headline has :EMBED: t property
;; Extract raw text for embedding if it's a headline
(raw-content (when (eq type :HEADLINE)
(format nil "~a~%~a"
(getf props :TITLE)
(or (cl:getf ast :raw-content) ""))))
(should-embed (and raw-content (equal (getf props :EMBED) "t")))
(child-ids nil))
;; Depth-first ingestion: Recurse into children first to gather their IDs.
@@ -56,12 +55,13 @@
(push (ingest-ast child id) child-ids)))
;; Create or overwrite the object in the hash table.
;; This is a 'late-binding' update—if the ID exists, we update its state.
(let ((obj (make-org-object
:id id
:type type
:attributes props
:content raw-content
:vector (when should-embed (get-embedding raw-content))
:vector (when raw-content (get-embedding raw-content))
:parent-id parent-id
:children (nreverse child-ids) ; Maintain document order
:version (get-universal-time)
@@ -123,7 +123,140 @@
*object-store*)
results))
;;; ================= ===========================================================
;;; ============================================================================
;;; Context API (System 1 Peripheral Vision)
;;; ============================================================================
;;; These functions provide the 'peripheral vision' for the LLM.
;;; When building a prompt, a skill can call these functions to gather
;;; relevant facts from the Object Store, preventing 'tunnel vision'.
(defun context-query-store (&key tag todo-state type)
"A high-level search engine for the Object Store.
TAG: String to search for in the :TAGS property.
TODO-STATE: The string state (e.g., 'TODO', 'DONE', 'WAITING').
TYPE: The keyword type (e.g., :HEADLINE).
Returns a list of org-object structs that satisfy ALL provided criteria."
(let ((results nil))
(maphash (lambda (id obj)
(declare (ignore id))
(let* ((attrs (org-object-attributes obj))
(obj-type (org-object-type obj))
(tags (getf attrs :TAGS))
(state (getf attrs :TODO-STATE))
(match t))
;; Filter by Type
(when (and type (not (eq obj-type type))) (setf match nil))
;; Filter by Tag (Org tags are often stored as a colon-delimited string like ':work:urgent:')
(when tag
(let ((tags-str (format nil "~a" tags)))
(unless (search tag tags-str :test #'string-equal)
(setf match nil))))
;; Filter by TODO State
(when (and todo-state (not (equal state todo-state))) (setf match nil))
(when match (push obj results))))
*object-store*)
results))
(defun context-get-active-projects ()
"Retrieves all headlines tagged with 'project' that are not yet complete.
This allows the agent to understand what the user is currently working on."
(let ((projects (context-query-store :tag "project" :type :HEADLINE)))
(remove-if (lambda (obj) (equal (getf (org-object-attributes obj) :TODO-STATE) "DONE"))
projects)))
(defun context-get-recent-completed-tasks ()
"Retrieves tasks that have been successfully finished.
Used to give the LLM context about the user's 'momentum' and recent wins."
(context-query-store :todo-state "DONE" :type :HEADLINE))
;;; ============================================================================
;;; Introspection API (Self-Awareness)
;;; ============================================================================
;;; These functions allow the agent to see its own internal configuration,
;;; such as its skill priorities and source code. This is critical for
;;; Phase 3 (Self-Editing) and autonomous priority negotiation.
(defun context-list-all-skills ()
"Returns a list of plists for all currently registered skills.
Each plist contains :name, :priority, and :dependencies.
This allows System 1 to understand the current 'Skill Graph'."
(let ((results nil))
(maphash (lambda (name skill)
(declare (ignore name))
(push (list :name (skill-name skill)
:priority (skill-priority skill)
:dependencies (skill-dependencies skill))
results))
*skills-registry*)
(sort results #'> :key (lambda (x) (getf x :priority)))))
(defun context-get-skill-source (skill-name)
"Reads the raw Org-mode source code of a specific skill.
Returns the file content as a string, or NIL if the file is missing."
(let* ((filename (format nil "~a.org" skill-name))
(skills-dir (merge-pathnames "skills/" (asdf:system-source-directory :org-agent)))
(full-path (merge-pathnames filename skills-dir)))
(if (uiop:file-exists-p full-path)
(uiop:read-file-string full-path)
nil)))
(defun context-get-system-logs (&optional (limit 20))
"Returns the most recent N lines from the kernel's execution history.
Allows the agent to 'perceive pain' (errors/rejections) and trigger self-repair."
(bt:with-lock-held (*logs-lock*)
(let ((count (min limit (length *system-logs*))))
(subseq *system-logs* 0 count))))
(defun context-get-skill-telemetry (skill-name)
"Returns performance metrics for a specific skill.
Returns a plist with :executions, :total-time, and :failures."
(bt:with-lock-held (*telemetry-lock*)
(gethash (string-downcase skill-name) *skill-telemetry*)))
(defun context-filter-sparse-tree (ast predicate)
"Recursively prunes an Org AST, keeping only nodes that match PREDICATE
and their parent hierarchies. Reduces token waste by removing noise."
(if (listp ast)
(let* ((type (getf ast :type))
(contents (getf ast :contents))
;; Recursively filter children
(filtered-contents
(remove-if #'null
(mapcar (lambda (c) (context-filter-sparse-tree c predicate))
contents))))
(if (or (funcall predicate ast)
(not (null filtered-contents)))
;; If this node matches OR has matching children, keep it
(let ((new-ast (copy-list ast)))
(setf (getf new-ast :contents) filtered-contents)
new-ast)
;; Otherwise, prune this entire branch
nil))
;; If it's a string (leaf content), keep it if the predicate says so,
;; but usually we keep it if the parent headline matches.
nil))
(defun context-resolve-path (path-string)
"Resolves environment variables in a path string (e.g., '$PROJECTS_DIR/my-proj').
This ensures project links remain valid even if base directories are moved."
(if (and (stringp path-string) (uiop:string-prefix-p "$" path-string))
(let* ((parts (uiop:split-string path-string :separator '(#\/)))
(var-name (subseq (car parts) 1)) ; Strip the '$'
(var-val (org-agent::get-env var-name))
(remaining (cl:reduce (lambda (a b) (format nil "~a/~a" a b)) (cdr parts))))
(if var-val
;; Strip any extra quotes that cl-dotenv might have preserved
(let ((clean-val (string-trim '(#\" #\Space) var-val)))
(format nil "~a/~a" (string-right-trim "/" clean-val) remaining))
path-string))
path-string))
;;; ============================================================================
;;; AST Helper Functions
;;; ============================================================================