amr/passepartout
1
0
Fork 0
Code Issues 15 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(arch): revamp architecture section with diagrams, paradigm comparison, and orders of autonomy

Browse Source
This commit is contained in:
Amr Gharbeia
2026-04-01 17:14:10 -04:00
parent 0eb9e3773f
commit 6536777803
3 changed files with 209 additions and 138 deletions
Download Patch File Download Diff File Expand all files Collapse all files

72
docs/README.org
View File

@@ -26,6 +26,63 @@ This architecture treats all interfaces as external **Actuators** and **Sensors*
- **Messaging Actuator (Signal/Telegram/Discord):** A delivery channel for proactive alerts and human-in-the-loop decisions.
- **Web Actuator (Dashboard):** A visual telemetry interface for monitoring the live kernel state.
* The Paradigm: Skills vs. Sub-Agents
Modern AI frameworks heavily rely on "Sub-agents" (e.g., passing text between isolated Python scripts). `org-agent` fundamentally rejects this in favor of **Org-Native Skills**.
** The Performance Advantage (Single Address Space)
Following CLOSOS principles, the Lisp Machine architecture uses a **Single Address Space**. All applications (Skills) and the kernel share one unified memory space. We completely eliminate standard Inter-Process Communication (IPC) overhead. Objects are shared via direct pointers, meaning zero serialization/deserialization latency when moving from perception to thought.
** Specialization (System 1 Prompt Injection)
Instead of spinning up separate sub-agent processes with static prompts, the kernel dynamically injects the `:neuro-prompt` of the active skill into the LLM context. The LLM is temporarily "re-programmed" at runtime per task.
** Context Length & Memory (Peripheral Vision)
Multi-agent systems struggle with context bloat. `org-agent` solves this via its **Peripheral Vision API**. The Lisp Kernel uses `context-filter-sparse-tree` to surgically prune the Org AST, passing only the relevant hierarchical nodes to the LLM and preventing context-window overload.
** Responsiveness (Dual-Process Advantage)
The system employs a Dual-Process Architecture:
- **System 1 (Neural):** Probabilistic, slow, creative (LLM).
- **System 2 (Symbolic):** Deterministic, instant, rigorous (Lisp).
Because Skills are compiled Lisp code, System 2 intercepts deterministic tasks (like assigning an ID) and executes them in milliseconds, bypassing the LLM entirely.
* Architecture Diagrams
** The Single Address Space
#+begin_src mermaid
graph TD
subgraph Lisp Machine (Single Address Space)
K[Kernel Core] --> OS[(Object Store)]
S1[Skill: Architect] --> OS
S2[Skill: Analyst] --> OS
S3[Skill: GTD] --> OS
K -- Pointers --> S1
K -- Pointers --> S2
end
subgraph IPC (Slow)
E[Emacs Actuator] -. JSON .-> K
end
#+end_src
** The Cognitive Loop (OODA)
#+begin_src mermaid
sequenceDiagram
participant Sensor
participant Kernel
participant System1 as System 1 (LLM)
participant System2 as System 2 (Lisp)
participant Actuator
Sensor->>Kernel: Perceive (Stimulus)
Kernel->>System1: Think (Inject Prompt)
System1-->>Kernel: Proposed Action
Kernel->>System2: Decide (Safety Gate)
alt Validation Failed
System2-->>Kernel: Reject / Log Error
else Validation Passed
System2->>Actuator: Act (Dispatch)
end
#+end_src
* The Architecture: The Cognitive Loop
The core engine is agnostic to both business logic and communication channels. It routes data through a strict four-stage cognitive pipeline:
@@ -164,7 +221,20 @@ Your laptop acts as the sensor/actuator array.
To see all available models, simply type `@agent list models` in any Org buffer and save.
* Current State: Phase 3 (The Self-Editing Kernel) Achieved
* The Long-Term Vision: Orders of Autonomy
The development of `org-agent` follows distinct orders of autonomy, progressing from a basic assistant to a fully sovereign entity.
** Order 1: The Reactive Kernel (Phase 1 & 2)
The agent acts as a strict "Delegator." It requires human stimulus to trigger the Cognitive Loop. Capabilities (Skills) are expanded, but the agent only speaks when spoken to.
** Order 2: The Self-Editing Kernel (Phase 3 - Current)
The agent achieves introspection. It can "perceive pain" (errors) via system logs and trigger a `skill-self-fix` loop to rewrite its own source code, hot-reloading the changes. It proactively maintains the system.
** Order 3: The Sovereign Architect (Phase 4+)
The agent transitions to full autonomy. It maintains the Consensus Loop entirely, identifying structural decay in the Memex, drafting its own PRDs, writing code, executing Chaos testing, and committing the final result without human intervention (unless authorization gates are explicitly set).
* Current State: Order 2 Achieved
- DONE Core Lisp microkernel (Cognitive Loop: Perceive -> Think -> Decide -> Act)
- DONE OACP (Swank/Socket communication protocol) implemented

134
src/context.lisp Normal file
View File

@@ -0,0 +1,134 @@
(in-package :org-agent)
;;; ============================================================================
;;; Context API (System 1 Peripheral Vision)
;;; ============================================================================
;;; These functions provide the 'peripheral vision' for the LLM.
;;; When building a prompt, a skill can call these functions to gather
;;; relevant facts from the Object Store, preventing 'tunnel vision'.
(defun context-query-store (&key tag todo-state type)
"A high-level search engine for the Object Store.
TAG: String to search for in the :TAGS property.
TODO-STATE: The string state (e.g., 'TODO', 'DONE', 'WAITING').
TYPE: The keyword type (e.g., :HEADLINE).
Returns a list of org-object structs that satisfy ALL provided criteria."
(let ((results nil))
(maphash (lambda (id obj)
(declare (ignore id))
(let* ((attrs (org-object-attributes obj))
(obj-type (org-object-type obj))
(tags (getf attrs :TAGS))
(state (getf attrs :TODO-STATE))
(match t))
;; Filter by Type
(when (and type (not (eq obj-type type))) (setf match nil))
;; Filter by Tag (Org tags are often stored as a colon-delimited string like ':work:urgent:')
(when tag
(let ((tags-str (format nil "~a" tags)))
(unless (search tag tags-str :test #'string-equal)
(setf match nil))))
;; Filter by TODO State
(when (and todo-state (not (equal state todo-state))) (setf match nil))
(when match (push obj results))))
*object-store*)
results))
(defun context-get-active-projects ()
"Retrieves all headlines tagged with 'project' that are not yet complete.
This allows the agent to understand what the user is currently working on."
(let ((projects (context-query-store :tag "project" :type :HEADLINE)))
(remove-if (lambda (obj) (equal (getf (org-object-attributes obj) :TODO-STATE) "DONE"))
projects)))
(defun context-get-recent-completed-tasks ()
"Retrieves tasks that have been successfully finished.
Used to give the LLM context about the user's 'momentum' and recent wins."
(context-query-store :todo-state "DONE" :type :HEADLINE))
;;; ============================================================================
;;; Introspection API (Self-Awareness)
;;; ============================================================================
;;; These functions allow the agent to see its own internal configuration,
;;; such as its skill priorities and source code. This is critical for
;;; Phase 3 (Self-Editing) and autonomous priority negotiation.
(defun context-list-all-skills ()
"Returns a list of plists for all currently registered skills.
Each plist contains :name, :priority, and :dependencies.
This allows System 1 to understand the current 'Skill Graph'."
(let ((results nil))
(maphash (lambda (name skill)
(declare (ignore name))
(push (list :name (skill-name skill)
:priority (skill-priority skill)
:dependencies (skill-dependencies skill))
results))
*skills-registry*)
(sort results #'> :key (lambda (x) (getf x :priority)))))
(defun context-get-skill-source (skill-name)
"Reads the raw Org-mode source code of a specific skill.
Returns the file content as a string, or NIL if the file is missing."
(let* ((filename (format nil "~a.org" skill-name))
(skills-dir (merge-pathnames "skills/" (asdf:system-source-directory :org-agent)))
(full-path (merge-pathnames filename skills-dir)))
(if (uiop:file-exists-p full-path)
(uiop:read-file-string full-path)
nil)))
(defun context-get-system-logs (&optional (limit 20))
"Returns the most recent N lines from the kernel's execution history.
Allows the agent to 'perceive pain' (errors/rejections) and trigger self-repair."
(bt:with-lock-held (*logs-lock*)
(let ((count (min limit (length *system-logs*))))
(subseq *system-logs* 0 count))))
(defun context-get-skill-telemetry (skill-name)
"Returns performance metrics for a specific skill.
Returns a plist with :executions, :total-time, and :failures."
(bt:with-lock-held (*telemetry-lock*)
(gethash (string-downcase skill-name) *skill-telemetry*)))
(defun context-filter-sparse-tree (ast predicate)
"Recursively prunes an Org AST, keeping only nodes that match PREDICATE
and their parent hierarchies. Reduces token waste by removing noise."
(if (listp ast)
(let* ((type (getf ast :type))
(contents (getf ast :contents))
;; Recursively filter children
(filtered-contents
(remove-if #'null
(mapcar (lambda (c) (context-filter-sparse-tree c predicate))
contents))))
(if (or (funcall predicate ast)
(not (null filtered-contents)))
;; If this node matches OR has matching children, keep it
(let ((new-ast (copy-list ast)))
(setf (getf new-ast :contents) filtered-contents)
new-ast)
;; Otherwise, prune this entire branch
nil))
;; If it's a string (leaf content), keep it if the predicate says so,
;; but usually we keep it if the parent headline matches.
nil))
(defun context-resolve-path (path-string)
"Resolves environment variables in a path string (e.g., '$PROJECTS_DIR/my-proj').
This ensures project links remain valid even if base directories are moved."
(if (and (stringp path-string) (uiop:string-prefix-p "$" path-string))
(let* ((parts (uiop:split-string path-string :separator '(#\/)))
(var-name (subseq (car parts) 1)) ; Strip the '$'
(var-val (org-agent::get-env var-name))
(remaining (cl:reduce (lambda (a b) (format nil "~a/~a" a b)) (cdr parts))))
(if var-val
;; Strip any extra quotes that cl-dotenv might have preserved
(let ((clean-val (string-trim '(#\" #\Space) var-val)))
(format nil "~a/~a" (string-right-trim "/" clean-val) remaining))
path-string))
path-string))

141
src/object-store.lisp
View File

@@ -42,11 +42,12 @@
(id (or (getf props :ID)
(format nil "temp-~a" (get-universal-time))))
(contents (getf ast :contents))
;; Extract raw text for embedding if it's a headline
;; Lazy Embedding: Only embed if the headline has :EMBED: t property
(raw-content (when (eq type :HEADLINE)
(format nil "~a~%~a"
(getf props :TITLE)
(or (cl:getf ast :raw-content) ""))))
(should-embed (and raw-content (equal (getf props :EMBED) "t")))
(child-ids nil))
;; Depth-first ingestion: Recurse into children first to gather their IDs.
@@ -55,13 +56,12 @@
(push (ingest-ast child id) child-ids)))
;; Create or overwrite the object in the hash table.
;; This is a 'late-binding' update—if the ID exists, we update its state.
(let ((obj (make-org-object
:id id
:type type
:attributes props
:content raw-content
:vector (when raw-content (get-embedding raw-content))
:vector (when should-embed (get-embedding raw-content))
:parent-id parent-id
:children (nreverse child-ids) ; Maintain document order
:version (get-universal-time)
@@ -123,140 +123,7 @@
*object-store*)
results))
;;; ============================================================================
;;; Context API (System 1 Peripheral Vision)
;;; ============================================================================
;;; These functions provide the 'peripheral vision' for the LLM.
;;; When building a prompt, a skill can call these functions to gather
;;; relevant facts from the Object Store, preventing 'tunnel vision'.
(defun context-query-store (&key tag todo-state type)
"A high-level search engine for the Object Store.
TAG: String to search for in the :TAGS property.
TODO-STATE: The string state (e.g., 'TODO', 'DONE', 'WAITING').
TYPE: The keyword type (e.g., :HEADLINE).
Returns a list of org-object structs that satisfy ALL provided criteria."
(let ((results nil))
(maphash (lambda (id obj)
(declare (ignore id))
(let* ((attrs (org-object-attributes obj))
(obj-type (org-object-type obj))
(tags (getf attrs :TAGS))
(state (getf attrs :TODO-STATE))
(match t))
;; Filter by Type
(when (and type (not (eq obj-type type))) (setf match nil))
;; Filter by Tag (Org tags are often stored as a colon-delimited string like ':work:urgent:')
(when tag
(let ((tags-str (format nil "~a" tags)))
(unless (search tag tags-str :test #'string-equal)
(setf match nil))))
;; Filter by TODO State
(when (and todo-state (not (equal state todo-state))) (setf match nil))
(when match (push obj results))))
*object-store*)
results))
(defun context-get-active-projects ()
"Retrieves all headlines tagged with 'project' that are not yet complete.
This allows the agent to understand what the user is currently working on."
(let ((projects (context-query-store :tag "project" :type :HEADLINE)))
(remove-if (lambda (obj) (equal (getf (org-object-attributes obj) :TODO-STATE) "DONE"))
projects)))
(defun context-get-recent-completed-tasks ()
"Retrieves tasks that have been successfully finished.
Used to give the LLM context about the user's 'momentum' and recent wins."
(context-query-store :todo-state "DONE" :type :HEADLINE))
;;; ============================================================================
;;; Introspection API (Self-Awareness)
;;; ============================================================================
;;; These functions allow the agent to see its own internal configuration,
;;; such as its skill priorities and source code. This is critical for
;;; Phase 3 (Self-Editing) and autonomous priority negotiation.
(defun context-list-all-skills ()
"Returns a list of plists for all currently registered skills.
Each plist contains :name, :priority, and :dependencies.
This allows System 1 to understand the current 'Skill Graph'."
(let ((results nil))
(maphash (lambda (name skill)
(declare (ignore name))
(push (list :name (skill-name skill)
:priority (skill-priority skill)
:dependencies (skill-dependencies skill))
results))
*skills-registry*)
(sort results #'> :key (lambda (x) (getf x :priority)))))
(defun context-get-skill-source (skill-name)
"Reads the raw Org-mode source code of a specific skill.
Returns the file content as a string, or NIL if the file is missing."
(let* ((filename (format nil "~a.org" skill-name))
(skills-dir (merge-pathnames "skills/" (asdf:system-source-directory :org-agent)))
(full-path (merge-pathnames filename skills-dir)))
(if (uiop:file-exists-p full-path)
(uiop:read-file-string full-path)
nil)))
(defun context-get-system-logs (&optional (limit 20))
"Returns the most recent N lines from the kernel's execution history.
Allows the agent to 'perceive pain' (errors/rejections) and trigger self-repair."
(bt:with-lock-held (*logs-lock*)
(let ((count (min limit (length *system-logs*))))
(subseq *system-logs* 0 count))))
(defun context-get-skill-telemetry (skill-name)
"Returns performance metrics for a specific skill.
Returns a plist with :executions, :total-time, and :failures."
(bt:with-lock-held (*telemetry-lock*)
(gethash (string-downcase skill-name) *skill-telemetry*)))
(defun context-filter-sparse-tree (ast predicate)
"Recursively prunes an Org AST, keeping only nodes that match PREDICATE
and their parent hierarchies. Reduces token waste by removing noise."
(if (listp ast)
(let* ((type (getf ast :type))
(contents (getf ast :contents))
;; Recursively filter children
(filtered-contents
(remove-if #'null
(mapcar (lambda (c) (context-filter-sparse-tree c predicate))
contents))))
(if (or (funcall predicate ast)
(not (null filtered-contents)))
;; If this node matches OR has matching children, keep it
(let ((new-ast (copy-list ast)))
(setf (getf new-ast :contents) filtered-contents)
new-ast)
;; Otherwise, prune this entire branch
nil))
;; If it's a string (leaf content), keep it if the predicate says so,
;; but usually we keep it if the parent headline matches.
nil))
(defun context-resolve-path (path-string)
"Resolves environment variables in a path string (e.g., '$PROJECTS_DIR/my-proj').
This ensures project links remain valid even if base directories are moved."
(if (and (stringp path-string) (uiop:string-prefix-p "$" path-string))
(let* ((parts (uiop:split-string path-string :separator '(#\/)))
(var-name (subseq (car parts) 1)) ; Strip the '$'
(var-val (org-agent::get-env var-name))
(remaining (cl:reduce (lambda (a b) (format nil "~a/~a" a b)) (cdr parts))))
(if var-val
;; Strip any extra quotes that cl-dotenv might have preserved
(let ((clean-val (string-trim '(#\" #\Space) var-val)))
(format nil "~a/~a" (string-right-trim "/" clean-val) remaining))
path-string))
path-string))
;;; ============================================================================
;;; ================= ===========================================================
;;; AST Helper Functions
;;; ============================================================================