23 KiB
The Skill Engine (skills.lisp)
- The Skill Engine (skills.lisp)
- Architectural Intent: Late-Binding Intelligence
- Skill Architecture
- Package Context
- Skill Metadata (defstruct skill)
- Skill Catalog Tracking
- Skill Selection (find-triggered-skill)
- Skill Definition Macro (defskill)
- Dependency Resolution (resolve-skill-dependencies)
- Metadata Parsing (parse-skill-metadata)
- Topological Sorting (topological-sort-skills)
- Syntax Validation (validate-lisp-syntax)
- Jailed Loading (load-skill-from-org)
- Safe Loading with Timeout (load-skill-with-timeout)
- Initializing All Skills (initialize-all-skills)
- Toolbelt Prompt Generation (generate-tool-belt-prompt)
- The Default Tool Belt
The Skill Engine (skills.lisp)
Architectural Intent: Late-Binding Intelligence
A static, hardcoded architecture is inherently fragile. To build a autonomous agent that can evolve alongside its user, the harness must be a "Thin Shell" that delegates its capabilities to dynamic, hot-reloadable modules known as Skills. This is the core of our Thin Harness / Thick Skill Microkernel Architecture.
Skills unify the "Why" (Literate Org documentation) and the "How" (Functional Lisp implementation). This allows the harness to "learn" new behaviors without a full system restart, enabling a continuous evolutionary loop where the agent can eventually inspect and improve its own code.
The True Microkernel (Decoupled Core Skills)
Historically, "core" skills (like State Persistence or Gateways) were statically compiled into the harness for performance. We have fundamentally decoupled this. Now, all behavioral skills are pure user-space dynamic modules. They do not tangle to `src/` and are not listed in `org-agent.asd`. The harness simply boots, scans the `skills/` directory, and evaluates the code inside a jailed package. If a user wishes to swap the IPFS persistence skill for an AWS S3 one, they simply swap the `.org` file; no kernel recompilation is required.
Dormant Verification (Tests)
Because skills are no longer statically compiled into the core `org-agent` ASDF system, their associated `FiveAM` test blocks are currently dormant during a standard static build. The tests still exist within the literate `.org` files as verifiable documentation, but executing them requires either dynamic evaluation at runtime or a dedicated test-loader skill.
1. The Package Jailing Principle
Every skill is evaluated within its own dedicated Common Lisp package (namespace). This "Jailing" prevents symbol collisions between disparate skills and ensures that a bug in one module cannot easily corrupt the internal state of another.
2. Deterministic Load Ordering
Skills often depend on one another. The harness implements a deterministic topological sorting algorithm to ensure that dependencies are loaded before the skills that require them.
Skill Architecture
flowchart TD
Registry[Skills Registry] --> S1[Skill: System Policy]
Registry --> S2[Skill: LLM Gateway]
Registry --> S3[Skill: Token Accountant]
S2 -- Depends On --> S1
S3 -- Depends On --> S2
subgraph Jailing[Package Jailing]
P1[Package: ORG-AGENT.SKILLS.S1]
P2[Package: ORG-AGENT.SKILLS.S2]
P3[Package: ORG-AGENT.SKILLS.S3]
end
S1 --> P1
S2 --> P2
S3 --> P3Package Context
We begin by ensuring we are in the correct isolated harness namespace.
(in-package :org-agent)Skill Metadata (defstruct skill)
The core data structure representing an agent capability. It includes the trigger condition, the probabilistic prompt generator, and the deterministic safety gate.
(defstruct skill name priority dependencies trigger-fn probabilistic-prompt deterministic-fn)Skill Catalog Tracking
The harness maintains a stateful tracking table for all skill files discovered in the environment (notes/org-skill-*.org).
(defvar *skill-catalog* (make-hash-table :test 'equal)
"A stateful tracking table for all skill files discovered in the environment.")
(defstruct skill-entry
filename
(status :discovered) ;; :discovered, :loading, :ready, :failed
error-log
(load-time 0))Skill Selection (find-triggered-skill)
The primary dispatcher for the Probabilistic Engine. It iterates through the registry to find the highest-priority skill whose trigger function matches the current cognitive context.
(defun find-triggered-skill (context)
"Returns the highest priority skill whose trigger condition matches the context."
(let ((triggered nil))
(maphash (lambda (name skill)
(declare (ignore name))
(when (ignore-errors (funcall (skill-trigger-fn skill) context))
(push skill triggered)))
*skills-registry*)
(first (sort triggered #'> :key #'skill-priority))))Skill Definition Macro (defskill)
The interface used within Org files to register new capabilities. Note that dependencies are explicitly quoted to prevent premature evaluation during the macro expansion phase.
(defmacro defskill (name &key priority dependencies trigger probabilistic deterministic)
"Registers a new skill into the global registry."
`(setf (gethash (string-downcase (string ,name)) *skills-registry*)
(make-skill :name (string-downcase (string ,name))
:priority (or ,priority 10)
:dependencies ',dependencies
:trigger-fn ,trigger
:probabilistic-prompt ,probabilistic
:deterministic-fn ,deterministic)))Dependency Resolution (resolve-skill-dependencies)
Recursively flattens the dependency graph for a given skill. This is used during hot-unloading to ensure that dependent skills are also refreshed.
(defun resolve-skill-dependencies (skill-name)
"Recursively resolves dependencies for a given skill name."
(let ((resolved nil) (seen nil))
(labels ((visit (name)
(unless (member name seen :test #'equal)
(push name seen)
(let ((skill (gethash (string-downcase (string name)) *skills-registry*)))
(when skill
(dolist (dep (skill-dependencies skill))
(visit dep))))
(push name resolved))))
(visit skill-name)
(nreverse resolved))))Metadata Parsing (parse-skill-metadata)
A robust, low-level scanner that extracts `#+DEPENDS_ON:` and `:ID:` tags from an Org file. This allows the harness to calculate the load order without needing to parse the full Org AST, which would create a boot-time circularity.
(defun parse-skill-metadata (filepath)
"Extracts ID and DEPENDS_ON tags using robust regex scanning."
(let ((dependencies nil)
(id nil)
(content (uiop:read-file-string filepath)))
;; Extract ID
(multiple-value-bind (match regs)
(ppcre:scan-to-strings "(?im:^:ID:\\s*([^\\s\\r\\n]+))" content)
(when match (setf id (aref regs 0))))
;; Extract all DEPENDS_ON lines
(ppcre:do-register-groups (deps-string)
("(?im:^#\\+DEPENDS_ON:\\s*(.*))" content)
(let ((deps (ppcre:split "\\s+" (string-trim " " deps-string))))
(setf dependencies (append dependencies (mapcar (lambda (s) (string-trim "[] " s)) deps)))))
(values id (remove-if (lambda (s) (= 0 (length s))) dependencies))))Topological Sorting (topological-sort-skills)
This is the core algorithm of the boot sequence. It uses a Depth-First Search (DFS) to resolve the skill dependency graph.
It performs three critical roles:
- Load Ordering: Ensures that "foundational" skills (like the LLM Gateway) are loaded before high-level "behavioral" skills.
- ID Resolution: Correct maps Org `:ID:` properties to filepaths.
- Cycle Detection: It uses a recursion stack to detect circular dependencies (e.g., A depends on B, B depends on A) and errors out safely before the Lisp image hangs.
(defun topological-sort-skills (skills-dir)
"Returns a list of skill filepaths sorted by dependency (dependencies first)."
(let ((files (uiop:directory-files skills-dir "org-skill-*.org"))
(adj (make-hash-table :test 'equal))
(name-to-file (make-hash-table :test 'equal))
(id-to-file (make-hash-table :test 'equal))
(result nil)
(visited (make-hash-table :test 'equal))
(stack (make-hash-table :test 'equal)))
(dolist (file files)
(let ((filename (pathname-name file)))
(multiple-value-bind (id deps) (parse-skill-metadata file)
(setf (gethash (string-downcase filename) name-to-file) file)
(when id (setf (gethash (string-downcase id) id-to-file) file))
(setf (gethash (string-downcase filename) adj) deps))))
(labels ((visit (file)
(let* ((filename (pathname-name file))
(node-key (string-downcase filename)))
(unless (gethash node-key visited)
(setf (gethash node-key stack) t)
(dolist (dep (gethash node-key adj))
(let* ((is-id-p (uiop:string-prefix-p "id:" (string-downcase dep)))
(dep-key (string-downcase (if is-id-p (subseq dep 3) dep)))
(dep-file (if is-id-p
(gethash dep-key id-to-file)
(or (gethash dep-key id-to-file)
(gethash dep-key name-to-file)))))
(when dep-file
(let ((dep-filename (pathname-name dep-file)))
(if (gethash (string-downcase dep-filename) stack)
(error "Circular dependency detected: ~a -> ~a" filename dep-filename)
(visit dep-file))))))
(setf (gethash node-key stack) nil)
(setf (gethash node-key visited) t)
(push file result)))))
(let ((filenames (sort (mapcar #'pathname-name files) #'string<)))
(dolist (name filenames)
(let ((file (gethash (string-downcase name) name-to-file)))
(when file (visit file)))))
(nreverse result))))Syntax Validation (validate-lisp-syntax)
A pre-flight safety check. Before evaluating any code from an Org file, the harness attempts to read the entire string. If the reader encounters a syntax error (like an unclosed parenthesis), the load is aborted before the Lisp image can crash.
(defun validate-lisp-syntax (code-string)
"Checks if a string contains valid, readable Common Lisp forms."
(handler-case
(let ((*read-eval* nil))
(with-input-from-string (stream (format nil "(progn ~a)" code-string))
(loop for form = (read stream nil :eof) until (eq form :eof))
(values t nil)))
(error (c) (values nil (format nil "~a" c)))))Jailed Loading (load-skill-from-org)
The core "hot-loading" mechanism. It extracts Lisp blocks from an Org file and evaluates them within a "Jail" (an isolated package).
The Jailing Algorithm:
- Isolation: It generates a package name based on the filename (e.g.,
ORG-AGENT.SKILLS.CORE-LOGIC). - Namespace Protection: It inherits external symbols from the
ORG-AGENTpackage, allowing the skill to use the harness API, but keeps its internal helper functions local. - Block Filtering: It explicitly ignores blocks that contain
:tangle, ensuring that harness-level code (which is already insrc/) is not accidentally re-evaluated as skill logic.
(defun load-skill-from-org (filepath)
"Parses and evaluates Lisp blocks from an Org file into a jailed package."
(let* ((skill-base-name (pathname-name filepath))
(entry (or (gethash skill-base-name *skill-catalog*) (make-skill-entry :filename skill-base-name))))
(setf (skill-entry-status entry) :loading)
(setf (gethash skill-base-name *skill-catalog*) entry)
(handler-case
(let* ((content (uiop:read-file-string filepath))
(lines (uiop:split-string content :separator '(#\Newline)))
(in-lisp-block nil)
(lisp-code "")
(pkg-name (intern (string-upcase (format nil "ORG-AGENT.SKILLS.~a" skill-base-name)) :keyword)))
(dolist (line lines)
(let ((clean-line (string-trim '(#\Space #\Tab #\Return) line)))
(cond ((uiop:string-prefix-p "#+begin_src lisp" (string-downcase clean-line))
;; Only load blocks that are NOT tangled to src/ or elsewhere
(if (search ":tangle" (string-downcase clean-line))
(setf in-lisp-block nil)
(setf in-lisp-block t)))
((uiop:string-prefix-p "#+end_src" (string-downcase clean-line))
(setf in-lisp-block nil))
(in-lisp-block
(unless (or (uiop:string-prefix-p ":PROPERTIES:" (string-upcase clean-line))
(uiop:string-prefix-p ":END:" (string-upcase clean-line)))
(setf lisp-code (concatenate 'string lisp-code line (string #\Newline))))))))
(if (= (length lisp-code) 0)
(progn (setf (skill-entry-status entry) :ready) t) ;; Valid empty skill
(progn
;; PRE-FLIGHT: Syntax Validation
(multiple-value-bind (valid-p err) (validate-lisp-syntax lisp-code)
(unless valid-p
(error "Syntax Error: ~a" err)))
(harness-log "HARNESS: Jailing skill '~a' in package ~a" skill-base-name pkg-name)
(unless (find-package pkg-name)
(let ((new-pkg (make-package pkg-name :use '(:cl))))
(do-external-symbols (sym (find-package :org-agent)) (shadowing-import sym new-pkg))))
(let ((*read-eval* nil) (*package* (find-package pkg-name)))
(eval (read-from-string (format nil "(progn ~a)" lisp-code))))
(setf (skill-entry-status entry) :ready)
t)))
(error (c)
(let ((msg (format nil "~a" c)))
(harness-log "LOADER ERROR in skill '~a': ~a" skill-base-name msg)
(setf (skill-entry-status entry) :failed)
(setf (skill-entry-error-log entry) msg)
nil)))))Safe Loading with Timeout (load-skill-with-timeout)
Wraps the skill loader in a thread with a hard timeout to prevent a single malformed skill from hanging the entire boot sequence.
(defun load-skill-with-timeout (filepath timeout-seconds)
"Loads a skill Org file with a hard execution timeout."
(let* ((finished nil)
(thread (bt:make-thread (lambda ()
(if (load-skill-from-org filepath)
(setf finished t)
(setf finished :error)))
:name (format nil "loader-~a" (pathname-name filepath))))
(start-time (get-internal-real-time))
(timeout-units (truncate (* timeout-seconds internal-time-units-per-second))))
(loop
(when (eq finished t) (return :success))
(when (eq finished :error) (return :error))
(unless (bt:thread-alive-p thread) (return :error))
(when (> (- (get-internal-real-time) start-time) timeout-units)
(harness-log "HARNESS: Timing out skill ~a..." (pathname-name filepath))
#+sbcl (sb-thread:terminate-thread thread)
#-sbcl (bt:destroy-thread thread)
(return :timeout))
(sleep 0.05))))Initializing All Skills (initialize-all-skills)
The `initialize-all-skills` function is the unified orchestrator for the system boot sequence. It enforces the Verification Lock:
- Mandatory Check: It reads the `MANDATORY_SKILLS` environment variable and ensures every required skill exists in the source directory.
- Topological Boot: It resolves inter-skill dependencies to ensure policies and actuators are loaded in the correct order.
- Timed Loading: Every skill is loaded with a 5-second timeout.
- Boot Halt: If a mandatory skill fails to load (e.g., due to a syntax error), the entire system halts with a `BOOT FAILURE` to prevent an unaligned or unsecure state.
(defun initialize-all-skills ()
"Scans the directory defined by SKILLS_DIR and hot-loads skills using topological order."
(let* ((env-path (uiop:getenv "SKILLS_DIR"))
(skills-dir-str (or env-path (namestring (merge-pathnames "notes/" (user-homedir-pathname)))))
(resolved-path (context-resolve-path skills-dir-str))
(skills-dir (if resolved-path (uiop:ensure-directory-pathname resolved-path) nil)))
(unless (and skills-dir (uiop:directory-exists-p skills-dir))
(harness-log "HARNESS ERROR: Skills directory not found: ~a" skills-dir-str)
(return-from initialize-all-skills nil))
(let ((sorted-files (topological-sort-skills skills-dir)))
;; MANDATE: Configurable mandatory skills must be present for a safe boot
(let* ((mandatory-env (uiop:getenv "MANDATORY_SKILLS"))
(mandatory-skills (if mandatory-env
(mapcar (lambda (s) (string-trim '(#\Space) s))
(uiop:split-string mandatory-env :separator '(#\,)))
'("org-skill-policy" "org-skill-bouncer"))))
(dolist (req mandatory-skills)
(unless (member req sorted-files :key #'pathname-name :test #'string-equal)
(error "BOOT FAILURE: Mandatory skill '~a' not found in skills directory." req))))
(harness-log "==================================================")
(harness-log " LOADER: Initializing ~a skills..." (length sorted-files))
(dolist (file sorted-files)
(let* ((skill-name (pathname-name file))
(is-mandatory (member skill-name mandatory-skills :test #'string-equal)))
(harness-log " LOADER: Loading ~a..." skill-name)
(let ((status (load-skill-with-timeout file 5)))
(unless (eq status :success)
(if is-mandatory
(error "BOOT FAILURE: Mandatory skill '~a' failed to load (Status: ~a)." skill-name status)
(harness-log "LOADER WARNING: Skill '~a' failed to load." skill-name))))))
;; Final Summary
(let ((ready 0) (failed 0))
(maphash (lambda (k v)
(declare (ignore k))
(if (eq (skill-entry-status v) :ready) (incf ready) (incf failed)))
*skill-catalog*)
(harness-log " LOADER: Boot Complete. [Ready: ~a] [Failed: ~a]" ready failed)
(harness-log "==================================================")
(values ready failed)))))Toolbelt Prompt Generation (generate-tool-belt-prompt)
Every cognitive tool registered by a skill is automatically documented and injected into the LLM system prompt. This ensures that the agent is always aware of its current physical capabilities.
flowchart LR
Registry[(Tool Registry)] --> Generator[Prompt Generator]
Generator --> Prompt[Final System Prompt]
subgraph Actuators
ToolA[Shell]
ToolB[Emacs]
ToolC[Grep]
end
ToolA --> Registry
ToolB --> Registry
ToolC --> Registry(defun generate-tool-belt-prompt ()
"Aggregates all registered cognitive tools into a descriptive prompt."
(let ((output (format nil "AVAILABLE TOOLS:
You can call tools by returning a Lisp plist: (:target :tool :action :call :tool <name> :args (...))
EXAMPLES:
(:target :tool :action :call :tool \"eval\" :args (:code \"(+ 1 1)\"))
(:target :tool :action :call :tool \"grep-search\" :args (:pattern \"autonomousty\"))
(:target :tool :action :call :tool \"shell\" :args (:cmd \"ls -la\"))
---
")))
(maphash (lambda (name tool)
(setf output (concatenate 'string output
(format nil "- ~a: ~a~% Parameters: ~s~%~%"
name
(cognitive-tool-description tool)
(cognitive-tool-parameters tool)))))
*cognitive-tools*)
output))The Default Tool Belt
The harness provides a baseline set of cognitive tools that enable core system interaction.
The Eval Tool (Internal Inspection)
(def-cognitive-tool :eval "Evaluates raw Common Lisp code in the harness image. Use this for complex calculations or internal state inspection."
((:code :type :string :description "The Lisp code to evaluate"))
:guard (lambda (args context)
(declare (ignore context))
(let ((code (getf args :code)))
(let ((harness-pkg (find-package :org-agent.skills.org-skill-lisp-validator)))
(if harness-pkg
(uiop:symbol-call :org-agent.skills.org-skill-lisp-validator :lisp-validator-validate code)
t))))
:body (lambda (args)
(let ((code (getf args :code)))
(handler-case (let ((result (eval (read-from-string code))))
(format nil "~s" result))
(error (c) (format nil "ERROR: ~a" c))))))The Grep Tool (File Discovery)
(def-cognitive-tool :grep-search "Searches for a pattern in the project files."
((:pattern :type :string :description "The regex pattern to search for")
(:dir :type :string :description "Directory to search in (default is project root)"))
:body (lambda (args)
(let ((pattern (getf args :pattern))
(dir (or (getf args :dir) (uiop:getenv "MEMEX_DIR"))))
(uiop:run-program (list "grep" "-r" "-n" "--exclude-dir=node_modules" pattern dir)
:output :string :ignore-error-status t))))The Shell Tool (Machine Actuation)
(def-cognitive-tool :shell "Executes a shell command on the local machine. Use this for file operations, system checks, or running tests."
((:cmd :type :string :description "The full bash command to execute"))
:guard (lambda (args context)
(declare (ignore context))
(let ((cmd (getf args :cmd)))
(not (or (search "rm -rf /" cmd) (search ":(){ :|:& };:" cmd)))))
:body (lambda (args)
(let ((cmd (getf args :cmd)))
(multiple-value-bind (out err code)
(uiop:run-program (list "bash" "-c" cmd) :output :string :error-output :string :ignore-error-status t)
(format nil "EXIT-CODE: ~a~%~%STDOUT:~%~a~%~%STDERR:~%~a" code out err)))))