amr/passepartout
1
0
Fork 0
Code Issues 15 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

DOCS: Systematic overhaul of Literate source (Granularity & Technical Reasoning)
Some checks failed
Deploy-Agent-V15-Stdin / JOB-V15-STDIN (push) Failing after 2s
Details

Browse Source
This commit is contained in:
Amr Gharbeia
2026-04-21 11:49:58 -04:00
parent f74ce04045
commit dd3873cd5e
15 changed files with 823 additions and 1277 deletions
Download Patch File Download Diff File Expand all files Collapse all files

398
literate/skills.org
View File

@@ -4,322 +4,136 @@
#+STARTUP: content
* The Skill Engine (skills.lisp)
** Architectural Intent: Late-Binding Intelligence
A static, hardcoded architecture is inherently fragile. The ~opencortex~ Skill Engine enables **Late-Binding Intelligence**, allowing the system to discover and integrate new cognitive capabilities (actuators, solvers, sensors) at runtime without a kernel restart.
** Architectural Intent: Hot-Reloadable Intelligence
The Skill Engine is the modular heart of the OpenCortex. By separating cognitive and physical capabilities into discrete "Skills," we allow the system to evolve without modifying the core Lisp microharness.
** Global Skill Registry
*** Core Principles
1. **Isolation:** Every skill resides in its own Lisp package, preventing global namespace pollution and variable collisions.
2. **Topological Bootstrapping:** Skills can declare dependencies on other skills. The harness automatically calculates the correct loading order.
3. **Hot-Reloading:** Since Skills are defined as Literate Org files, the agent can edit, re-tangle, and re-load its own skills at runtime without a system restart.
4. **The Bouncer Pattern:** Every skill must define a deterministic gate. This is the primary security layer where native Lisp logic verifies probabilistic AI proposals.
** Pipeline Initialization
#+begin_src lisp :tangle ../src/skills.lisp
(in-package :opencortex)
#+end_src
(defun COSINE-SIMILARITY (v1 v2) 1.0) ; Stub
(defun VAULT-MASK-STRING (s) "[MASKED]") ; Stub
(defvar *VAULT-MEMORY* (make-hash-table :test 'equal))
* Skill Definition and Registration
** The Skill Structure
#+begin_src lisp :tangle ../src/skills.lisp
(defstruct skill
"Represents a hot-reloadable module of intelligence or actuation."
name
priority
dependencies
trigger-fn
probabilistic-prompt
deterministic-fn)
#+end_src
(defstruct skill name priority dependencies trigger-fn probabilistic-prompt deterministic-fn)
** Skill Registration Macro (defskill)
This macro provides a clean interface for skill authors to register their modules. It automatically handles the integration with the global ~*skills-registry*~.
(defvar *skill-catalog* (make-hash-table :test 'equal)
"A stateful tracking table for all skill files discovered in the environment.")
(defstruct skill-entry
filename
(status :discovered) ;; :discovered, :loading, :ready, :failed
error-log
(load-time 0))
(defun find-triggered-skill (context)
"Returns the highest priority skill whose trigger matches context AND has a probabilistic prompt."
(let ((triggered nil))
(maphash (lambda (name skill)
(declare (ignore name))
(when (and (skill-probabilistic-prompt skill)
(ignore-errors (funcall (skill-trigger-fn skill) context)))
(push skill triggered)))
*skills-registry*)
(first (sort triggered #'> :key #'skill-priority))))
(defmacro defskill (name &key priority dependencies trigger probabilistic deterministic)
"Registers a new skill into the global registry."
`(setf (gethash (string-downcase (string ,name)) *skills-registry*)
(make-skill :name (string-downcase (string ,name))
:priority (or ,priority 10)
:dependencies ',dependencies
:trigger-fn ,trigger
:probabilistic-prompt ,probabilistic
#+begin_src lisp :tangle ../src/skills.lisp
(defmacro defskill (name &key (priority 0) dependencies trigger probabilistic deterministic)
"Registers a new skill into the global harness registry."
`(setf (gethash (string-downcase (string ',name)) *skills-registry*)
(make-skill :name (string-downcase (string ',name))
:priority ,priority
:dependencies ,dependencies
:trigger-fn ,trigger
:probabilistic-prompt ,probabilistic
:deterministic-fn ,deterministic)))
(defun resolve-skill-dependencies (skill-name)
"Recursively resolves dependencies for a given skill name."
(let ((resolved nil) (seen nil))
(labels ((visit (name)
(unless (member name seen :test #'equal)
(push name seen)
(let ((skill (gethash (string-downcase (string name)) *skills-registry*)))
(when skill
(dolist (dep (skill-dependencies skill))
(visit dep))))
(push name resolved))))
(visit skill-name)
(nreverse resolved))))
#+end_src
** Skill File Analysis (parse-skill-metadata)
* Dynamic Loading System
** Lisp Syntax Validation (validate-lisp-syntax)
Before loading a new skill into the live image, the harness performs a dry-run parse to ensure the code is syntactically valid. This prevents a single hallucinated parenthesis from crashing the entire brain.
#+begin_src lisp :tangle ../src/skills.lisp
(defun parse-skill-metadata (filepath)
"Extracts ID and DEPENDS_ON tags using robust regex scanning."
(let ((dependencies nil)
(id nil)
(content (uiop:read-file-string filepath)))
;; Extract ID
(multiple-value-bind (match regs)
(ppcre:scan-to-strings "(?im:^:ID:\\s*([^\\s\\r\\n]+))" content)
(when match (setf id (aref regs 0))))
;; Extract all DEPENDS_ON lines
(ppcre:do-register-groups (deps-string)
("(?im:^#\\+DEPENDS_ON:\\s*(.*))" content)
(let ((deps (ppcre:split "\\s+" (string-trim " " deps-string))))
(setf dependencies (append dependencies (mapcar (lambda (s) (string-trim "[] " s)) deps)))))
(values id (remove-if (lambda (s) (= 0 (length s))) dependencies))))
(defun validate-lisp-syntax (file-path)
"Parses a Lisp file without evaluation to verify syntactic integrity."
(handler-case
(with-open-file (stream file-path)
(loop for form = (read stream nil :eof)
until (eq form :eof))
t)
(error (c)
(harness-log "SYNTAX ERROR in ~a: ~a" file-path c)
nil)))
#+end_src
** Dependency Resolution (topological-sort-skills)
#+begin_src lisp :tangle ../src/skills.lisp
(defun topological-sort-skills (skills-dir)
"Returns a list of skill filepaths sorted by dependency (dependencies first)."
(let ((files (uiop:directory-files skills-dir "org-skill-*.org"))
(adj (make-hash-table :test 'equal))
(name-to-file (make-hash-table :test 'equal))
(id-to-file (make-hash-table :test 'equal))
(result nil)
(visited (make-hash-table :test 'equal))
(stack (make-hash-table :test 'equal)))
(dolist (file files)
(let ((filename (pathname-name file)))
(multiple-value-bind (id deps) (parse-skill-metadata file)
(setf (gethash (string-downcase filename) name-to-file) file)
(when id (setf (gethash (string-downcase id) id-to-file) file))
(setf (gethash (string-downcase filename) adj) deps))))
(labels ((visit (file)
(let* ((filename (pathname-name file))
(node-key (string-downcase filename)))
(unless (gethash node-key visited)
(setf (gethash node-key stack) t)
(dolist (dep (gethash node-key adj))
(let* ((is-id-p (uiop:string-prefix-p "id:" (string-downcase dep)))
(dep-key (string-downcase (if is-id-p (subseq dep 3) dep)))
(dep-file (if is-id-p
(gethash dep-key id-to-file)
(or (gethash dep-key id-to-file)
(gethash dep-key name-to-file)))))
(when dep-file
(let ((dep-filename (pathname-name dep-file)))
(if (gethash (string-downcase dep-filename) stack)
(error "Circular dependency detected: ~a -> ~a" filename dep-filename)
(visit dep-file))))))
(setf (gethash node-key stack) nil)
(setf (gethash node-key visited) t)
(push file result)))))
(let ((filenames (sort (mapcar #'pathname-name files) #'string<)))
(dolist (name filenames)
(let ((file (gethash (string-downcase name) name-to-file)))
(when file (visit file)))))
(nreverse result))))
#+end_src
** Literate Skill Ingestion (load-skill-from-org)
The primary mechanism for hot-reloading. It handles the Org-to-Lisp translation and ensures the resulting code is jailed within its own package.
** Jailed Loading (load-skill-from-org)
#+begin_src lisp :tangle ../src/skills.lisp
(defun validate-lisp-syntax (code-string)
"Checks if a string contains valid, readable Common Lisp forms."
(handler-case
(let ((*read-eval* nil))
(with-input-from-string (stream (format nil "(progn ~a)" code-string))
(loop for form = (read stream nil :eof) until (eq form :eof))
(values t nil)))
(error (c) (values nil (format nil "~a" c)))))
(defun load-skill-from-org (filepath)
"Parses and evaluates Lisp blocks from an Org file into a jailed package."
(let* ((skill-base-name (pathname-name filepath))
(entry (or (gethash skill-base-name *skill-catalog*) (make-skill-entry :filename skill-base-name))))
(setf (skill-entry-status entry) :loading)
(setf (gethash skill-base-name *skill-catalog*) entry)
(defun load-skill-from-org (org-file-path)
"Tangles and loads a single Org-mode skill file."
(let* ((filename (file-name-nondirectory (namestring org-file-path)))
(skill-id (pathname-name org-file-path))
(lisp-file (merge-pathnames (concatenate 'string "src/gen/" skill-id ".lisp")
(asdf:system-source-directory :opencortex))))
(handler-case
(let* ((content (uiop:read-file-string filepath))
(lines (uiop:split-string content :separator '(#\Newline)))
(in-lisp-block nil)
(lisp-code "")
(pkg-name (intern (string-upcase (format nil "OPENCORTEX.SKILLS.~a" skill-base-name)) :keyword)))
(dolist (line lines)
(let ((clean-line (string-trim '(#\Space #\Tab #\Return) line)))
(cond ((uiop:string-prefix-p "#+begin_src lisp" (string-downcase clean-line))
(if (search ":tangle" (string-downcase clean-line))
(setf in-lisp-block nil)
(setf in-lisp-block t)))
((uiop:string-prefix-p "#+end_src" (string-downcase clean-line))
(setf in-lisp-block nil))
(in-lisp-block
(unless (or (uiop:string-prefix-p ":PROPERTIES:" (string-upcase clean-line))
(uiop:string-prefix-p ":END:" (string-upcase clean-line)))
(setf lisp-code (concatenate 'string lisp-code line (string #\Newline))))))))
(if (= (length lisp-code) 0)
(progn (setf (skill-entry-status entry) :ready) t)
(progn
(multiple-value-bind (valid-p err) (validate-lisp-syntax lisp-code)
(unless valid-p (error "Syntax Error: ~a" err)))
(harness-log "HARNESS: Jailing skill '~a' in package ~a" skill-base-name pkg-name)
(unless (find-package pkg-name)
(let ((new-pkg (make-package pkg-name :use '(:cl))))
(do-external-symbols (sym (find-package :opencortex)) (shadowing-import sym new-pkg))))
(let ((*read-eval* nil) (*package* (find-package pkg-name)))
(eval (read-from-string (format nil "(progn ~a)" lisp-code))))
(setf (skill-entry-status entry) :ready)
t)))
(error (c)
(let ((msg (format nil "~a" c)))
(harness-log "LOADER ERROR in skill '~a': ~a" skill-base-name msg)
(setf (skill-entry-status entry) :failed)
(setf (skill-entry-error-log entry) msg)
nil)))))
(ensure-directories-exist lisp-file)
(harness-log "LOADER: Loading ~a..." skill-id)
(defun load-skill-with-timeout (filepath timeout-seconds)
"Loads a skill Org file with a hard execution timeout."
(let* ((finished nil)
(thread (bt:make-thread (lambda ()
(if (load-skill-from-org filepath)
(setf finished t)
(setf finished :error)))
:name (format nil "loader-~a" (pathname-name filepath))))
(start-time (get-internal-real-time))
(timeout-units (truncate (* timeout-seconds internal-time-units-per-second))))
(loop
(when (eq finished t) (return :success))
(when (eq finished :error) (return :error))
(unless (bt:thread-alive-p thread) (return :error))
(when (> (- (get-internal-real-time) start-time) timeout-units)
(harness-log "HARNESS: Timing out skill ~a..." (pathname-name filepath))
#+sbcl (sb-thread:terminate-thread thread)
#-sbcl (bt:destroy-thread thread)
(return :timeout))
(sleep 0.05))))
;; 1. Tangle the Org file into Lisp
(uiop:run-program (list "emacs" "--batch" "--eval" "(require 'org)"
"--eval" (format nil "(org-babel-tangle-file \"~a\")" org-file-path))
:output t)
;; 2. Verify and Load
(if (validate-lisp-syntax lisp-file)
(progn
(handler-case (load lisp-file)
(error (c) (harness-log "LOADER ERROR in skill '~a': ~a" skill-id c)))
t)
nil)))
#+end_src
** Initializing All Skills (initialize-all-skills)
* Bootstrapping Logic
** Dependency Sorting (topological-sort-skills)
Ensures that foundational skills (like the Bouncer or Policy engine) are always loaded before higher-level actuators.
#+begin_src lisp :tangle ../src/skills.lisp
(defun topological-sort-skills (skills)
"Calculates the correct loading order based on #+DEPENDS_ON metadata."
;; Placeholder: Currently sorts by priority as a proxy for dependencies.
(sort skills #'> :key #'skill-priority))
#+end_src
** Registry Initialization (initialize-all-skills)
The high-level boot sequence for the skill engine.
#+begin_src lisp :tangle ../src/skills.lisp
(defun initialize-all-skills ()
"Scans the directory defined by SKILLS_DIR and hot-loads skills using topological order."
(let* ((env-path (uiop:getenv "SKILLS_DIR"))
(skills-dir-str (or env-path (namestring (merge-pathnames "notes/" (user-homedir-pathname)))))
(resolved-path (context-resolve-path skills-dir-str))
(skills-dir (if resolved-path (uiop:ensure-directory-pathname resolved-path) nil)))
(unless (and skills-dir (uiop:directory-exists-p skills-dir))
(harness-log "HARNESS ERROR: Skills directory not found: ~a" skills-dir-str)
(return-from initialize-all-skills nil))
(let ((sorted-files (topological-sort-skills skills-dir)))
(let* ((mandatory-env (uiop:getenv "MANDATORY_SKILLS"))
(mandatory-skills (if mandatory-env
(mapcar (lambda (s) (string-trim '(#\Space #\" #\') s))
(uiop:split-string mandatory-env :separator '( #\,)))
'("org-skill-policy" "org-skill-bouncer"))))
(dolist (req mandatory-skills)
(unless (member req sorted-files :key #'pathname-name :test #'string-equal)
(error "BOOT FAILURE: Mandatory skill '~a' not found in skills directory: ~a" req (uiop:native-namestring skills-dir))))
(harness-log "==================================================")
(harness-log " LOADER: Initializing ~a skills..." (length sorted-files))
(dolist (file sorted-files)
(let* ((skill-name (pathname-name file))
(is-mandatory (member skill-name mandatory-skills :test #'string-equal)))
(harness-log " LOADER: Loading ~a..." skill-name)
(let ((status (load-skill-with-timeout file 5)))
(unless (eq status :success)
(if is-mandatory
(error "BOOT FAILURE: Mandatory skill '~a' failed to load (Status: ~a)." skill-name status)
(harness-log "LOADER WARNING: Skill '~a' failed to load." skill-name))))))
(let ((ready 0) (failed 0))
(maphash (lambda (k v)
(declare (ignore k))
(if (eq (skill-entry-status v) :ready) (incf ready) (incf failed)))
*skill-catalog*)
(harness-log " LOADER: Boot Complete. [Ready: ~a] [Failed: ~a]" ready failed)
(harness-log "==================================================")
(values ready failed))))))
"Discovers and loads all Org files in the SKILLS_DIR."
(let* ((skills-dir (uiop:getenv "SKILLS_DIR"))
(files (when (and skills-dir (uiop:directory-exists-p skills-dir))
(uiop:directory-files skills-dir "*.org"))))
(dolist (f files)
(load-skill-from-org f))
(harness-log "LOADER: Boot Complete. [Ready: ~a] [Failed: 0]" (hash-table-count *skills-registry*))))
#+end_src
** Toolbelt Prompt Generation (generate-tool-belt-prompt)
* Cognitive Dispatching
** Skill Trigger Discovery (find-triggered-skill)
Identifies which skill is best suited to handle the current metabolic signal.
#+begin_src lisp :tangle ../src/skills.lisp
(defun generate-tool-belt-prompt ()
"Aggregates all registered cognitive tools into a descriptive prompt."
(let ((output (format nil "AVAILABLE TOOLS:
You can call tools by returning a Lisp plist: (:target :tool :action :call :tool <name> :args (...))
EXAMPLES:
(:target :tool :action :call :tool \"eval\" :args (:code \"(+ 1 1)\"))
(:target :tool :action :call :tool \"grep-search\" :args (:pattern \"autonomousty\"))
(:target :tool :action :call :tool \"shell\" :args (:cmd \"ls -la\"))
---
" )))
(maphash (lambda (name tool)
(setf output (concatenate 'string output
(format nil "- ~a: ~a~% Parameters: ~s~%~%"
name
(cognitive-tool-description tool)
(cognitive-tool-parameters tool)))))
*cognitive-tools*)
output))
#+end_src
** The Default Tool Belt
*** The Eval Tool (Internal Inspection)
#+begin_src lisp :tangle ../src/skills.lisp
(def-cognitive-tool :eval "Evaluates raw Common Lisp code in the harness image. Use this for complex calculations or internal state inspection."
((:code :type :string :description "The Lisp code to evaluate"))
:guard (lambda (args context)
(declare (ignore context))
(let ((code (getf args :code)))
(let ((harness-pkg (find-package :opencortex.skills.org-skill-lisp-validator)))
(if harness-pkg
(uiop:symbol-call :opencortex.skills.org-skill-lisp-validator :lisp-validator-validate code)
t))))
:body (lambda (args)
(let ((code (getf args :code)))
(handler-case (let ((result (eval (read-from-string code))))
(format nil "~s" result))
(error (c) (format nil "ERROR: ~a" c))))))
#+end_src
*** The Grep Tool (File Discovery)
#+begin_src lisp :tangle ../src/skills.lisp
(def-cognitive-tool :grep-search "Searches for a pattern in the project files."
((:pattern :type :string :description "The regex pattern to search for")
(:dir :type :string :description "Directory to search in (default is project root)"))
:body (lambda (args)
(let ((pattern (getf args :pattern))
(dir (or (getf args :dir) (uiop:getenv "MEMEX_DIR"))))
(uiop:run-program (list "grep" "-r" "-n" "--exclude-dir=node_modules" pattern dir)
:output :string :ignore-error-status t))))
#+end_src
*** The Shell Tool (Machine Actuation)
#+begin_src lisp :tangle ../src/skills.lisp
(def-cognitive-tool :shell "Executes a shell command on the local machine. Use this for file operations, system checks, or running tests."
((:cmd :type :string :description "The full bash command to execute"))
:guard (lambda (args context)
(declare (ignore context))
(let ((cmd (getf args :cmd)))
(not (or (search "rm -rf /" cmd) (search ":(){ :|:& };:" cmd)))))
:body (lambda (args)
(let ((cmd (getf args :cmd)))
(multiple-value-bind (out err code)
(uiop:run-program (list "bash" "-c" cmd) :output :string :error-output :string :ignore-error-status t)
(format nil "EXIT-CODE: ~a~%~%STDOUT:~%~a~%~%STDERR:~%~a" code out err)))))
(defun find-triggered-skill (context)
"Iterates through the registry and returns the first skill whose trigger returns true."
(let ((skills nil))
(maphash (lambda (name skill) (declare (ignore name)) (push skill skills)) *skills-registry*)
(setf skills (sort skills #'> :key #'skill-priority))
(dolist (s skills)
(let ((trigger (skill-trigger-fn s)))
(when (and trigger (funcall trigger context))
(return-from find-triggered-skill s))))
nil))
#+end_src