#+TITLE: The Skill Engine (skills.lisp)
#+AUTHOR: Amr
#+FILETAGS: :harness:skills:
#+STARTUP: content

* The Skill Engine (skills.lisp)
** Architectural Intent: Late-Binding Intelligence

A static, hardcoded architecture is inherently fragile. To build a sovereign agent that can evolve alongside its user, the harness must be a "Thin Shell" that delegates its capabilities to dynamic, hot-reloadable modules known as **Skills**. This is the core of our **Thin Harness / Thick Skill Microkernel Architecture**.

Skills unify the **"Why"** (Literate Org documentation) and the **"How"** (Functional Lisp implementation). This allows the harness to "learn" new behaviors without a full system restart, enabling a continuous evolutionary loop where the agent can eventually inspect and improve its own code.

*** The True Microkernel (Decoupled Core Skills)
Historically, "core" skills (like State Persistence or Gateways) were statically compiled into the harness for performance. We have fundamentally decoupled this. Now, *all* behavioral skills are pure user-space dynamic modules. They do not tangle to `src/` and are not listed in `org-agent.asd`. The harness simply boots, scans the `skills/` directory, and evaluates the code inside a jailed package. If a user wishes to swap the IPFS persistence skill for an AWS S3 one, they simply swap the `.org` file; no kernel recompilation is required.

*** Dormant Verification (Tests)
Because skills are no longer statically compiled into the core `org-agent` ASDF system, their associated `FiveAM` test blocks are currently dormant during a standard static build. The tests still exist within the literate `.org` files as verifiable documentation, but executing them requires either dynamic evaluation at runtime or a dedicated test-loader skill.

*** 1. The Package Jailing Principle
Every skill is evaluated within its own dedicated Common Lisp package (namespace). This "Jailing" prevents symbol collisions between disparate skills and ensures that a bug in one module cannot easily corrupt the internal state of another.

*** 2. Deterministic Load Ordering
Skills often depend on one another. The harness implements a deterministic topological sorting algorithm to ensure that dependencies are loaded before the skills that require them.

** Skill Architecture
#+begin_src mermaid
flowchart TD
    Registry[Skills Registry] --> S1[Skill: System Policy]
    Registry --> S2[Skill: LLM Gateway]
    Registry --> S3[Skill: Token Accountant]
    S2 -- Depends On --> S1
    S3 -- Depends On --> S2
    
    subgraph Jailing[Package Jailing]
        P1[Package: ORG-AGENT.SKILLS.S1]
        P2[Package: ORG-AGENT.SKILLS.S2]
        P3[Package: ORG-AGENT.SKILLS.S3]
    end
    
    S1 --> P1
    S2 --> P2
    S3 --> P3
#+end_src

** Package Context
We begin by ensuring we are in the correct isolated harness namespace.

#+begin_src lisp :tangle ../src/skills.lisp
(in-package :org-agent)
#+end_src

** Skill Metadata (defstruct skill)
The core data structure representing an agent capability. It includes the trigger condition, the probabilistic prompt generator, and the deterministic safety gate.

#+begin_src lisp :tangle ../src/skills.lisp
(defstruct skill name priority dependencies trigger-fn probabilistic-prompt deterministic-fn)
#+end_src

** Skill Catalog Tracking
The harness maintains a stateful tracking table for all skill files discovered in the environment (~notes/org-skill-*.org~).

#+begin_src lisp :tangle ../src/skills.lisp
(defvar *skill-catalog* (make-hash-table :test 'equal)
  "A stateful tracking table for all skill files discovered in the environment.")

(defstruct skill-entry 
  filename 
  (status :discovered) ;; :discovered, :loading, :ready, :failed
  error-log
  (load-time 0))
#+end_src

** Skill Selection (find-triggered-skill)
The primary dispatcher for the Probabilistic Engine. It iterates through the registry to find the highest-priority skill whose trigger function matches the current cognitive context.

#+begin_src lisp :tangle ../src/skills.lisp
(defun find-triggered-skill (context)
  "Returns the highest priority skill whose trigger condition matches the context."
  (let ((triggered nil))
    (maphash (lambda (name skill) 
               (declare (ignore name)) 
               (when (ignore-errors (funcall (skill-trigger-fn skill) context)) 
                 (push skill triggered))) 
             *skills-registry*)
    (first (sort triggered #'> :key #'skill-priority))))
#+end_src

** Skill Definition Macro (defskill)
The interface used within Org files to register new capabilities. Note that dependencies are explicitly quoted to prevent premature evaluation during the macro expansion phase.

#+begin_src lisp :tangle ../src/skills.lisp
(defmacro defskill (name &key priority dependencies trigger probabilistic deterministic)
  "Registers a new skill into the global registry."
  `(setf (gethash (string-downcase (string ,name)) *skills-registry*)
         (make-skill :name (string-downcase (string ,name)) 
                     :priority (or ,priority 10) 
                     :dependencies ',dependencies
                     :trigger-fn ,trigger 
                     :probabilistic-prompt ,probabilistic 
                     :deterministic-fn ,deterministic)))
#+end_src

** Dependency Resolution (resolve-skill-dependencies)
Recursively flattens the dependency graph for a given skill. This is used during hot-unloading to ensure that dependent skills are also refreshed.

#+begin_src lisp :tangle ../src/skills.lisp
(defun resolve-skill-dependencies (skill-name)
  "Recursively resolves dependencies for a given skill name."
  (let ((resolved nil) (seen nil))
    (labels ((visit (name) 
               (unless (member name seen :test #'equal) 
                 (push name seen)
                 (let ((skill (gethash (string-downcase (string name)) *skills-registry*)))
                   (when skill 
                     (dolist (dep (skill-dependencies skill)) 
                       (visit dep))))
                 (push name resolved))))
      (visit skill-name) 
      (nreverse resolved))))
#+end_src

** Metadata Parsing (parse-skill-metadata)
A robust, low-level scanner that extracts `#+DEPENDS_ON:` and `:ID:` tags from an Org file. This allows the harness to calculate the load order without needing to parse the full Org AST, which would create a boot-time circularity.

#+begin_src lisp :tangle ../src/skills.lisp
(defun parse-skill-metadata (filepath)
  "Extracts ID and DEPENDS_ON tags using robust regex scanning."
  (let ((dependencies nil)
        (id nil)
        (content (uiop:read-file-string filepath)))
    ;; Extract ID
    (multiple-value-bind (match regs)
        (ppcre:scan-to-strings "(?im:^:ID:\\s*([^\\s\\r\\n]+))" content)
      (when match (setf id (aref regs 0))))
    ;; Extract all DEPENDS_ON lines
    (ppcre:do-register-groups (deps-string)
        ("(?im:^#\\+DEPENDS_ON:\\s*(.*))" content)
      (let ((deps (ppcre:split "\\s+" (string-trim " " deps-string))))
        (setf dependencies (append dependencies (mapcar (lambda (s) (string-trim "[] " s)) deps)))))
    (values id (remove-if (lambda (s) (= 0 (length s))) dependencies))))
#+end_src

** Topological Sorting (topological-sort-skills)
This is the core algorithm of the boot sequence. It uses a **Depth-First Search (DFS)** to resolve the skill dependency graph. 

It performs three critical roles:
1.  **Load Ordering:** Ensures that "foundational" skills (like the LLM Gateway) are loaded before high-level "behavioral" skills.
2.  **ID Resolution:** Correct maps Org `:ID:` properties to filepaths.
3.  **Cycle Detection:** It uses a recursion stack to detect circular dependencies (e.g., A depends on B, B depends on A) and errors out safely before the Lisp image hangs.

#+begin_src lisp :tangle ../src/skills.lisp
(defun topological-sort-skills (skills-dir)
  "Returns a list of skill filepaths sorted by dependency (dependencies first)."
  (let ((files (uiop:directory-files skills-dir "org-skill-*.org"))
        (adj (make-hash-table :test 'equal))
        (name-to-file (make-hash-table :test 'equal))
        (id-to-file (make-hash-table :test 'equal))
        (result nil)
        (visited (make-hash-table :test 'equal))
        (stack (make-hash-table :test 'equal)))
    (dolist (file files)
      (let ((filename (pathname-name file)))
        (multiple-value-bind (id deps) (parse-skill-metadata file)
          (setf (gethash (string-downcase filename) name-to-file) file)
          (when id (setf (gethash (string-downcase id) id-to-file) file))
          (setf (gethash (string-downcase filename) adj) deps))))
    (labels ((visit (file)
               (let* ((filename (pathname-name file))
                      (node-key (string-downcase filename)))
                 (unless (gethash node-key visited)
                   (setf (gethash node-key stack) t)
                   (dolist (dep (gethash node-key adj))
                     (let* ((is-id-p (uiop:string-prefix-p "id:" (string-downcase dep)))
                            (dep-key (string-downcase (if is-id-p (subseq dep 3) dep)))
                            (dep-file (if is-id-p 
                                          (gethash dep-key id-to-file)
                                          (or (gethash dep-key id-to-file)
                                              (gethash dep-key name-to-file)))))
                       (when dep-file
                         (let ((dep-filename (pathname-name dep-file)))
                           (if (gethash (string-downcase dep-filename) stack)
                               (error "Circular dependency detected: ~a -> ~a" filename dep-filename)
                               (visit dep-file))))))
                   (setf (gethash node-key stack) nil)
                   (setf (gethash node-key visited) t)
                   (push file result)))))
      (let ((filenames (sort (mapcar #'pathname-name files) #'string<)))
        (dolist (name filenames)
          (let ((file (gethash (string-downcase name) name-to-file)))
            (when file (visit file)))))
      (nreverse result))))
#+end_src

** Syntax Validation (validate-lisp-syntax)
A pre-flight safety check. Before evaluating any code from an Org file, the harness attempts to ~read~ the entire string. If the reader encounters a syntax error (like an unclosed parenthesis), the load is aborted before the Lisp image can crash.

#+begin_src lisp :tangle ../src/skills.lisp
(defun validate-lisp-syntax (code-string)
  "Checks if a string contains valid, readable Common Lisp forms."
  (handler-case 
      (let ((*read-eval* nil)) 
        (with-input-from-string (stream (format nil "(progn ~a)" code-string))
          (loop for form = (read stream nil :eof) until (eq form :eof)) 
          (values t nil)))
    (error (c) (values nil (format nil "~a" c)))))
#+end_src

** Jailed Loading (load-skill-from-org)
The core "hot-loading" mechanism. It extracts Lisp blocks from an Org file and evaluates them within a "Jail" (an isolated package).

*** The Jailing Algorithm:
1.  **Isolation:** It generates a package name based on the filename (e.g., ~ORG-AGENT.SKILLS.CORE-LOGIC~).
2.  **Namespace Protection:** It inherits external symbols from the ~ORG-AGENT~ package, allowing the skill to use the harness API, but keeps its internal helper functions local.
3.  **Block Filtering:** It explicitly ignores blocks that contain ~:tangle~, ensuring that harness-level code (which is already in ~src/~) is not accidentally re-evaluated as skill logic.

#+begin_src lisp :tangle ../src/skills.lisp
(defun load-skill-from-org (filepath)
  "Parses and evaluates Lisp blocks from an Org file into a jailed package."
  (let* ((skill-base-name (pathname-name filepath))
         (entry (or (gethash skill-base-name *skill-catalog*) (make-skill-entry :filename skill-base-name))))
    (setf (skill-entry-status entry) :loading)
    (setf (gethash skill-base-name *skill-catalog*) entry)
    
    (handler-case
        (let* ((content (uiop:read-file-string filepath)) 
               (lines (uiop:split-string content :separator '(#\Newline)))
               (in-lisp-block nil) 
               (lisp-code "") 
               (pkg-name (intern (string-upcase (format nil "ORG-AGENT.SKILLS.~a" skill-base-name)) :keyword)))
          
          (dolist (line lines)
            (let ((clean-line (string-trim '(#\Space #\Tab #\Return) line)))
              (cond ((uiop:string-prefix-p "#+begin_src lisp" (string-downcase clean-line))
                     ;; Only load blocks that are NOT tangled to src/ or elsewhere
                     (if (search ":tangle" (string-downcase clean-line))
                         (setf in-lisp-block nil)
                         (setf in-lisp-block t)))
                    ((uiop:string-prefix-p "#+end_src" (string-downcase clean-line))
                     (setf in-lisp-block nil))
                    (in-lisp-block 
                     (unless (or (uiop:string-prefix-p ":PROPERTIES:" (string-upcase clean-line))
                                 (uiop:string-prefix-p ":END:" (string-upcase clean-line)))
                       (setf lisp-code (concatenate 'string lisp-code line (string #\Newline))))))))
          
          (if (= (length lisp-code) 0)
              (progn (setf (skill-entry-status entry) :ready) t) ;; Valid empty skill
              (progn
                ;; PRE-FLIGHT: Syntax Validation
                (multiple-value-bind (valid-p err) (validate-lisp-syntax lisp-code)
                  (unless valid-p
                    (error "Syntax Error: ~a" err)))
                
                (harness-log "HARNESS: Jailing skill '~a' in package ~a" skill-base-name pkg-name)
                (unless (find-package pkg-name)
                  (let ((new-pkg (make-package pkg-name :use '(:cl))))
                    (do-external-symbols (sym (find-package :org-agent)) (shadowing-import sym new-pkg))))
                
                (let ((*read-eval* nil) (*package* (find-package pkg-name)))
                  (eval (read-from-string (format nil "(progn ~a)" lisp-code))))
                
                (setf (skill-entry-status entry) :ready)
                t)))
      (error (c)
        (let ((msg (format nil "~a" c)))
          (harness-log "LOADER ERROR in skill '~a': ~a" skill-base-name msg)
          (setf (skill-entry-status entry) :failed)
          (setf (skill-entry-error-log entry) msg)
          nil)))))
#+end_src

** Safe Loading with Timeout (load-skill-with-timeout)
Wraps the skill loader in a thread with a hard timeout to prevent a single malformed skill from hanging the entire boot sequence.

#+begin_src lisp :tangle ../src/skills.lisp
(defun load-skill-with-timeout (filepath timeout-seconds)
  "Loads a skill Org file with a hard execution timeout."
  (let* ((finished nil)
         (thread (bt:make-thread (lambda () 
                                   (if (load-skill-from-org filepath)
                                       (setf finished t)
                                       (setf finished :error)))
                                 :name (format nil "loader-~a" (pathname-name filepath))))
         (start-time (get-internal-real-time))
         (timeout-units (truncate (* timeout-seconds internal-time-units-per-second))))
    (loop 
      (when (eq finished t) (return :success))
      (when (eq finished :error) (return :error))
      (unless (bt:thread-alive-p thread) (return :error))
      (when (> (- (get-internal-real-time) start-time) timeout-units)
        (harness-log "HARNESS: Timing out skill ~a..." (pathname-name filepath))
        #+sbcl (sb-thread:terminate-thread thread)
        #-sbcl (bt:destroy-thread thread)
        (return :timeout))
      (sleep 0.05))))
#+end_src

** Initializing All Skills (initialize-all-skills)
The unified orchestrator for the system boot sequence.

#+begin_src lisp :tangle ../src/skills.lisp
(defun initialize-all-skills ()
  "Scans the directory defined by SKILLS_DIR and hot-loads skills using topological order."
  (let* ((env-path (uiop:getenv "SKILLS_DIR"))
         (skills-dir-str (or env-path (namestring (merge-pathnames "notes/" (user-homedir-pathname)))))
         (resolved-path (context-resolve-path skills-dir-str))
         (skills-dir (if resolved-path (uiop:ensure-directory-pathname resolved-path) nil)))
    
    (unless (and skills-dir (uiop:directory-exists-p skills-dir))
      (harness-log "HARNESS ERROR: Skills directory not found: ~a" skills-dir-str)
      (return-from initialize-all-skills nil))

    (let ((sorted-files (topological-sort-skills skills-dir)))
      ;; MANDATE: Configurable mandatory skills must be present for a safe boot
      (let* ((mandatory-env (uiop:getenv "MANDATORY_SKILLS"))
             (mandatory-skills (if mandatory-env 
                                   (mapcar (lambda (s) (string-trim '(#\Space) s)) 
                                           (uiop:split-string mandatory-env :separator '(#\,)))
                                   '("org-skill-policy" "org-skill-bouncer"))))
        (dolist (req mandatory-skills)
          (unless (member req sorted-files :key #'pathname-name :test #'string-equal)
            (error "BOOT FAILURE: Mandatory skill '~a' not found in skills directory." req))))
      
      (harness-log "==================================================")
      (harness-log " LOADER: Initializing ~a skills..." (length sorted-files))
      
      (dolist (file sorted-files)
        (let* ((skill-name (pathname-name file))
               (is-mandatory (member skill-name mandatory-skills :test #'string-equal)))
          (harness-log " LOADER: Loading ~a..." skill-name)
          (let ((status (load-skill-with-timeout file 5)))
            (unless (eq status :success)
              (if is-mandatory
                  (error "BOOT FAILURE: Mandatory skill '~a' failed to load (Status: ~a)." skill-name status)
                  (harness-log "LOADER WARNING: Skill '~a' failed to load." skill-name))))))
      
      ;; Final Summary
      (let ((ready 0) (failed 0))
        (maphash (lambda (k v) 
                   (declare (ignore k))
                   (if (eq (skill-entry-status v) :ready) (incf ready) (incf failed)))
                 *skill-catalog*)
        (harness-log " LOADER: Boot Complete. [Ready: ~a] [Failed: ~a]" ready failed)
        (harness-log "==================================================")
        (values ready failed)))))
#+end_src

** Toolbelt Prompt Generation (generate-tool-belt-prompt)
Every cognitive tool registered by a skill is automatically documented and injected into the LLM system prompt. This ensures that the agent is always aware of its current physical capabilities.

#+begin_src mermaid
flowchart LR
    Registry[(Tool Registry)] --> Generator[Prompt Generator]
    Generator --> Prompt[Final System Prompt]
    subgraph Actuators
        ToolA[Shell]
        ToolB[Emacs]
        ToolC[Grep]
    end
    ToolA --> Registry
    ToolB --> Registry
    ToolC --> Registry
#+end_src

#+begin_src lisp :tangle ../src/skills.lisp
(defun generate-tool-belt-prompt ()
  "Aggregates all registered cognitive tools into a descriptive prompt."
  (let ((output (format nil "AVAILABLE TOOLS:
You can call tools by returning a Lisp plist: (:target :tool :action :call :tool <name> :args (...))

EXAMPLES:
(:target :tool :action :call :tool \"eval\" :args (:code \"(+ 1 1)\"))
(:target :tool :action :call :tool \"grep-search\" :args (:pattern \"sovereignty\"))
(:target :tool :action :call :tool \"shell\" :args (:cmd \"ls -la\"))

---
")))
    (maphash (lambda (name tool)
               (setf output (concatenate 'string output
                                         (format nil "- ~a: ~a~%  Parameters: ~s~%~%"
                                                 name
                                                 (cognitive-tool-description tool)
                                                 (cognitive-tool-parameters tool)))))
             *cognitive-tools*)
    output))
#+end_src

** The Default Tool Belt
The harness provides a baseline set of cognitive tools that enable core system interaction.

*** The Eval Tool (Internal Inspection)
#+begin_src lisp :tangle ../src/skills.lisp
(def-cognitive-tool :eval "Evaluates raw Common Lisp code in the harness image. Use this for complex calculations or internal state inspection."
  ((:code :type :string :description "The Lisp code to evaluate"))
  :guard (lambda (args context)
           (declare (ignore context))
           (let ((code (getf args :code)))
             (let ((harness-pkg (find-package :org-agent.skills.org-skill-lisp-validator)))
               (if harness-pkg 
                   (uiop:symbol-call :org-agent.skills.org-skill-lisp-validator :lisp-validator-validate code)
                   t))))
  :body (lambda (args)
          (let ((code (getf args :code)))
            (handler-case (let ((result (eval (read-from-string code))))
                            (format nil "~s" result))
              (error (c) (format nil "ERROR: ~a" c))))))
#+end_src

*** The Grep Tool (File Discovery)
#+begin_src lisp :tangle ../src/skills.lisp
(def-cognitive-tool :grep-search "Searches for a pattern in the project files."
  ((:pattern :type :string :description "The regex pattern to search for")
   (:dir :type :string :description "Directory to search in (default is project root)"))
  :body (lambda (args)
          (let ((pattern (getf args :pattern))
                (dir (or (getf args :dir) (uiop:getenv "MEMEX_DIR"))))
            (uiop:run-program (list "grep" "-r" "-n" "--exclude-dir=node_modules" pattern dir) 
                              :output :string :ignore-error-status t))))
#+end_src

*** The Shell Tool (Machine Actuation)
#+begin_src lisp :tangle ../src/skills.lisp
(def-cognitive-tool :shell "Executes a shell command on the local machine. Use this for file operations, system checks, or running tests."
  ((:cmd :type :string :description "The full bash command to execute"))
  :guard (lambda (args context)
           (declare (ignore context))
           (let ((cmd (getf args :cmd)))
             (not (or (search "rm -rf /" cmd) (search ":(){ :|:& };:" cmd)))))
  :body (lambda (args)
          (let ((cmd (getf args :cmd)))
            (multiple-value-bind (out err code)
                (uiop:run-program (list "bash" "-c" cmd) :output :string :error-output :string :ignore-error-status t)
              (format nil "EXIT-CODE: ~a~%~%STDOUT:~%~a~%~%STDERR:~%~a" code out err)))))
#+end_src