#+TITLE: The Skill Engine (skills.lisp)
#+AUTHOR: Amr
#+FILETAGS: :harness:skills:
#+STARTUP: content

* The Skill Engine (skills.lisp)
** Architectural Intent: Late-Binding Intelligence

A static, hardcoded architecture is inherently fragile. To build a autonomous agent that can evolve alongside its user, the harness must be a "Thin Shell" that delegates its capabilities to dynamic, hot-reloadable modules known as **Skills**. This is the core of our **Thin Harness / Thick Skill Microkernel Architecture**.

Skills unify the **"Why"** (Literate Org documentation) and the **"How"** (Functional Lisp implementation). This allows the harness to "learn" new behaviors without a full system restart, enabling a continuous evolutionary loop where the agent can eventually inspect and improve its own code.

*** The True Microkernel (Decoupled Core Skills)
Historically, "core" skills (like State Persistence or Gateways) were statically compiled into the harness for performance. We have fundamentally decoupled this. Now, *all* behavioral skills are pure user-space dynamic modules. 

**** MANDATE: Dynamic Loading (No Tangling)
Skills are defined as single-file Literate Org notes. Unlike the core harness, **Skills MUST NOT use :tangle headers.** 

Instead of being compiled into static source files, skills are:
1. **Discovered:** The harness scans the `skills/` directory for `.org` files.
2. **Parsed:** The harness extracts Lisp code blocks directly from the Org AST.
3. **Jailed:** Each skill is evaluated inside its own temporary, isolated package namespace.
4. **Hot-Loaded:** Skills can be added, modified, or removed at runtime without restarting the Lisp image.

This ensures the agent can safely read, write, and repair its own capabilities without breaking the physical file structure. If a user wishes to swap the IPFS persistence skill for an AWS S3 one, they simply swap the `.org` file; no kernel recompilation is required.

*** Dormant Verification (Tests)
Because skills are no longer statically compiled into the core `opencortex` ASDF system, their associated `FiveAM` test blocks are currently dormant during a standard static build. The tests still exist within the literate `.org` files as verifiable documentation, but executing them requires either dynamic evaluation at runtime or a dedicated test-loader skill.

*** 1. The Package Jailing Principle
Every skill is evaluated within its own dedicated Common Lisp package (namespace). This "Jailing" prevents symbol collisions between disparate skills and ensures that a bug in one module cannot easily corrupt the internal state of another.

*** 2. Deterministic Load Ordering
Skills often depend on one another. The harness implements a deterministic topological sorting algorithm to ensure that dependencies are loaded before the skills that require them.

** Skill Architecture
#+begin_src mermaid
flowchart TD
    Registry[Skills Registry] --> S1[Skill: System Policy]
    Registry --> S2[Skill: LLM Gateway]
    Registry --> S3[Skill: Token Accountant]
    S2 -- Depends On --> S1
    S3 -- Depends On --> S2
    
    subgraph Jailing[Package Jailing]
        P1[Package: OPENCORTEX.SKILLS.S1]
        P2[Package: OPENCORTEX.SKILLS.S2]
        P3[Package: OPENCORTEX.SKILLS.S3]
    end
    
    S1 --> P1
    S2 --> P2
    S3 --> P3
#+end_src

** Package Context
We begin by ensuring we are in the correct isolated harness namespace.

#+begin_src lisp :tangle ../src/skills.lisp
(in-package :opencortex)
#+end_src

** Skill Metadata (defstruct skill)
The core data structure representing an agent capability. It includes the trigger condition, the probabilistic prompt generator, and the deterministic safety gate.

#+begin_src lisp :tangle ../src/skills.lisp
(defstruct skill name priority dependencies trigger-fn probabilistic-prompt deterministic-fn)
#+end_src

** Skill Catalog Tracking
The harness maintains a stateful tracking table for all skill files discovered in the environment (~notes/org-skill-*.org~).

#+begin_src lisp :tangle ../src/skills.lisp
(defvar *skill-catalog* (make-hash-table :test 'equal)
  "A stateful tracking table for all skill files discovered in the environment.")

(defstruct skill-entry 
  filename 
  (status :discovered) ;; :discovered, :loading, :ready, :failed
  error-log
  (load-time 0))
#+end_src

** Skill Selection (find-triggered-skill)
The primary dispatcher for the Probabilistic Engine. It iterates through the registry to find the highest-priority skill whose trigger function matches the current cognitive context.

#+begin_src lisp :tangle ../src/skills.lisp
(defun find-triggered-skill (context)
  "Returns the highest priority skill whose trigger condition matches the context."
  (let ((triggered nil))
    (maphash (lambda (name skill) 
               (declare (ignore name)) 
               (when (ignore-errors (funcall (skill-trigger-fn skill) context)) 
                 (push skill triggered))) 
             *skills-registry*)
    (first (sort triggered #'> :key #'skill-priority))))
#+end_src

** Skill Definition Macro (defskill)
The interface used within Org files to register new capabilities. Note that dependencies are explicitly quoted to prevent premature evaluation during the macro expansion phase.

#+begin_src lisp :tangle ../src/skills.lisp
(defmacro defskill (name &key priority dependencies trigger probabilistic deterministic)
  "Registers a new skill into the global registry."
  `(setf (gethash (string-downcase (string ,name)) *skills-registry*)
         (make-skill :name (string-downcase (string ,name)) 
                     :priority (or ,priority 10) 
                     :dependencies ',dependencies
                     :trigger-fn ,trigger 
                     :probabilistic-prompt ,probabilistic 
                     :deterministic-fn ,deterministic)))
#+end_src

** Dependency Resolution (resolve-skill-dependencies)
Recursively flattens the dependency graph for a given skill. This is used during hot-unloading to ensure that dependent skills are also refreshed.

#+begin_src lisp :tangle ../src/skills.lisp
(defun resolve-skill-dependencies (skill-name)
  "Recursively resolves dependencies for a given skill name."
  (let ((resolved nil) (seen nil))
    (labels ((visit (name) 
               (unless (member name seen :test #'equal) 
                 (push name seen)
                 (let ((skill (gethash (string-downcase (string name)) *skills-registry*)))
                   (when skill 
                     (dolist (dep (skill-dependencies skill)) 
                       (visit dep))))
                 (push name resolved))))
      (visit skill-name) 
      (nreverse resolved))))
#+end_src

** Metadata Parsing (parse-skill-metadata)
A robust, low-level scanner that extracts `#+DEPENDS_ON:` and `:ID:` tags from an Org file. This allows the harness to calculate the load order without needing to parse the full Org AST, which would create a boot-time circularity.

#+begin_src lisp :tangle ../src/skills.lisp
(defun parse-skill-metadata (filepath)
  "Extracts ID and DEPENDS_ON tags using robust regex scanning."
  (let ((dependencies nil)
        (id nil)
        (content (uiop:read-file-string filepath)))
    ;; Extract ID
    (multiple-value-bind (match regs)
        (ppcre:scan-to-strings "(?im:^:ID:\\s*([^\\s\\r\\n]+))" content)
      (when match (setf id (aref regs 0))))
    ;; Extract all DEPENDS_ON lines
    (ppcre:do-register-groups (deps-string)
        ("(?im:^#\\+DEPENDS_ON:\\s*(.*))" content)
      (let ((deps (ppcre:split "\\s+" (string-trim " " deps-string))))
        (setf dependencies (append dependencies (mapcar (lambda (s) (string-trim "[] " s)) deps)))))
    (values id (remove-if (lambda (s) (= 0 (length s))) dependencies))))
#+end_src

** Topological Sorting (topological-sort-skills)
This is the core algorithm of the boot sequence. It uses a **Depth-First Search (DFS)** to resolve the skill dependency graph. 

It performs three critical roles:
1.  **Load Ordering:** Ensures that "foundational" skills (like the LLM Gateway) are loaded before high-level "behavioral" skills.
2.  **ID Resolution:** Correct maps Org `:ID:` properties to filepaths.
3.  **Cycle Detection:** It uses a recursion stack to detect circular dependencies (e.g., A depends on B, B depends on A) and errors out safely before the Lisp image hangs.

#+begin_src lisp :tangle ../src/skills.lisp
(defun topological-sort-skills (skills-dir)
  "Returns a list of skill filepaths sorted by dependency (dependencies first)."
  (let ((files (uiop:directory-files skills-dir "org-skill-*.org"))
        (adj (make-hash-table :test 'equal))
        (name-to-file (make-hash-table :test 'equal))
        (id-to-file (make-hash-table :test 'equal))
        (result nil)
        (visited (make-hash-table :test 'equal))
        (stack (make-hash-table :test 'equal)))
    (dolist (file files)
      (let ((filename (pathname-name file)))
        (multiple-value-bind (id deps) (parse-skill-metadata file)
          (setf (gethash (string-downcase filename) name-to-file) file)
          (when id (setf (gethash (string-downcase id) id-to-file) file))
          (setf (gethash (string-downcase filename) adj) deps))))
    (labels ((visit (file)
               (let* ((filename (pathname-name file))
                      (node-key (string-downcase filename)))
                 (unless (gethash node-key visited)
                   (setf (gethash node-key stack) t)
                   (dolist (dep (gethash node-key adj))
                     (let* ((is-id-p (uiop:string-prefix-p "id:" (string-downcase dep)))
                            (dep-key (string-downcase (if is-id-p (subseq dep 3) dep)))
                            (dep-file (if is-id-p 
                                          (gethash dep-key id-to-file)
                                          (or (gethash dep-key id-to-file)
                                              (gethash dep-key name-to-file)))))
                       (when dep-file
                         (let ((dep-filename (pathname-name dep-file)))
                           (if (gethash (string-downcase dep-filename) stack)
                               (error "Circular dependency detected: ~a -> ~a" filename dep-filename)
                               (visit dep-file))))))
                   (setf (gethash node-key stack) nil)
                   (setf (gethash node-key visited) t)
                   (push file result)))))
      (let ((filenames (sort (mapcar #'pathname-name files) #'string<)))
        (dolist (name filenames)
          (let ((file (gethash (string-downcase name) name-to-file)))
            (when file (visit file)))))
      (nreverse result))))
#+end_src

** Syntax Validation (validate-lisp-syntax)
A pre-flight safety check. Before evaluating any code from an Org file, the harness attempts to ~read~ the entire string. If the reader encounters a syntax error (like an unclosed parenthesis), the load is aborted before the Lisp image can crash.

#+begin_src lisp :tangle ../src/skills.lisp
(defun validate-lisp-syntax (code-string)
  "Checks if a string contains valid, readable Common Lisp forms."
  (handler-case 
      (let ((*read-eval* nil)) 
        (with-input-from-string (stream (format nil "(progn ~a)" code-string))
          (loop for form = (read stream nil :eof) until (eq form :eof)) 
          (values t nil)))
    (error (c) (values nil (format nil "~a" c)))))
#+end_src

** Jailed Loading (load-skill-from-org)
The core "hot-loading" mechanism. It extracts Lisp blocks from an Org file and evaluates them within a "Jail" (an isolated package).

*** The Jailing Algorithm:
1.  **Isolation:** It generates a package name based on the filename (e.g., ~OPENCORTEX.SKILLS.CORE-LOGIC~).
2.  **Namespace Protection:** It inherits external symbols from the ~OPENCORTEX~ package, allowing the skill to use the harness API, but keeps its internal helper functions local.
3.  **Block Filtering:** It explicitly ignores blocks that contain ~:tangle~, ensuring that harness-level code (which is already in ~src/~) is not accidentally re-evaluated as skill logic.

#+begin_src lisp :tangle ../src/skills.lisp
(defun load-skill-from-org (filepath)
  "Parses and evaluates Lisp blocks from an Org file into a jailed package."
  (let* ((skill-base-name (pathname-name filepath))
         (entry (or (gethash skill-base-name *skill-catalog*) (make-skill-entry :filename skill-base-name))))
    (setf (skill-entry-status entry) :loading)
    (setf (gethash skill-base-name *skill-catalog*) entry)
    
    (handler-case
        (let* ((content (uiop:read-file-string filepath)) 
               (lines (uiop:split-string content :separator '(#\Newline)))
               (in-lisp-block nil) 
               (lisp-code "") 
               (pkg-name (intern (string-upcase (format nil "OPENCORTEX.SKILLS.~a" skill-base-name)) :keyword)))
          
          (dolist (line lines)
            (let ((clean-line (string-trim '(#\Space #\Tab #\Return) line)))
              (cond ((uiop:string-prefix-p "#+begin_src lisp" (string-downcase clean-line))
                     ;; Only load blocks that are NOT tangled to src/ or elsewhere
                     (if (search ":tangle" (string-downcase clean-line))
                         (setf in-lisp-block nil)
                         (setf in-lisp-block t)))
                    ((uiop:string-prefix-p "#+end_src" (string-downcase clean-line))
                     (setf in-lisp-block nil))
                    (in-lisp-block 
                     (unless (or (uiop:string-prefix-p ":PROPERTIES:" (string-upcase clean-line))
                                 (uiop:string-prefix-p ":END:" (string-upcase clean-line)))
                       (setf lisp-code (concatenate 'string lisp-code line (string #\Newline))))))))
          
          (if (= (length lisp-code) 0)
              (progn (setf (skill-entry-status entry) :ready) t) ;; Valid empty skill
              (progn
                ;; PRE-FLIGHT: Syntax Validation
                (multiple-value-bind (valid-p err) (validate-lisp-syntax lisp-code)
                  (unless valid-p
                    (error "Syntax Error: ~a" err)))
                
                (harness-log "HARNESS: Jailing skill '~a' in package ~a" skill-base-name pkg-name)
                (unless (find-package pkg-name)
                  (let ((new-pkg (make-package pkg-name :use '(:cl))))
                    (do-external-symbols (sym (find-package :opencortex)) (shadowing-import sym new-pkg))))
                
                (let ((*read-eval* nil) (*package* (find-package pkg-name)))
                  (eval (read-from-string (format nil "(progn ~a)" lisp-code))))
                
                (setf (skill-entry-status entry) :ready)
                t)))
      (error (c)
        (let ((msg (format nil "~a" c)))
          (harness-log "LOADER ERROR in skill '~a': ~a" skill-base-name msg)
          (setf (skill-entry-status entry) :failed)
          (setf (skill-entry-error-log entry) msg)
          nil)))))
#+end_src

** Safe Loading with Timeout (load-skill-with-timeout)
Wraps the skill loader in a thread with a hard timeout to prevent a single malformed skill from hanging the entire boot sequence.

#+begin_src lisp :tangle ../src/skills.lisp
(defun load-skill-with-timeout (filepath timeout-seconds)
  "Loads a skill Org file with a hard execution timeout."
  (let* ((finished nil)
         (thread (bt:make-thread (lambda () 
                                   (if (load-skill-from-org filepath)
                                       (setf finished t)
                                       (setf finished :error)))
                                 :name (format nil "loader-~a" (pathname-name filepath))))
         (start-time (get-internal-real-time))
         (timeout-units (truncate (* timeout-seconds internal-time-units-per-second))))
    (loop 
      (when (eq finished t) (return :success))
      (when (eq finished :error) (return :error))
      (unless (bt:thread-alive-p thread) (return :error))
      (when (> (- (get-internal-real-time) start-time) timeout-units)
        (harness-log "HARNESS: Timing out skill ~a..." (pathname-name filepath))
        #+sbcl (sb-thread:terminate-thread thread)
        #-sbcl (bt:destroy-thread thread)
        (return :timeout))
      (sleep 0.05))))
#+end_src

** Initializing All Skills (initialize-all-skills)
The `initialize-all-skills` function is the unified orchestrator for the system boot sequence. It enforces the **Verification Lock**:
1. **Mandatory Check:** It reads the `MANDATORY_SKILLS` environment variable and ensures every required skill exists in the source directory.
2. **Topological Boot:** It resolves inter-skill dependencies to ensure policies and actuators are loaded in the correct order.
3. **Timed Loading:** Every skill is loaded with a 5-second timeout.
4. **Boot Halt:** If a *mandatory* skill fails to load (e.g., due to a syntax error), the entire system halts with a `BOOT FAILURE` to prevent an unaligned or unsecure state.

#+begin_src lisp :tangle ../src/skills.lisp
(defun initialize-all-skills ()
  "Scans the directory defined by SKILLS_DIR and hot-loads skills using topological order."
  (let* ((env-path (uiop:getenv "SKILLS_DIR"))
         (skills-dir-str (or env-path (namestring (merge-pathnames "notes/" (user-homedir-pathname)))))
         (resolved-path (context-resolve-path skills-dir-str))
         (skills-dir (if resolved-path (uiop:ensure-directory-pathname resolved-path) nil)))
    
    (unless (and skills-dir (uiop:directory-exists-p skills-dir))
      (harness-log "HARNESS ERROR: Skills directory not found: ~a" skills-dir-str)
      (return-from initialize-all-skills nil))

    (let ((sorted-files (topological-sort-skills skills-dir)))
      ;; MANDATE: Configurable mandatory skills must be present for a safe boot
      (let* ((mandatory-env (uiop:getenv "MANDATORY_SKILLS"))
             (mandatory-skills (if mandatory-env 
                                   (mapcar (lambda (s) (string-trim '(#\Space) s)) 
                                           (uiop:split-string mandatory-env :separator '(#\,)))
                                   '("org-skill-policy" "org-skill-bouncer"))))
        (dolist (req mandatory-skills)
          (unless (member req sorted-files :key #'pathname-name :test #'string-equal)
            (error "BOOT FAILURE: Mandatory skill '~a' not found in skills directory." req)))
      
        (harness-log "==================================================")
        (harness-log " LOADER: Initializing ~a skills..." (length sorted-files))
      
        (dolist (file sorted-files)
          (let* ((skill-name (pathname-name file))
                 (is-mandatory (member skill-name mandatory-skills :test #'string-equal)))
            (harness-log " LOADER: Loading ~a..." skill-name)
            (let ((status (load-skill-with-timeout file 5)))
              (unless (eq status :success)
                (if is-mandatory
                    (error "BOOT FAILURE: Mandatory skill '~a' failed to load (Status: ~a)." skill-name status)
                    (harness-log "LOADER WARNING: Skill '~a' failed to load." skill-name))))))
      
        ;; Final Summary
        (let ((ready 0) (failed 0))
          (maphash (lambda (k v) 
                     (declare (ignore k))
                     (if (eq (skill-entry-status v) :ready) (incf ready) (incf failed)))
                   *skill-catalog*)
          (harness-log " LOADER: Boot Complete. [Ready: ~a] [Failed: ~a]" ready failed)
          (harness-log "==================================================")
          (values ready failed))))))
#+end_src

** Toolbelt Prompt Generation (generate-tool-belt-prompt)
Every cognitive tool registered by a skill is automatically documented and injected into the LLM system prompt. This ensures that the agent is always aware of its current physical capabilities.

#+begin_src mermaid
flowchart LR
    Registry[(Tool Registry)] --> Generator[Prompt Generator]
    Generator --> Prompt[Final System Prompt]
    subgraph Actuators
        ToolA[Shell]
        ToolB[Emacs]
        ToolC[Grep]
    end
    ToolA --> Registry
    ToolB --> Registry
    ToolC --> Registry
#+end_src

#+begin_src lisp :tangle ../src/skills.lisp
(defun generate-tool-belt-prompt ()
  "Aggregates all registered cognitive tools into a descriptive prompt."
  (let ((output (format nil "AVAILABLE TOOLS:
You can call tools by returning a Lisp plist: (:target :tool :action :call :tool <name> :args (...))

EXAMPLES:
(:target :tool :action :call :tool \"eval\" :args (:code \"(+ 1 1)\"))
(:target :tool :action :call :tool \"grep-search\" :args (:pattern \"autonomousty\"))
(:target :tool :action :call :tool \"shell\" :args (:cmd \"ls -la\"))

---
")))
    (maphash (lambda (name tool)
               (setf output (concatenate 'string output
                                         (format nil "- ~a: ~a~%  Parameters: ~s~%~%"
                                                 name
                                                 (cognitive-tool-description tool)
                                                 (cognitive-tool-parameters tool)))))
             *cognitive-tools*)
    output))
#+end_src

** The Default Tool Belt
The harness provides a baseline set of cognitive tools that enable core system interaction.

*** The Eval Tool (Internal Inspection)
#+begin_src lisp :tangle ../src/skills.lisp
(def-cognitive-tool :eval "Evaluates raw Common Lisp code in the harness image. Use this for complex calculations or internal state inspection."
  ((:code :type :string :description "The Lisp code to evaluate"))
  :guard (lambda (args context)
           (declare (ignore context))
           (let ((code (getf args :code)))
             (let ((harness-pkg (find-package :opencortex.skills.org-skill-lisp-validator)))
               (if harness-pkg 
                   (uiop:symbol-call :opencortex.skills.org-skill-lisp-validator :lisp-validator-validate code)
                   t))))
  :body (lambda (args)
          (let ((code (getf args :code)))
            (handler-case (let ((result (eval (read-from-string code))))
                            (format nil "~s" result))
              (error (c) (format nil "ERROR: ~a" c))))))
#+end_src

*** The Grep Tool (File Discovery)
#+begin_src lisp :tangle ../src/skills.lisp
(def-cognitive-tool :grep-search "Searches for a pattern in the project files."
  ((:pattern :type :string :description "The regex pattern to search for")
   (:dir :type :string :description "Directory to search in (default is project root)"))
  :body (lambda (args)
          (let ((pattern (getf args :pattern))
                (dir (or (getf args :dir) (uiop:getenv "MEMEX_DIR"))))
            (uiop:run-program (list "grep" "-r" "-n" "--exclude-dir=node_modules" pattern dir) 
                              :output :string :ignore-error-status t))))
#+end_src

*** The Shell Tool (Machine Actuation)
#+begin_src lisp :tangle ../src/skills.lisp
(def-cognitive-tool :shell "Executes a shell command on the local machine. Use this for file operations, system checks, or running tests."
  ((:cmd :type :string :description "The full bash command to execute"))
  :guard (lambda (args context)
           (declare (ignore context))
           (let ((cmd (getf args :cmd)))
             (not (or (search "rm -rf /" cmd) (search ":(){ :|:& };:" cmd)))))
  :body (lambda (args)
          (let ((cmd (getf args :cmd)))
            (multiple-value-bind (out err code)
                (uiop:run-program (list "bash" "-c" cmd) :output :string :error-output :string :ignore-error-status t)
              (format nil "EXIT-CODE: ~a~%~%STDOUT:~%~a~%~%STDERR:~%~a" code out err)))))
#+end_src
