memex/notes/org-skill-performance-auditor.org

SKILL: Autonomous Performance Auditor (Universal Literate Note)

• Overview
• Phase A: Demand (PRD)
  • 1. Purpose
  • 2. User Needs
  • 3. Success Criteria
    • Failure rate calculation logic verification
    • Autonomous stimulus injection for Scribe-RCA skill
    • Verified logging of audit results in kernel history
• Phase B: Blueprint (PROTOCOL)
  • 1. Architectural Intent
  • 2. Semantic Interfaces
• Phase D: Build (Implementation)
  • Diagnostic Logic
• Registration

Overview

The Autonomous Performance Auditor is the system's "Quality Control" agent. It monitors the internal `*skill-telemetry*` registry to identify skills with high failure rates or excessive latency. When a performance threshold is breached, it autonomously triggers the Scribe-RCA role to analyze the failure and record it in the Institutional Memory.

Phase A: Demand (PRD)

1. Purpose

Define automated behaviors for system-wide skill performance monitoring and failure alerting.

2. User Needs

• Continuous Monitoring: Analyze skill metrics (executions, failures, latency) on every heartbeat.
• Threshold Alerts: Detect skills with failure rates exceeding a defined limit (e.g., >20%).
• Loop Closure: Autonomously trigger Root Cause Analysis (RCA) for offending skills.
• Transparency: Log audit results to the kernel history for user visibility.

3. Success Criteria

TODO Failure rate calculation logic verification

TODO Autonomous stimulus injection for Scribe-RCA skill

TODO Verified logging of audit results in kernel history

Phase B: Blueprint (PROTOCOL)

1. Architectural Intent

Interfaces for telemetry inspection and diagnostic dispatch. Source of truth is the kernel's internal `*skill-telemetry*` hash table.

2. Semantic Interfaces

(defun audit-calculate-failure-rate (skill-name)
  "Returns the percentage of failed executions for a given skill.")

(defun audit-scan-all-skills ()
  "Iterates through the telemetry registry and identifies failing components.")

Phase D: Build (Implementation)

Diagnostic Logic

(defun audit-calculate-failure-rate (skill-name)
  (let ((metrics (org-agent:context-get-skill-telemetry skill-name)))
    (if (and metrics (> (getf metrics :executions) 0))
        (* 100 (/ (getf metrics :failures) (getf metrics :executions)))
        0)))

(defun audit-scan-all-skills ()
  (let ((failing-skills '()))
    (dolist (skill-info (org-agent:context-list-all-skills))
      (let* ((name (getf skill-info :name))
             (rate (audit-calculate-failure-rate name)))
        (when (> rate 20) ; Threshold: 20% failure rate
          (kernel-log "AUDITOR - FAILURE DETECTED: Skill '~a' is failing at ~a%" name rate)
          (push name failing-skills))))
    failing-skills))

Registration

(defskill :skill-performance-auditor
  :priority 95 ; High-priority meta-cognition
  :trigger (lambda (context) (eq (getf (getf context :payload) :sensor) :heartbeat))
  :neuro (lambda (context) nil)
  :symbolic (lambda (action context) 
              (let ((failing (audit-scan-all-skills)))
                (dolist (name failing)
                  ;; Trigger Scribe-RCA for each failing skill
                  (org-agent:inject-stimulus 
                   `(:type :EVENT :payload (:sensor :audit-failure :skill ,name)))))))