#+TITLE: System Interface (package.lisp)
#+AUTHOR: Amr
#+FILETAGS: :harness:interface:
#+STARTUP: content

* System Interface (package.lisp)
The ~package.lisp~ file defines the public API of the ~opencortex~ harness. It serves as the primary membrane between the deterministic core modules and the dynamic world of skills and actuators.

** Architectural Intent: The Package Membrane
By strictly defining the public interface, we ensure that skills remain decoupled from the harness implementation details. This allows for autonomous replacement of any component (e.g., swapping the Memory or the Probabilistic Engine) without breaking existing skills.

#+begin_src mermaid
flowchart TD
    External[Actuators / Clients] -- communication protocol --> Package[Package Membrane: API]
    Skills[Dynamic Skills] -- API Calls --> Package
    Package --> Internal[Harness Internal Modules]
    style Package fill:#f9f,stroke:#333,stroke-width:4px
#+end_src

** Public API Export
#+begin_src lisp :tangle ../library/package.lisp
(defpackage :opencortex
  (:use :cl)
  (:export 
   ;; --- Communication Protocol ---
   #:frame-message
   #:read-framed-message
   #:PROTO-GET
   #:LIST-OBJECTS-WITH-ATTRIBUTE
   #:COSINE-SIMILARITY
   #:VAULT-MASK-STRING
   #:*VAULT-MEMORY*
   #:parse-message
   #:make-hello-message
   #:validate-communication-protocol-schema
   
   ;; --- Daemon Lifecycle ---
   #:start-daemon
   #:stop-daemon
   #:harness-log
   #:main
   
   ;; --- Memory (CLOSOS) ---
   #:ingest-ast
   #:lookup-object
   #:list-objects-by-type
   #:org-id-new
   #:*memory*
   #:*history-store*
   #:org-object
   #:make-org-object
   #:org-object-id
   #:org-object-type
   #:org-object-attributes
   #:org-object-parent-id
   #:org-object-children
   #:org-object-version
   #:org-object-last-sync
   #:org-object-vector
   #:org-object-content
   #:org-object-hash
   #:snapshot-memory
   #:rollback-memory
   
   ;; --- Context API (Peripheral Vision) ---
   #:context-query-store
   #:context-get-active-projects
   #:context-get-recent-completed-tasks
   #:context-list-all-skills
   #:context-get-skill-source
   #:context-get-system-logs
   #:context-resolve-path
   #:context-get-skill-telemetry
   #:harness-track-telemetry
   #:context-assemble-global-awareness
   
   ;; --- Reactive Signal Pipeline ---
   #:process-signal
   #:perceive-gate
   #:probabilistic-gate
   #:consensus-gate
   #:act-gate
   #:reason-gate
   #:perceive-gate
   #:dispatch-gate
   #:inject-stimulus
   #:initialize-actuators
   #:dispatch-action
   #:register-actuator
   
   ;; --- Skill Engine ---
   #:load-skill-from-org
   #:initialize-all-skills
   #:load-skill-with-timeout
   #:topological-sort-skills
   #:validate-lisp-syntax
   #:defskill
   #:*skills-registry*
   #:skill
   #:skill-name
   #:skill-priority
   #:skill-dependencies
   #:skill-trigger-fn
   #:skill-probabilistic-prompt
   #:skill-deterministic-fn

   ;; --- Tool Registry ---
   #:def-cognitive-tool
   #:*cognitive-tools*
   #:cognitive-tool
   #:cognitive-tool-name
   #:cognitive-tool-description
   #:cognitive-tool-parameters
   #:cognitive-tool-guard
   #:cognitive-tool-body

   ;; --- Emacs Client Registry ---
   #:*emacs-clients*
   #:*clients-lock*
   #:register-emacs-client
   #:unregister-emacs-client

   ;; --- Probabilistic Engine ---
   #:ask-probabilistic
   #:register-probabilistic-backend
   #:distill-prompt
   #:*provider-cascade*
   
   ;; --- Security Vault ---
   #:vault-get-secret
   #:vault-set-secret
   
   ;; --- Deterministic Logic ---
   #:list-objects-with-attribute
   #:deterministic-verify
   
   ;; --- AST Helpers ---
   #:find-headline-missing-id))
#+end_src

** Package Implementation Initialization
Ensuring the compiler enters the correct namespace for all subsequent definitions.
#+begin_src lisp :tangle ../library/package.lisp
(in-package :opencortex)
#+end_src

* System State Management
The package layer manages the core data structures that represent the live state of the harness.

** Harness Logging State
OpenCortex maintains a thread-safe circular log buffer. This is critical for two reasons:
1. *Neural Introspection:* The probabilistic engine can read the recent system logs to understand why an action failed.
2. *Real-time Debugging:* Clients can subscribe to a live log stream without needing to read the physical log file.

#+begin_src lisp :tangle ../library/package.lisp
(defvar *system-logs* nil
  "Thread-safe list of the most recent system messages.")
#+end_src

#+begin_src lisp :tangle ../library/package.lisp
(defvar *logs-lock* (bt:make-lock "harness-logs-lock")
  "Protects the circular log buffer from race conditions during concurrent skill execution.")
#+end_src

#+begin_src lisp :tangle ../library/package.lisp
(defvar *max-log-history* 100
  "The maximum number of entries to preserve in the in-memory log buffer.")
#+end_src

** Skills Registry
All Literate Skills, once compiled, are registered here. This allows for topological sorting and priority-based execution.

#+begin_src lisp :tangle ../library/package.lisp
(defvar *skills-registry* (make-hash-table :test 'equal)
  "Global registry of all loaded skills, keyed by their unique identifier.")
#+end_src

** Skill Telemetry State
To ensure the system remains performant and reliable, the harness tracks execution metrics for every skill.

#+begin_src lisp :tangle ../library/package.lisp
(defvar *skill-telemetry* (make-hash-table :test 'equal)
  "Stores execution duration and failure counts for every registered skill.")
#+end_src

#+begin_src lisp :tangle ../library/package.lisp
(defvar *telemetry-lock* (bt:make-lock "harness-telemetry-lock")
  "Protects the telemetry store from concurrent updates.")
#+end_src

* Support Functions

** Protocol Property Access (proto-get)
Lisp keywords can be inconsistent between capitalized and lowercase versions depending on the client (e.g., Emacs vs. Python socket). ~proto-get~ provides a robust abstraction to ensure the system correctly extracts values regardless of keyword casing.

#+begin_src lisp :tangle ../library/package.lisp
(defun proto-get (plist key)
  "Robustly retrieves a value from a plist, checking both uppercase and lowercase keyword versions."
  (let* ((s (string key))
         (up (intern (string-upcase s) :keyword))
         (dn (intern (string-downcase s) :keyword)))
    (or (getf plist up) (getf plist dn))))
#+end_src

** Telemetry Tracking
The ~harness-track-telemetry~ function provides the hook for the metabolic loop to report performance data.

#+begin_src lisp :tangle ../library/package.lisp
(defun harness-track-telemetry (skill-name duration status)
  "Updates performance metrics for a specific skill. Status should be :success or :rejected."
  (when skill-name 
    (bt:with-lock-held (*telemetry-lock*)
      (let ((entry (or (gethash skill-name *skill-telemetry*) (list :executions 0 :total-time 0 :failures 0))))
        (incf (getf entry :executions)) 
        (incf (getf entry :total-time) duration)
        (when (eq status :rejected) (incf (getf entry :failures))) 
        (setf (gethash skill-name *skill-telemetry*) entry)))))
#+end_src

* Cognitive Tooling System
The Tool Registry is the agent's physical interface. It separates the /proposal/ of an action from its /execution/.

** Tool Structure
#+begin_src lisp :tangle ../library/package.lisp
(defvar *cognitive-tools* (make-hash-table :test 'equal)
  "The active set of physical capabilities available to the agent.")
#+end_src

#+begin_src lisp :tangle ../library/package.lisp
(defstruct cognitive-tool
  "Represents a physical or virtual capability with explicit documentation and security guards."
  name
  description
  parameters
  guard
  body)
#+end_src

** Tool Registration Macro (def-cognitive-tool)
We use a macro to ensure that tools are consistently registered and accessible to the LLM's "tool-belt" prompt generator.

#+begin_src lisp :tangle ../library/package.lisp
(defmacro def-cognitive-tool (name description parameters &key guard body)
  "Registers a new cognitive tool. 
   NAME: Keyword identifier.
   DESCRIPTION: Human-readable intent (used in LLM prompts).
   PARAMETERS: List of property lists defining arguments.
   GUARD: (context -> boolean) function to prevent unsafe calls.
   BODY: The actual Lisp execution logic."
  `(setf (gethash (string-downcase (string ',name)) *cognitive-tools*)
         (make-cognitive-tool :name (string-downcase (string ',name))
                              :description ,description
                              :parameters ',parameters
                              :guard ,guard
                              :body ,body)))
#+end_src

* Logging Implementation

** Centralized Logging (harness-log)
The primary mechanism for system transparency. It ensures all activity is both visible to the user and recorded for neural reasoning.

#+begin_src lisp :tangle ../library/package.lisp
(defun harness-log (msg &rest args)
  "Centralized logging for the harness. Writes to STDOUT and the thread-safe circular buffer."
  (let ((formatted-msg (apply #'format nil msg args)))
    (bt:with-lock-held (*logs-lock*)
      (push formatted-msg *system-logs*)
      (when (> (length *system-logs*) *max-log-history*)
        (setq *system-logs* (subseq *system-logs* 0 *max-log-history*))))
    (format t "~a~%" formatted-msg)
    (finish-output)))
#+end_src