:PROPERTIES: :ID: state-persistence-skill :CREATED: [2026-04-09 Thu] :END: #+TITLE: SKILL: State Persistence Layer (Universal Literate Note) #+STARTUP: content #+FILETAGS: :memory:persistence:closos:ipfs:autonomy: * Overview The *State Persistence Layer* ensures the durability and autonomousty of the agent's memory. It unifies local, high-performance Lisp image dumps with decentralized, immutable IPFS checkpointing. This dual-path approach provides both rapid operational recovery and long-term historical integrity. ** Deep Reasoning: Protection Against External Tampering While the *Prover* and *Bouncer* protect against internal skill failures, the Merkle-Tree architecture within the State Layer protects against **External Threats** (e.g., a hacker or virus modifying your `.org` files directly on disk). 1. **Skill Hashing:** Every code block and headline in a skill file has a unique Merkle hash recorded in the Memory. 2. **Integrity Verification:** Upon loading or reloading a skill, the harness re-calculates the hash and compares it against the "known good" state in the Merkle Tree. 3. **Automatic Lockdown:** If a file has been tampered with externally, the hash mismatch triggers an immediate lockdown. the harness refuses to execute the skill and alerts the Autonomous via Signal/Telegram. * Phase A: Demand (PRD) :PROPERTIES: :STATUS: SIGNED :END: ** 1. Purpose Define automated behaviors for knowledge graph serialization, local persistence, and decentralized archival. ** 2. User Needs - *Instant Recall:* Rapid local loading of the Memory from a persistent image. - *Decentralized Archival:* Pushing immutable snapshots to IPFS for cross-node sync and autonomousty. - *Merkle Integrity:* Every save operation must respect and record the Merkle-Tree history. - *Safety:* Sanitize and validate data during restoration to prevent code injection. * Phase B: Blueprint (PROTOCOL) :PROPERTIES: :STATUS: SIGNED :END: ** 1. Architectural Intent The persistence layer acts as a bridge between the volatile RAM-resident Memory and permanent storage backends. It provides two adapters: `LOCAL` (fast, SBCL-native) and `IPFS` (autonomous, content-addressed). ** 2. Semantic Interfaces #+begin_src lisp (defun persistence-dump-local () "Serializes RAM state to a local Lisp image file.") (defun persistence-push-ipfs () "Pushes an immutable snapshot of the graph to IPFS.") (defun persistence-restore-ipfs (cid) "Hydrates the RAM state from an IPFS content identifier.") #+end_src * Phase C: Success (QUALITY) :PROPERTIES: :STATUS: SIGNED :END: ** 1. Success Criteria - [ ] *Speed:* Local image load must be <500ms for a 10k node graph. - [ ] *Fidelity:* IPFS round-trip must result in a bit-identical Memory. - [ ] *Validation:* Restoration must block any `read-eval` reader macros in content. ** 2. TDD Plan Tests in `tests/persistence-tests.lisp` will verify the local dump/load cycle and the JSON serialization format for IPFS. * Phase D: Build (Implementation) ** Package Context #+begin_src lisp ;; Skill logic is evaluated in a jailed package by the Harness. #+end_src ** Helper: Local State Path Ensures we have a standardized location for local memory images. #+begin_src lisp (defun persistence-get-local-path () "Returns the path to the local memory image file." (let ((state-dir (or (uiop:getenv "SYSTEM_DIR") "system/"))) (merge-pathnames "state/memory-image.lisp" state-dir))) #+end_src ** Local Image Dump (persistence-dump-local) Serializes the Merkle history and current pointers to a Lisp file. #+begin_src lisp (defun persistence-dump-local () "Serializes the entire history store and current pointers to a local Lisp image." (let ((image-file (persistence-get-local-path))) (ensure-directories-exist image-file) (harness-log "PERSISTENCE - Dumping local image to ~a..." (uiop:native-namestring image-file)) (with-open-file (out image-file :direction :output :if-exists :supersede) (format out "~%") ;; 1. Dump all immutable objects in the history store (maphash (lambda (hash obj) (print `(setf (gethash ,hash *history-store*) ,obj) out)) *history-store*) ;; 2. Dump the current active pointers (maphash (lambda (id obj) (print `(setf (gethash ,id *memory*) (gethash ,(org-object-hash obj) *history-store*)) out)) *memory*)) t)) #+end_src ** Local Image Load (persistence-load-local) Restores the state from the local disk. #+begin_src lisp (defun persistence-load-local () "Loads the memory image from local disk." (let ((image-file (persistence-get-local-path))) (if (uiop:file-exists-p image-file) (progn (harness-log "PERSISTENCE - Loading local image...") (load image-file) t) (progn (harness-log "PERSISTENCE ERROR - Local image not found.") nil)))) #+end_src ** IPFS Serialization (persistence-serialize-for-archival) Converts the live `*memory*` into a list of Lisp Property Lists (Plists) for autonomous, homoiconic transport. #+begin_src lisp (defun persistence-serialize-for-archival () "Serializes the entire memory for IPFS transport as native S-Expressions." (let ((objects nil)) (maphash (lambda (id obj) (declare (ignore id)) (push (list :id (org-object-id obj) :type (org-object-type obj) :attributes (org-object-attributes obj) :content (org-object-content obj) :parent-id (org-object-parent-id obj) :children (org-object-children obj) :version (org-object-version obj) :last-sync (org-object-last-sync obj) :hash (org-object-hash obj)) objects)) *memory*) objects)) #+end_src ** IPFS Push (persistence-push-ipfs) Pushes the serialized knowledge graph to the decentralized network as a Lisp string. #+begin_src lisp (defun persistence-push-ipfs () "Serializes the store and pushes it to IPFS as a Lisp text block, returning the CID." (let* ((data (persistence-serialize-for-archival)) (lisp-payload (format nil "~s" data)) (ipfs-url "http://127.0.0.1:5001/api/v0/add")) (handler-case (let* ((response (dex:post ipfs-url :content `(("file" . ,lisp-payload)) :headers '(("Content-Type" . "multipart/form-data")))) (result-str (flexi-streams:octets-to-string response)) (start-idx (search "\"Hash\":\"" result-str)) (cid (when start-idx (let* ((val-start (+ start-idx 8)) (val-end (position #\" result-str :start val-start))) (subseq result-str val-start val-end))))) (harness-log "PERSISTENCE - Checkpoint to IPFS successful. CID: ~a" cid) cid) (error (c) (harness-log "PERSISTENCE ERROR - IPFS push failed: ~a" c) nil)))) #+end_src ** IPFS Restore (persistence-restore-ipfs) Restores the graph from IPFS, using `read-from-string` with `*read-eval* nil` to prevent injection. #+begin_src lisp (defun persistence-restore-ipfs (cid) "Fetches data from IPFS and safely hydrates the memory." (let ((ipfs-url (format nil "http://127.0.0.1:5001/api/v0/cat?arg=~a" cid))) (handler-case (let* ((response (dex:post ipfs-url)) (payload-str (flexi-streams:octets-to-string response))) (let ((*read-eval* nil)) (let ((data (read-from-string payload-str))) (clrhash *memory*) (dolist (item data) (let* ((id (getf item :id)) (obj (make-org-object :id id :type (getf item :type) :attributes (getf item :attributes) :content (getf item :content) :parent-id (getf item :parent-id) :children (getf item :children) :version (getf item :version) :last-sync (getf item :last-sync) :hash (getf item :hash)))) (setf (gethash id *memory*) obj))) (harness-log "PERSISTENCE - Restored from IPFS: ~a" cid) t))) (error (c) (harness-log "PERSISTENCE ERROR - IPFS restoration failed: ~a" c) nil)))) #+end_src ** Cognitive Tools Expose persistence capabilities to the neural Probabilistic Engine. #+begin_src lisp (progn (def-cognitive-tool :checkpoint-memory "Creates both a local image and a decentralized IPFS snapshot." nil :body (lambda (args) (declare (ignore args)) (persistence-dump-local) (let ((cid (persistence-push-ipfs))) (format nil "Local dump complete. IPFS CID: ~a" (or cid "FAILED"))))) (def-cognitive-tool :restore-memory "Restores the state from a specific source." ((:source :type :keyword :description "Either :LOCAL or :IPFS") (:cid :type :string :description "Required if source is :IPFS")) :body (lambda (args) (case (getf args :source) (:local (if (persistence-load-local) "Restored from disk." "Local restore failed.")) (:ipfs (if (persistence-restore-ipfs (getf args :cid)) "Restored from network." "IPFS restore failed.")))))) #+end_src ** Registration #+begin_src lisp (defskill :skill-state-persistence :priority 100 :trigger (lambda (ctx) (let ((sensor (getf (getf ctx :payload) :sensor))) (member sensor '(:heartbeat :manual-persist)))) :probabilistic nil :deterministic (lambda (action ctx) (persistence-dump-local) action)) #+end_src * Phase E: Chaos (Verification) ** 1. Unit Tests (FiveAM) #+begin_src lisp (defpackage :org-agent-persistence-tests (:use :cl :fiveam :org-agent)) (in-package :org-agent-persistence-tests) (def-suite persistence-suite :description "Tests for State Persistence Layer.") (in-suite persistence-suite) (test test-local-roundtrip "Ensure RAM -> Disk -> RAM preserves data integrity." (let ((test-id "persist-test-1") (test-hash "fake-hash-123")) (let ((obj (make-org-object :id test-id :content "Integrity Check" :hash test-hash))) (setf (gethash test-hash *history-store*) obj) (setf (gethash test-id *memory*) obj)) (org-agent:persistence-dump-local) (clrhash *memory*) (clrhash *history-store*) (org-agent:persistence-load-local) (is (not (null (gethash test-id *memory*)))) (is (equal "Integrity Check" (org-object-content (gethash test-id *memory*)))))) #+end_src ** 2. Chaos Scenarios - *Scenario A (IPFS Daemon Down):* Kill the IPFS daemon and verify `persistence-push-ipfs` returns a standardized error instead of hanging the harness. - *Scenario B (Corrupt Image):* Intentionally mangle the `memory-image.lisp` file and verify the loader catches the error during `load` and falls back to a clean state. * Phase F: Memory (RCA) - *[2026-04-09 Thu]:* Unified local SBCL image dumps with IPFS decentralized snapshots. Implemented safety-first restoration logic.