From c9cc874e53fb04b1346a3af56dff92db7276da93 Mon Sep 17 00:00:00 2001 From: Amr Gharbeia Date: Wed, 13 May 2026 12:54:38 -0400 Subject: [PATCH] tools: add repl-block, check-tangle; move existing tools into memex MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New tools (projects// — standalone, git-committed): - repl-block: extract and pipe lisp blocks from org files to the REPL - check-tangle: tangle + compile in one step, reports errors Existing tools moved from ~/.opencode/bin/ into memex (survives reinstalls): - repl, tangle, org-eval, verify-repl AGENTS.md updated: - Tool reference table with all 7 tools - Package reference table for passepartout and cl-tty - Updated tangle command to use project-local tools .opencode/commands/ added: check-parens, repl-block, check-tangle commands --- .opencode/commands/check-parens.md | 8 ++ .opencode/commands/check-tangle.md | 8 ++ .opencode/commands/repl-block.md | 13 +++ projects/AGENTS.md | 47 ++++++++- projects/check-tangle/check-tangle | 62 ++++++++++++ projects/org-eval-tool/org-eval | 43 ++++++++ projects/repl-block/repl-block | 108 ++++++++++++++++++++ projects/repl-tool/repl | 74 ++++++++++++++ projects/tangle-tool/tangle | 27 +++++ projects/verify-repl-tool/verify-repl | 140 ++++++++++++++++++++++++++ 10 files changed, 526 insertions(+), 4 deletions(-) create mode 100644 .opencode/commands/check-parens.md create mode 100644 .opencode/commands/check-tangle.md create mode 100644 .opencode/commands/repl-block.md create mode 100755 projects/check-tangle/check-tangle create mode 100755 projects/org-eval-tool/org-eval create mode 100755 projects/repl-block/repl-block create mode 100755 projects/repl-tool/repl create mode 100755 projects/tangle-tool/tangle create mode 100755 projects/verify-repl-tool/verify-repl diff --git a/.opencode/commands/check-parens.md b/.opencode/commands/check-parens.md new file mode 100644 index 0000000..54b9cbc --- /dev/null +++ b/.opencode/commands/check-parens.md @@ -0,0 +1,8 @@ +--- +description: Check paren balance in lisp blocks of .org files +--- + +Run `projects/check-parens/check-parens` on the given .org files to verify all +`#+begin_src lisp` blocks have balanced parentheses. + +Usage: /check-parens [ ...] diff --git a/.opencode/commands/check-tangle.md b/.opencode/commands/check-tangle.md new file mode 100644 index 0000000..8b20cef --- /dev/null +++ b/.opencode/commands/check-tangle.md @@ -0,0 +1,8 @@ +--- +description: Tangle an org file and compile the result +--- + +Tangle an .org file to .lisp then compile with SBCL. Reports the first +compile error. Exit 0 = clean, exit 1 = compilation error. + +Usage: /check-tangle diff --git a/.opencode/commands/repl-block.md b/.opencode/commands/repl-block.md new file mode 100644 index 0000000..b35d8c5 --- /dev/null +++ b/.opencode/commands/repl-block.md @@ -0,0 +1,13 @@ +--- +description: Send a lisp block from an org file to the REPL +--- + +Extract a `#+begin_src lisp` block from an .org file and send it to the +running daemon REPL. + +Usage: /repl-block --function + /repl-block --block + /repl-block --function --package + +The --package flag wraps the block in an (in-package ...) form. +Use --block to identify by 1-based index, --function to find by defun name. diff --git a/projects/AGENTS.md b/projects/AGENTS.md index f9d8cc6..c235cec 100644 --- a/projects/AGENTS.md +++ b/projects/AGENTS.md @@ -125,13 +125,53 @@ goes to main. If it spans sessions or might be abandoned for a better approach, it gets a branch. +## Tools + +All tools are standalone scripts in `projects//`, committed to the memex. +They survive upgrades, reinstalls, and new clones. + +| Tool | What | Usage | +|------|------|-------| +| `check-parens` | Validate paren balance in org lisp blocks (uses SBCL reader) | `projects/check-parens/check-parens org/file.org` | +| `repl-block` | Extract and send an org lisp block to the daemon REPL | `projects/repl-block/repl-block org/file.org --function foo` | +| `check-tangle` | Tangle an org file and compile the result in SBCL | `projects/check-tangle/check-tangle org/file.org` | +| `repl` | Send lisp code to the running daemon | `projects/repl-tool/repl "(+ 1 2)"` | +| `tangle` | Tangle an org file via Emacs batch | `projects/tangle-tool/tangle org/file.org` | +| `org-eval` | Evaluate an org src block via Emacs | `projects/org-eval-tool/org-eval org/file.org` | +| `verify-repl` | Compliance: check defuns have REPL-VERIFIED comments | `projects/verify-repl-tool/verify-repl org/` | + +## Package reference + +When sending code to the REPL, use the correct `(in-package ...)` form first. + +### Passepartout + +| Package | When to use | +|---------|-------------| +| `:passepartout` | Core system — skills, gates, memory, dispatcher | +| `:passepartout.channel-tui` | TUI — event handlers, view, state | +| `-tests` | Test suites — run with `(fiveam:run (intern ...))` | + +### cl-tty + +| Package | When to use | +|---------|-------------| +| `:cl-tty.backend` | Backend protocol — terminal init, draw-text, read-event | +| `:cl-tty.rendering` | Framebuffer — make-framebuffer, flush-framebuffer | +| `:cl-tty.input` | Input — key-event, defkeymap, dispatch-key-event | +| `:cl-tty.layout` | Layout — vbox, hbox, spacer, layout-calculate | +| `:cl-tty.dialog` | Dialog system — dialog stack, select-dialog | +| `:cl-tty.select` | Select widget — filter, handle-key | +| `:cl-tty.slot` | Slot/plugin system | +| `:cl-tty.markdown` | Markdown rendering | + ## Commands Tangle a single file: - emacs --batch --eval "(progn (require 'org) (find-file \"org/FILE.org\") (org-babel-tangle) (kill-buffer))" + projects/tangle-tool/tangle org/FILE.org -Validate structural integrity (org/ source files only): - emacs --batch -Q --eval '(progn (find-file "org/FILE.org") (check-parens) (kill-buffer))' +Or: + emacs --batch --eval "(progn (require 'org) (find-file \"org/FILE.org\") (org-babel-tangle) (kill-buffer))" Run tests (from REPL): (fiveam:run (intern "SUITE-NAME" :project-TESTS)) @@ -139,7 +179,6 @@ Run tests (from REPL): Run tests (SBCL fallback — only when the runtime cannot start): sbcl --noinform \ - --eval '(load (merge-pathnames "quicklisp/setup.lisp" (user-homedir-pathname)))' \ --eval '(ql:quickload :your-project :silent t)' \ --eval '(load "lisp/FILE.lisp")' \ --eval '(fiveam:run (intern "SUITE-NAME" :project-TESTS))' --quit diff --git a/projects/check-tangle/check-tangle b/projects/check-tangle/check-tangle new file mode 100755 index 0000000..9ae00ab --- /dev/null +++ b/projects/check-tangle/check-tangle @@ -0,0 +1,62 @@ +#!/usr/bin/env bash +# Tangle an .org file and compile the resulting .lisp with SBCL. +# Reports the first compile error with line numbers. +# +# Usage: check-tangle +# Exit 0 if compile succeeds, 1 if tangled or compilation fails + +set -euo pipefail + +if [ $# -ne 1 ]; then + echo "Usage: check-tangle " >&2 + exit 2 +fi + +ORG_FILE="$1" + +if [ ! -f "$ORG_FILE" ]; then + echo "Error: File not found: $ORG_FILE" >&2 + exit 1 +fi + +# Determine the tangle target from the org file +TANGLE=$(grep -m1 'header-args.*:tangle' "$ORG_FILE" | sed 's/.*:tangle //' | head -1 || true) +if [ -z "$TANGLE" ]; then + echo "SKIP: $ORG_FILE — no :tangle header" >&2 + exit 0 +fi + +# Resolve relative tangle path +ORG_DIR=$(dirname "$ORG_FILE") +LISP_FILE=$(cd "$ORG_DIR" && realpath -m "$TANGLE" 2>/dev/null || echo "$ORG_DIR/$TANGLE") + +echo "Tangling: $ORG_FILE → $LISP_FILE" >&2 + +# Tangle using opencode's tangle tool +TANGLE_CMD=$(command -v tangle 2>/dev/null || echo "/home/user/.opencode/bin/tangle") +if ! "$TANGLE_CMD" "$ORG_FILE" 2>/dev/null; then + echo "FAIL: Tangling $ORG_FILE failed" >&2 + exit 1 +fi + +if [ ! -f "$LISP_FILE" ]; then + echo "SKIP: Tangle target $LISP_FILE not found after tangling" >&2 + exit 0 +fi + +echo "Compiling: $LISP_FILE" >&2 + +# Compile with SBCL, capturing errors +OUTPUT=$(sbcl --noinform --no-userinit --disable-debugger --quit \ + --eval "(handler-case + (progn (compile-file \"$LISP_FILE\") (print :OK)) + (error (c) (print c) (sb-ext:exit :code 1)))" 2>&1 || true) + +if echo "$OUTPUT" | grep -q ":OK"; then + echo "OK: $ORG_FILE compiles cleanly" >&2 + exit 0 +else + echo "$OUTPUT" | grep -v '^;' | grep -v '^$' | head -5 + echo "FAIL: $ORG_FILE — compilation error" >&2 + exit 1 +fi diff --git a/projects/org-eval-tool/org-eval b/projects/org-eval-tool/org-eval new file mode 100755 index 0000000..e52c7d0 --- /dev/null +++ b/projects/org-eval-tool/org-eval @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# Evaluate an org src block using Emacs batch mode +# Usage: org-eval [block-index] +# If block-index is not provided, evaluates all blocks + +set -e + +if [ $# -lt 1 ]; then + echo "Usage: org-eval [block-index]" + echo " Evaluates src blocks in the org file" + echo " If block-index is omitted, evaluates ALL blocks" + exit 1 +fi + +ORG_FILE="$1" +BLOCK_INDEX="${2:-}" + +if [ ! -f "$ORG_FILE" ]; then + echo "Error: File not found: $ORG_FILE" + exit 1 +fi + +echo "Evaluating: $ORG_FILE" + +if [ -n "$BLOCK_INDEX" ]; then + # Evaluate specific block + emacs --batch \ + --load org \ + --eval "(setq org-confirm-babel-evaluate nil)" \ + --eval "(with-current-buffer (find-file-noselect \"$ORG_FILE\") \ + (goto-char (point-min)) \ + (dotimes (_ $BLOCK_INDEX) \ + (org-babel-next-src-block)) \ + (org-babel-execute-src-block))" +else + # Evaluate all blocks + emacs --batch \ + --load org \ + --eval "(setq org-confirm-babel-evaluate nil)" \ + --eval "(org-babel-execute-buffer)" +fi + +echo "Done." diff --git a/projects/repl-block/repl-block b/projects/repl-block/repl-block new file mode 100755 index 0000000..64d646b --- /dev/null +++ b/projects/repl-block/repl-block @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +"""Extract a #+begin_src lisp block from an .org file and print it. + +Identify the block by: + - index: --block 3 (1-based, counting all #+begin_src lisp blocks) + - function: --function view-status (finds a defun/demacro matching the name) + +Output is the block content between begin and end markers, sans markers. + +Usage: + repl-block org/file.org --function foo | repl + repl-block org/file.org --block 3 + repl-block org/file.org --function foo --package :my-package (adds in-package prefix) +""" + +import sys +import re +import argparse + + +LISP_BEGIN = re.compile(r"#\+begin_src\s+lisp\b", re.IGNORECASE) +END_SRC = re.compile(r"#\+end_src\b", re.IGNORECASE) + + +def extract_blocks(lines): + blocks = [] + start = None + buf = None + for i, line in enumerate(lines, start=1): + if start is None: + if LISP_BEGIN.match(line.lstrip()): + start = i + buf = [] + else: + if END_SRC.match(line.lstrip()): + blocks.append((start, buf)) + start = None + buf = None + else: + buf.append(line.rstrip("\n")) + return blocks + + +def find_by_function(blocks, name, lines): + for line_no, body in blocks: + for bline in body: + if re.match(rf"\(def(un|macro|method|var|parameter|class|struct|package)\s+{re.escape(name)}\b", bline): + return line_no, body + return None, None + + +def find_by_index(blocks, idx): + if 1 <= idx <= len(blocks): + return blocks[idx - 1] + return None, None + + +def main(): + parser = argparse.ArgumentParser(description="Extract lisp blocks from org files") + parser.add_argument("file", help=".org file to extract from") + parser.add_argument("--block", type=int, default=None, help="Block number (1-based)") + parser.add_argument("--function", type=str, default=None, help="Function name to find") + parser.add_argument("--package", type=str, default=None, help="(in-package ...) prefix") + args = parser.parse_args() + + try: + with open(args.file, encoding="utf-8") as f: + lines = f.readlines() + except FileNotFoundError: + print(f"File not found: {args.file}", file=sys.stderr) + return 1 + except Exception as e: + print(f"Error reading {args.file}: {e}", file=sys.stderr) + return 1 + + blocks = extract_blocks(lines) + + if args.function: + line_no, body = find_by_function(blocks, args.function, lines) + if body is None: + print(f"No block found containing function '{args.function}'", file=sys.stderr) + return 1 + elif args.block: + line_no, body = find_by_index(blocks, args.block) + if body is None: + print(f"Block {args.block} not found (file has {len(blocks)} blocks)", file=sys.stderr) + return 1 + else: + # Print listing + for idx, (line_no, body) in enumerate(blocks, 1): + first = (body or [""])[0][:60] + print(f" {idx}: line {line_no}: {first}") + print(f"\n{len(blocks)} total blocks", file=sys.stderr) + return 1 + + if args.package: + pkg = args.package + if not pkg.startswith(":"): + pkg = f":{pkg}" + print(f"(in-package {pkg})") + print() + + for line in body: + print(line) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/projects/repl-tool/repl b/projects/repl-tool/repl new file mode 100755 index 0000000..011e160 --- /dev/null +++ b/projects/repl-tool/repl @@ -0,0 +1,74 @@ +#!/usr/bin/env perl +# repl — evaluate Lisp forms against the running Passepartout daemon +# Usage: repl +# or: echo '' | repl +# +# Connects to the daemon on 127.0.0.1:9105, sends a repl-eval request +# via the framed TCP protocol, reads and prints the response. + +use strict; +use warnings; +use IO::Socket::INET; + +my $HOST = $ENV{PASSEPARTOUT_HOST} || "127.0.0.1"; +my $PORT = $ENV{PASSEPARTOUT_PORT} || "9105"; +my $TIMEOUT = $ENV{PASSEPARTOUT_REPL_TIMEOUT} || 10; + +my $expr = join(" ", @ARGV); +if (!$expr) { + $expr = do { local $/; }; +} +chomp($expr); +# Quote the expression for embedding in a Lisp string +$expr =~ s/\\/\\\\/g; # backslash → doubled +$expr =~ s/"/\\"/g; # " → \" +if (!$expr) { + die "Usage: repl \n or: echo '(+ 1 2)' | repl\n"; +} + +my $sock = IO::Socket::INET->new( + PeerHost => $HOST, + PeerPort => $PORT, + Proto => "tcp", + Timeout => 2 +) or die "Cannot connect to $HOST:$PORT: $!\n"; + +sub read_frame { + my ($sock) = @_; + my $hex; + $sock->read($hex, 6) or return undef; + my $len = hex($hex); + my $content; + $sock->read($content, $len); + return $content; +} + +sub write_frame { + my ($sock, $content) = @_; + my $len = sprintf("%06X", length($content)); + $sock->send($len . $content); +} + +# Read handshake (discard) +my $handshake = read_frame($sock); + +# Build framed message +my $msg = '(:TYPE :EVENT :PAYLOAD (:SENSOR :repl-eval :CODE "' . $expr . '"))'; +write_frame($sock, $msg); + +# Read response +my $response = read_frame($sock); +if ($response) { + if ($response =~ /:VALUE "([^"]*)"/s) { + print "$1\n"; + } elsif ($response =~ /:message "([^"]*)"/s) { + print STDERR "$1\n"; + exit 1; + } else { + print "Response: $response\n"; + } +} else { + print STDERR "No response from daemon (is it running on $HOST:$PORT?)\n"; + exit 1; +} +$sock->close(); diff --git a/projects/tangle-tool/tangle b/projects/tangle-tool/tangle new file mode 100755 index 0000000..4e8cfa1 --- /dev/null +++ b/projects/tangle-tool/tangle @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# Tangle an org file using Emacs batch mode +# Usage: tangle + +set -e + +if [ $# -ne 1 ]; then + echo "Usage: tangle " + echo " Tangles all src blocks with :tangle directives to their targets" + exit 1 +fi + +ORG_FILE="$1" + +if [ ! -f "$ORG_FILE" ]; then + echo "Error: File not found: $ORG_FILE" + exit 1 +fi + +echo "Tangling: $ORG_FILE" + +emacs --batch \ + --load org \ + --eval "(setq org-confirm-babel-evaluate nil)" \ + --eval "(org-babel-tangle-file \"$ORG_FILE\")" + +echo "Done." diff --git a/projects/verify-repl-tool/verify-repl b/projects/verify-repl-tool/verify-repl new file mode 100755 index 0000000..ada3b56 --- /dev/null +++ b/projects/verify-repl-tool/verify-repl @@ -0,0 +1,140 @@ +#!/bin/bash +# verify-repl — compliance checker for the OpenCode Engineering Discipline +# +# Usage: verify-repl +# Scans all .org files in the given directory for violations of: +# 1. REPL-First: every (defun|defmacro|defvar|defparameter|defstruct|defmethod|defclass) +# block must have ";; REPL-VERIFIED:" on the line above #+begin_src lisp +# 2. One-per-block: each #+begin_src lisp block must contain exactly one top-level form +# 3. Prose-before-code: each code block must be preceded by an Org headline +# +# Returns 0 if all checks pass, 1 if violations found. + +set -euo pipefail + +ORG_DIR="${1:-}" +if [ -z "$ORG_DIR" ] || [ ! -d "$ORG_DIR" ]; then + echo "Usage: verify-repl " + exit 1 +fi + +VIOLATIONS=0 +FILES_CHECKED=0 + +# Blacklist: files exempt from REPL verification (core infrastructure, tests) +BLACKLIST=( + "core-defpackage.org" + "core-manifest.org" + "core-skills.org" + "core-communication.org" + "package.lisp" + "setup.org" +) + +is_blacklisted() { + local fname + fname=$(basename "$1") + for bl in "${BLACKLIST[@]}"; do + [ "$fname" = "$bl" ] && return 0 + done + return 1 +} + +check_file() { + local file="$1" + local fname + fname=$(basename "$file") + local in_block=0 + local block_start=0 + local prev_line="" + local prev_was_headline=0 + local block_content="" + local line_no=0 + local has_repl_verify=0 + local def_count=0 + local violations_in_file=0 + + is_blacklisted "$file" && return 0 + + FILES_CHECKED=$((FILES_CHECKED + 1)) + + while IFS= read -r line || [ -n "$line" ]; do + line_no=$((line_no + 1)) + local trimmed="${line#"${line%%[![:space:]]*}"}" + + # Track headlines + if echo "$trimmed" | grep -qE '^\*+[[:space:]]'; then + prev_was_headline=1 + fi + + # Enter code block + if echo "$trimmed" | grep -qE '^#\+begin_src[[:space:]]+lisp'; then + in_block=1 + block_start=$line_no + block_content="" + def_count=0 + has_repl_verify=0 + # Check for REPL-VERIFIED comment on previous line(s) + if echo "$prev_line" | grep -qE ';;.REPL.VERIFIED:'; then + has_repl_verify=1 + fi + # Check for prose requirement: was there a headline before this block? + if [ "$prev_was_headline" -eq 0 ]; then + echo " $fname:$block_start: PROSE-BEFORE-CODE: no Org headline before code block" + violations_in_file=$((violations_in_file + 1)) + fi + fi + + # Inside code block: collect content + if [ "$in_block" -eq 1 ]; then + if echo "$trimmed" | grep -qE '^#\+end_src'; then + in_block=0 + # Check: REPL verification for definition blocks + if [ "$def_count" -gt 0 ] && [ "$has_repl_verify" -eq 0 ]; then + echo " $fname:$block_start: REPL-FIRST: $(if [ "$def_count" -gt 1 ]; then echo "$def_count definitions"; else echo "defun/defmacro"; fi) without REPL-VERIFIED comment" + violations_in_file=$((violations_in_file + 1)) + fi + # Check: one-per-block + if [ "$def_count" -gt 1 ]; then + echo " $fname:$block_start: ONE-PER-BLOCK: $def_count definitions in a single block (must be exactly 1)" + violations_in_file=$((violations_in_file + 1)) + fi + prev_was_headline=0 + block_content="" + else + # Count definitions in block + if echo "$trimmed" | grep -qE '^\((defun|defmacro|defvar|defparameter|defstruct|defmethod|defclass)[[:space:](]'; then + def_count=$((def_count + 1)) + fi + fi + fi + + prev_line="$line" + done < "$file" + + if [ "$violations_in_file" -gt 0 ]; then + echo " ✗ $fname: $violations_in_file violation(s)" + VIOLATIONS=$((VIOLATIONS + violations_in_file)) + fi +} + +echo "=== REPL Compliance Check ===" +echo "Directory: $ORG_DIR" +echo "" + +for file in "$ORG_DIR"/*.org; do + [ -f "$file" ] || continue + check_file "$file" +done + +echo "" +echo "Files checked: $FILES_CHECKED" +echo "Violations: $VIOLATIONS" + +if [ "$VIOLATIONS" -eq 0 ]; then + echo "Status: ✓ PASS — all checks satisfied" + exit 0 +else + echo "Status: ✗ FAIL — fix violations before committing" + exit 1 +fi