139 lines
13 KiB
Org Mode
139 lines
13 KiB
Org Mode
:PROPERTIES:
|
||
:CREATED: [2026-05-27 Wed]
|
||
:ID: dddd52a7-adb8-470e-a459-614ade5f76af
|
||
:END:
|
||
#+title: Closing the Lisp Gap
|
||
#+filetags: :passepartout:lisp:compiler:performance:ecosystem:asics:bootstrapping:
|
||
|
||
Lisp has a performance gap with C and Rust of roughly 10-20% on hot numerical code and a much larger ecosystem gap (package manager, LSP, cross-compilation, library count ~2k vs ~180k). This page diagnoses both gaps and argues that [[id:28c46769-c14b-42aa-ac7a-69d310157f8f][Passepartout]]'s architecture — fact store, gates, ACL2 prover, self-improving loop — changes the nature of the problem from community-coordination to knowledge-capture, making the gap closable in 1-2 years rather than 20-30 person-years.
|
||
|
||
See [[id:9af13fff-9725-542b-93b1-a555bc74ad72][why Lisp is economically viable now]] for the economic foundation. See [[id:13e6ae54-2d24-5aa0-b1cd-a7e8e749aa70][self-driving Lisp Machine]] for the hardware endpoint.
|
||
|
||
* The Performance Gap
|
||
|
||
What Lisp (SBCL) still needs to close the remaining gap with C/Rust:
|
||
|
||
- **Alias analysis on SBCL's IR.** Rust's borrow checker gives LLVM perfect aliasing at function scope. SBCL conservatively assumes any memory reference could alias any other. Fix: a static analysis pass proving non-aliasing for common patterns (different simple-array types, arrays vs scalars). ~1-2 person-years, biggest single investment.
|
||
|
||
- **LLVM backend (Clasp).** Clasp gives LLVM's full pipeline — auto-vectorization, LICM, inlining cost modeling — at the cost of fighting LLVM's static worldview with CLOS's dynamic semantics. Clasp's hybrid approach (fast-path LLVM, slow-path runtime) is the right design. ~5-10 person-years to reach SBCL's general performance with better hot-path optimization.
|
||
|
||
- **PGO (profile-guided optimization).** SBCL has profiling (sb-sprof) but no feedback loop — the compiler cannot reweight branches or layout hot/cold blocks. Fix: instrumentation pass that emits counter VOPs, a profile binary, and a recompilation step that annotates IR with branch weights. ~1 person-year.
|
||
|
||
- **Post-link optimization.** Facebook's BOLT profiles binary execution and rewrites code layout to reduce I-cache misses. SBCL's unusual code layout (trampolines, closure entry points) chokes BOLT. Hard research problem.
|
||
|
||
- **Vectorization VOPs.** No standard SIMD library for Lisp. Every ISA extension needs its own VOP set. C gets this via xmmintrin.h and auto-vectorization. ~2-3 person-years for a portable SIMD abstraction backed by platform VOPs.
|
||
|
||
* The Ecosystem Gap
|
||
|
||
| Feature | Rust | Common Lisp |
|
||
|---------|------|-------------|
|
||
| Package manager | Cargo (lockfile, binary cache, workspace) | Quicklisp (no lockfile, source-only) |
|
||
| LSP | rust-analyzer (excellent) | No universal LSP |
|
||
| Formatter | rustfmt (universal) | cl-form (not universal) |
|
||
| Linter | Clippy (>700 rules) | No equivalent |
|
||
| Doc system | rustdoc (integrated) | Conventions only |
|
||
| Cross-compilation | --target flag | Manual, source-based |
|
||
| Test framework | Built-in with benchmarks | Various (FiveAM, Prove) |
|
||
| Sanitizers | ASan, TSan, MSan, UBSan | None |
|
||
| Library count | ~180k | ~2k |
|
||
| WASM target | First-class | Modest |
|
||
| Mobile targets | First-class | None native |
|
||
|
||
The largest gaps are the package manager (the single highest-leverage investment — ~2-3 person-years + community coordination) and the library count (~100× fewer).
|
||
|
||
* Lisp ASICs: What They Change
|
||
|
||
A Lisp ASIC (Symbolics Ivory, CADR, or a modern RISC-V variant) **raises the floor** more than the ceiling:
|
||
|
||
**What it gives in hardware:**
|
||
- Tag checking on every memory access — types, GC bits, forwarding pointers in parallel with ALU
|
||
- Hardware CONS — single-cycle allocation from a dedicated region with automatic write barriers
|
||
- Hardware GC support — generational barriers at cache level
|
||
- Hardware generic function dispatch — type-code lookup in CAM instead of method cache
|
||
- Hardware stack groups and shallow binding — zero-cost special variable rebinding
|
||
|
||
**Does it eliminate the need for compiler optimization?** Both yes and no. The worst case becomes much faster — naive code and optimized code are closer together. But the compiler still matters for instruction scheduling, inlining decisions, dead code elimination, and loop optimizations. The ASIC shifts the optimizer's job from "avoid Lisp overhead" to "use special instructions effectively." The gap between C and Lisp on hot numerical loops narrows from ~20% to maybe 5%.
|
||
|
||
* Embedded and Small Systems: RISC-V Lisp Extension
|
||
|
||
A full Lisp Machine chip doesn't make sense for embedded, but a **RISC-V extension with 5-10 Lisp primitives** does:
|
||
|
||
- ~5 new instructions added to a RV32/RV64 core (CONS with write barrier, tag dispatch, read barrier, tag extract, GC check)
|
||
- Implementable as an FPGA soft-core or small coprocessor alongside a microcontroller
|
||
- ~50K additional gates on a RISC-V e300-class core
|
||
|
||
This makes Lisp viable where C is currently mandatory: real-time control, sensor nodes, IoT. The hardware CONS eliminates allocation cost; the hardware read barrier eliminates GC pause unpredictability by making the collector a concurrent background task instead of a stop-the-world event. The gap goes from 10-100× (current, boxed allocation + GC) to ~2-5×.
|
||
|
||
**Strategic play:** Define the extension, open-source an FPGA implementation, get it ratified as a standard RISC-V extension. Then any chip vendor can add it.
|
||
|
||
* Why Passepartout Changes the Equation
|
||
|
||
The standard model for closing the library gap: *more users → more libraries → more users (virtuous cycle).*
|
||
|
||
Passepartout's model: *agent synthesizes libraries on demand → fewer blockers → more users.*
|
||
|
||
**Knowledge-capture replaces community-coordination.** The rate-limiter shifts from "people who can write Lisp libraries" to "people who can explain their domain well." A 3-hour conversation with a domain expert captures the specifications, invariants, known techniques, and correctness criteria. The agent synthesizes the library, verifies it against stored properties via ACL2, and hot-reloads it.
|
||
|
||
**What changes:**
|
||
- Standard protocols (HTTP, SQL, JSON, crypto, file formats) — the agent synthesizes these from specs it has seen in training.
|
||
- Wrapper libraries (CFFI bindings, REST clients) — mechanical, the agent handles well.
|
||
- Domain-specific libraries — the hardest category *without* a knowledge architecture, and the most tractable *with* one. The gate becomes the bridge: human provides $10K/year-level insight, agent provides $200K/year-level engineering.
|
||
- What stays hard: libraries where the algorithm requires original research (not yet done by anyone), libraries requiring specialized hardware knowledge the human cannot articulate, and battle-tested implementations where verification of constant-time correctness is near-impossible.
|
||
|
||
**Passepartout's position to direct Lisp development:**
|
||
|
||
1. **Instrumentation advantage.** It runs in Lisp, can profile its own execution in Lisp terms, identify SBCL's compiler bottlenecks, and generate patches — all in a closed feedback loop that C compilers lack.
|
||
|
||
2. **Coordination advantage by adoption.** If Passepartout becomes the standard Lisp agent framework, its tooling choices become de facto standards: ocicl with lockfiles becomes the package manager, cl-lsp gets maintenance, sb-sprof fed into the optimizer becomes the PGO pipeline.
|
||
|
||
3. **Automated contribution pipeline.** Profile → identify hot path in SBCL → generate candidate VOP or optimization pass → run test suite (verified by ACL2) → submit patch. Cycle time from "this would be faster if SBCL did X" to "SBCL does X" drops from years to days.
|
||
|
||
4. **The prover multiplier.** ACL2 makes safe optimization an engineering discipline instead of an experimental art. The agent can generate thousands of variants, verify correctness, benchmark, and keep the Pareto-optimal set — coverage a human cannot match.
|
||
|
||
* The Same Compression Applies to Social Infrastructure
|
||
|
||
The Passepartout architecture compresses the *social* ecosystem of the internet the same way it compresses the *software* ecosystem — replacing multiplicative duplication with additive specification.
|
||
|
||
**The problem:** Twitter (~10M lines), Facebook (~60M lines), Reddit (~5M lines), Discord (~8M lines), Signal (~3M lines), Telegram (~5M lines), LinkedIn (~15M lines), WhatsApp (~3M lines). Each independently implements the same primitives: identity, identity verification, key recovery, encrypted messaging, group messaging, moderation, content addressing, access control, payment integration, rate limiting, spam detection, reporting, appeal, ban propagation, federation, data export, deletion. Each is a separate 3-60 million line codebase with the *same semantics* but different APIs, different bugs, different compliance postures, different threat models. None compose with any other.
|
||
|
||
**Under Passepartout:**
|
||
|
||
- One spec for identity: keys, recovery, delegation, attestation. ~500 lines.
|
||
- One spec for messaging: content-addressed, encrypted, ordered by a Merkle DAG. ~1,000 lines.
|
||
- One spec for groups: membership, permissions, moderation. ~1,500 lines.
|
||
- One spec for federation: cross-server state replication, conflict resolution. ~1,000 lines.
|
||
- Gates for policy: what content types are allowed, what moderation actions exist, how appeals work. Configurable per community, enforced by the prover.
|
||
|
||
Total spec surface: ~5,000 lines. Total synthesized code (client + server for any platform): ~50,000 lines of Lisp.
|
||
|
||
**5,000 lines of spec + 50,000 lines of synthesized implementation vs ~100 million lines of independent, incompatible, duplicative implementations** that each require their own security audits, compliance reviews, and maintenance pipelines.
|
||
|
||
**Composition replaces isolation.** In the current model, Signal's identity system cannot talk to WhatsApp's. Reddit's moderation cannot federate with Discord's. The protocols don't compose because they were designed independently with no shared semantic foundation. In the Passepartout model, every synthesized client and server speaks the same spec. A community can use a Telegram-like frontend and a Reddit-like frontend — both synthesized from the same messaging and content-addressing specs — and they share identity, encryption, and federation because those are the same underlying specifications. The frontend is a gate: a policy on how to render the same underlying protocol data.
|
||
|
||
**The multiplication disappears.** Under the old model, N platforms cost N × (full stack). Under Passepartout, N communities cost one stack + N × (~100 lines of gate policy). The marginal cost of adding a new community approaches zero. This is the same argument as the library gap: N libraries × (different maintainers, different bugs, different API surfaces) collapses to N specifications × (one verified synthesized implementation).
|
||
|
||
**The deeper claim:** the protocol isn't a separate feature of Passepartout. It is the same compression architecture applied to a different domain — coordination infrastructure instead of library infrastructure. Both are about replacing multiplicative maintenance burden with additive specification complexity, amplified by Lisp's density and the prover's correctness guarantees.
|
||
|
||
* Timeline Compression
|
||
|
||
With the self-improving loop, the marginal cost of each incremental improvement drops to near zero. The system improves continuously rather than in discrete releases. Every time compute is added to the pipeline, the system gets faster.
|
||
|
||
| Gap | Human-only | AI-assisted | Passepartout loop |
|
||
|-----|-----------|-------------|------------------|
|
||
| SBCL alias analysis | 18 mo | 10 mo | 3-4 mo |
|
||
| PGO pipeline | 12 mo | 6 mo | 2-3 mo |
|
||
| Cargo-equivalent | 24 mo | 8 mo | 4-6 mo |
|
||
| LSP server | 18 mo | 6 mo | 3-4 mo |
|
||
| RISC-V Lisp extension | 24 mo | 12 mo | 6-8 mo |
|
||
| Library coverage (100 APIs) | 3-5 yrs | 12 mo | 3-4 mo (synthesis) |
|
||
| Clasp maturity | 5-10 yrs | 3-5 yrs | 1-2 yrs |
|
||
|
||
The meta-level effect: once the pipeline exists (profile → generate → verify → test → deploy), the system improves continuously. The gap-closing problem goes from "we need enough talented volunteers" to "we need enough compute" — and compute grows on a known exponential, while talent doesn't.
|
||
|
||
* Relationship to Other Concepts
|
||
|
||
- [[id:9af13fff-9725-542b-93b1-a555bc74ad72][Lisp economics]] — the economic viability argument provides the why; this page provides the how.
|
||
- [[id:13e6ae54-2d24-5aa0-b1cd-a7e8e749aa70][Self-driving Lisp Machine]] — the hardware endpoint enabled by this software-level gap-closing.
|
||
- [[id:efc76898-03f7-57ba-923d-35d65da88bb7][Sufficiency flip]] — the threshold where symbolic reasoning becomes cheaper than LLM calls; applies to compiler optimization as a domain.
|
||
- [[id:84a537b4-4256-50c8-91f5-dd5b4538418f][Verification appliance]] — the $5K/year hardware that replaces $500K/year in compliance costs; RISC-V Lisp extension is the technical path to making this run everywhere.
|