amr/hermes-brain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Restructure three-pronged → knowledge-layers: collapse 11 files to 3, integrate into main architecture

Browse Source 
- Rename 'three-pronged' folder to 'knowledge-layers' — prong metaphor
  was misleading (implied parallel tines), replaced with epistemic layers
  (deductive base, empirical middle, probabilistic oracle — vertical stack)
- Collapse 11 overlapping files into 3 coherent documents:
  - knowledge-layers/_index.org: core framework (two engines + one store,
    World Model formula, 0-14 layer table, provenance store design,
    conflict resolution, cold-start, stage mapping)
  - knowledge-layers/practical-implications.org: design-world-aware-of-
    physics, 10 powers, Schafmeister existence proof, epistemic transparency
  - knowledge-layers/neurological-empirical.org: neural networks in
    provenance framework (kept intact)
- Relocate wolfram/mathematica and Schafmeister docs to ideas/viability/
- Integrate into main architecture _index.org:
  - Gate: expanded from two vectors (ACL2+LLM) to three (deductive,
    provenance/empirical, LLM oracle)
  - Autodidactic loop: split into Track 1 (deductive hardening, fast)
    and Track 2 (empirical validation, slow, experimental-feedback-driven)
  - See also: added Knowledge Layers cross-reference
- Add all-lisp geometry engine note (ideas/lisp-geometry-engine.org) as
  concrete illustration of the empirical layer's effect on design work
- Rebuild site: 148 files, 0 errors
This commit is contained in:
Hermes
2026-06-04 19:09:44 +00:00
parent 2e8cf19f9e
commit 6e992cc0c5
92 changed files with 921 additions and 2628 deletions
Download Patch File Download Diff File Expand all files Collapse all files

74
ideas/viability/open-source-wolfram-lisp.org Normal file
View File

@@ -0,0 +1,74 @@
:PROPERTIES:
:CREATED: [2026-05-24 Sun]
:ID: 7a8b9c0d-1e2f-3a4b-5c6d-7e8f9a0b1c2d
:END:
#+title: Viability of an Open-Source Wolfram Language / Mathematica in Common Lisp
#+filetags: :ideas:lisp:mathematics:open-source:
An assessment of what it would take to build a viable open-source equivalent of Wolfram Language and Mathematica in Common Lisp, based on the existing ecosystem and the fundamental architectural alignment between Lisp and symbolic computation.
**The alignment is natural, not forced.**
Wolfram Language is, at its core, a term-rewriting system with pattern matching, rule-based transformation, and a uniform symbolic representation for everything (expressions are trees of the form head[arg1, arg2, ...] — the Wolfram equivalent of cons cells). This is very close to what Lisp is natively. A Lisp implementation of the core evaluator — pattern matching, rule application, substitution, term rewriting — is not a foreign port. It is an exercise in expressing Wolfram's semantics in a language whose semantics were designed for the same problem domain.
Maxima proves this historically. It is a direct descendant of Macsyma, the MIT computer algebra system that inspired Mathematica. Macsyma was written in Lisp. Mathematica's core evaluation model inherits heavily from Macsyma. An open-source Common Lisp computer algebra system already exists, has existed for decades, and works. The question is not whether it can be done, but how much of the modern Mathematica ecosystem can be replicated and at what cost.
**What already exists.**
| Layer | Existing CL work | Status |
|---|---|---|
| Symbolic engine / term rewriting | Lisp readers, pattern matching libs (trivia, optima, fare-matcher), rule systems | Foundational primitives exist, no unified Wolfram-equivalent evaluator |
| Computer algebra system | Maxima, FriCAS (Axiom), reduce-algebra | Mature CASes — Maxima alone has differentiation, integration, ODEs, linear algebra, tensors, series, limits, Laplace transforms |
| Numerical computing | magicl, lla (Lisp Linear Algebra), CL-NUM, GSLL (GNU Scientific Library bindings) | Solid — covers BLAS, LAPACK, random numbers, special functions, optimization |
| Visualization | cl-cairo2, Vecto, CLG, CommonQt, cl-zxing | Exists but scattered — no unified plotting framework like Mathematica's |
| Notebook interface | cl-jupyter, common-lisp-jupyter, Lem | Jupyter kernel works. Lem is a native editor approaching notebook capability. No Mathematica-level notebook yet. |
| Rule-based programming | fare-matcher, optima, prolog implementations | Pattern matching is good. Full term-rewriting system needs assembly. |
| Knowledge graph | gbrain, various triplestore libs | Possible but would need Wolfram Alpha-level investment |
| Deployment | ASDF, Quicklisp, SBCL standalone executables | Better than Mathematica's deployment story — Lisp produces real executables |
**The hard parts.**
1. **The standard library is the product, not the engine.** Mathematica ships thousands of built-in functions — every mathematical special function, every statistical distribution, every graph algorithm, every image processing filter, every string operation, every data format parser. This is not a technical challenge; it is a sheer volume problem. The open-source answer is to wrap existing C/C++ libraries (GSL for special functions, OpenCV for image processing, igraph for graph algorithms, etc.) and expose them through a unified symbolic interface. This is the Clasp approach: interop with mature C++ libraries rather than rebuilding everything from scratch in Lisp. The Wolfram equivalent would be a CLOS-based symbolic dispatch layer that wraps these libraries and makes them accessible through a consistent term-rewriting evaluator.
2. **The notebook interface is a product in itself.** Mathematica's notebook is not a terminal with nice formatting. It is a computational notebook with inline typeset math, dynamic graphics, collapsible sections, live evaluation, and a rich document model. The Jupyter ecosystem solves half of this. A Lisp-native notebook would need a rendering engine for mathematical notation (LaTeX or MathJax integration), inline interactive graphics, and a document model compatible with literate computation. Lem is the most promising starting point — it already has Emacs-like extensibility, a GTK frontend, and a Lisp-native codebase. Extending Lem to support computational notebooks with inline graphics and typeset output is the shortest path.
3. **Performance for specialized domains.** Mathematica's kernel is highly optimized for symbolic operations — pattern matching over large expressions, automatic algorithm selection, memoization, and incremental compilation. A naive Lisp implementation would match Mathematica for small-to-medium expressions but would need significant optimization work for the heavy cases (symbolic integration of large expressions, graph operations on million-node graphs, image processing pipelines). The advantage is that Lisp's compiler infrastructure (SBCL's type inference, VOPs, inlining) gives a much better baseline than most languages. SBCL can generate code that approaches C speed for numerical kernels.
4. **The knowledge graph (Wolfram Alpha).** Mathematica's integration with Wolfram Alpha — querying computable data about chemistry, geography, finance, linguistics, etc. — is a separate product with a massive engineering investment in data curation. An open-source equivalent would not replicate this. It would either provide a local, user-curatable knowledge base (gbrain fits here) or integrate with existing open knowledge graphs (Wikidata, DBpedia). The gbrain connection is interesting: if Passepartout's knowledge store can answer factual queries with provenance, that becomes the Wolfram Alpha equivalent for the Lisp Mathematica.
5. **Package ecosystem and community.** Mathematica's advantage is not just its engine but its ecosystem — thousands of paclets, the Wolfram Function Repository, the community that shares notebooks. An open-source equivalent needs a package manager (Quicklisp solves this for Lisp libraries), a repository for symbolic packages (a Wolfram Function Repository equivalent), and a critical mass of users who both use and contribute. Maxima has users but not contributors. The gap is community formation, not technical capability.
**The viability assessment.**
| Domain | Viability | Timeline estimate | Risk |
|---|---|---|---|
| Core symbolic evaluator | High — Lisp was designed for this | 6-12 months for working prototype | Low — well-understood problem |
| Computer algebra | High — Maxima already exists | Integrate now; polish 1-2 years | Low — needs UI/UX investment |
| Numerical computing | High — wrappers exist | 3-6 months for unified interface | Low — wrapping problem |
| Visualization | Medium — scattered pieces | 1-2 years for unified framework | Medium — needs new work |
| Notebook interface | Medium — Lem as foundation | 1-2 years to Mathematica parity | Medium — significant UX engineering |
| Standard library breadth | Low — volume problem | 3-5 years with community | High — needs sustained contribution |
| Knowledge graph | Low — curation cost | 2-3 years for basic integration | High — different product category |
| Deployment | High — Lisp executables | Works now | None |
**The strategic question.**
The real question is not "can we replicate Mathematica in Lisp" but "should we?" — and if so, for whom.
Mathematica serves two distinct use cases:
- **Interactive exploration** — a researcher types an integral, gets a result, visualizes it, iterates. Lisp + Maxima + a good notebook interface already does this, and the experience is competitive for anyone comfortable with Lisp syntax.
- **Deployed computation** — a company builds a production pipeline around Mathematica kernels, deploying computation as a service. Lisp executables are dramatically better here — they are real compiled binaries, not managed by a proprietary kernel, deployable without license fees, embeddable anywhere.
For the second use case, the open-source Lisp alternative is already superior. The gap is the first use case: the interactive exploration experience, the breadth of built-in functions, and the cultural acceptance of Lisp syntax in communities that currently write Wolfram Language.
The most viable path is not to clone Mathematica but to integrate Maxima + numerical Lisp libraries under a unified symbolic interface, expose all of it through a Lem-based notebook, and make the Jupyter bridge the primary entry point for users who prefer Python notebooks. This gives you 80% of Mathematica's capability with a fraction of the development cost, and it connects to the existing scientific Python ecosystem rather than competing with it.
**The deeper point.**
Mathematica's architecture is Lisp-like because it was inspired by a Lisp system (Macsyma). An open-source Mathematica in Lisp is not a port. It is a return to the original architectural vision, implemented in the language that vision was originally expressed in. The question is whether the community investment materializes — and that depends on whether there is a use case that justifies it. Passepartout's verification infrastructure may be that use case: a verified symbolic computation engine that reasons about its own results is a Mathematica-like system by necessity, and building it in Lisp is the natural path.
---
- [[id:f4e5d6c7-b8a9-0c1d-2e3f-4a5b6c7d8e9f][Schafmeister and Clasp]] — Lisp in computational nanotechnology, existence proof for Lisp viability in scientific computing
- [[id:1c3ec48b-446c-50d2-b53e-126a81f5143f][Passepartout Architecture]] — why Lisp and where the symbolic engine fits

92
ideas/viability/passepartout-bootstrap-mathematica.org Normal file
View File

@@ -0,0 +1,92 @@
:PROPERTIES:
:CREATED: [2026-05-24 Sun]
:ID: 8b9c0d1e-2f3a-4b5c-6d7e-8f9a0b1c2d3e
:END:
#+title: Could Passepartout Bootstrap Mathematica or mathlib by Itself?
#+filetags: :ideas:lisp:passepartout:mathematics:
This extends the previous viability analysis with a specific scenario: assuming Passepartout exists at Stage 3+ (full Lisp machine with neurosymbolic engine, verification gate, and self-modification capability), how hard would it be for it to recreate Mathematica or mathlib in pure Common Lisp on its own?
**The bootstrap is architectural, not aspirational.**
The neurosymbolic engine is not just a faster way to write code. It is a closed loop: the LLM proposes implementations, the symbolic engine and ACL2 prover verify correctness, and the self-modification system hot-reloads the result into the running image. This loop runs autonomously, without human intervention. The system writes code, tests it formally, improves it, and keeps the result — permanently expanding its own capability.
This is fundamentally different from a human writing Mathematica. A human writes code, compiles, tests, debugs. Passepartout writes code, has it verified against a formal specification, and loads it into its own runtime. The iteration speed is not hours or days — it is seconds per function, limited only by the LLM's generation latency and the prover's checking time.
**What Passepartout already has.**
At Stage 3, the system ships with:
- A symbolic term-rewriting engine (the evaluator itself is one)
- Pattern matching and rule-based transformation (native to the gate architecture)
- ACL2 as a verification backend (can prove properties of generated code)
- An LLM oracle for proposing implementations (the probabilistic brain)
- A self-modification system (hot-reloads verified code into the running image)
- A knowledge store with persistent facts (gbrain-derived)
These are not general-purpose tools that happen to be useful for symbolic mathematics. They are the same tools that a computer algebra system needs, expressed in the same architecture. The evaluator that rewrites a gate policy is the same mechanism that rewrites a symbolic expression.
**What it needs to generate.**
| Component | Can Passepartout generate it? | How |
|---|---|---|
| Core symbolic evaluator | Yes, trivially — this is what Lisp *is* | The existing evaluator already does term rewriting. The neurosymbolic engine would create a higher-level pattern-matching layer over it. |
| Computer algebra (differentiation, integration, simplification) | Yes — known algorithms, formally specifiable | LLM proposes implementation of Risch algorithm, polynomial GCD, Gröbner bases. ACL2 verifies the specification. |
| Numerical libraries (BLAS, special functions, optimization) | Partial — better to wrap | ACL2 cannot verify floating-point numerics to the same standard. Passepartout would wrap existing C/C++ libraries via Clasp-style interop and verify the interface, not the numerics. |
| Visualization framework | Yes — UI code, not math | The environment subsystem (Nyxt/Lish) already has rendering primitives. The neurosymbolic engine generates plotting and graphics code against them. |
| The 5,000+ function standard library | Yes — volume, not novelty | This is the dominant cost. Each function is individually trivial (differentiate x^3 → 3x^2) but there are thousands. Passepartout generates them at LLM speed — roughly one function every 10-30 seconds including verification. |
| Formal proofs of mathematical theorems (mathlib) | Qualified yes — different logic | mathlib is in Lean's dependent type theory. Passepartout's ACL2 is first-order logic. The theorems can be re-proven in ACL2, but the proofs are not portable. The LLM proposes proof strategies, ACL2 checks them. |
**The rate limit is generation, not computation.**
If Passepartout generates one verified function every 20 seconds (conservative — LLM proposal time + ACL2 verification), that is 180 functions per hour, ~4,300 per day. Mathematica's standard library contains roughly 6,000 documented functions. At this rate, the standard library would take ~1.5 days of continuous generation — assuming the LLM has the domain knowledge to produce correct implementations and ACL2 can verify them.
This is the critical assumption. The LLM (at, say, GPT-4 or DeepSeek level) already knows what every Mathematica function does. It has seen them in training data. The question is whether it can generate a correct Lisp implementation with a formal specification that ACL2 can verify. For most elementary functions (differentiate, integrate polynomial, singular value decomposition, string split, image histogram), the answer is yes — these are well-understood algorithms with clear specifications.
For specialized domains (elliptic curve cryptography, tensor network contractions, symbolic regression of differential equations), the LLM may generate approximately correct implementations that need refinement. The neurosymbolic loop handles this: ACL2 catches the mismatch, feeds the error back, and the LLM regenerates.
**mathlib is a different problem.**
mathlib is not a library of algorithms but a library of formal proofs — mathematical theorems expressed in Lean's dependent type theory, structured as a hierarchy of definitions, lemmas, and tactics. It represents hundreds of person-years of community effort, formalizing undergraduate mathematics and beyond.
Passepartout's verification layer is ACL2, which operates in a different logical framework (first-order logic with induction for total functions, not dependent types). There is no porting mathlib — it would have to be re-proven in ACL2's logic.
The advantage is that the theorems are already known. mathlib tells you exactly what to prove. The LLM reads the Lean statement, translates it to an ACL2 theorem, proposes a proof strategy, and ACL2 attempts the proof. This is a well-structured task for the neurosymbolic loop: the LLM generates proof plans, ACL2 verifies them, and failed attempts feed back to refine the next plan.
The bootstrapping advantage: early proofs (basic arithmetic, set theory) strengthen the ACL2 reasoning library, which makes later proofs (real analysis, topology) faster. The system accelerates as it goes. mathlib's proof dependency graph is the natural generation order.
Estimated timeline for mathlib-equivalent in ACL2, with Passepartout generating autonomously:
| Milestone | Time estimate | Note |
|---|---|---|
| Basic arithmetic, algebra, number theory | Days — standard library material | Well-known proofs, simple structure |
| Real analysis, measure theory | Weeks — proof complexity increases | Non-trivial but well-studied |
| Abstract algebra (groups, rings, fields) | Weeks — structural, builds on itself | The neurosymbolic loop excels here |
| Topology, algebraic topology | Months — conceptual depth | Proofs are longer, more strategic |
| Category theory, homological algebra | Months — abstraction barrier | High-level abstraction, fewer verification primitives |
| Number theory deep results (FLT, modular forms) | Unknown — research frontier | Passepartout is not proving open problems. It formalizes known results. |
**The bootstrapping compound effect.**
The most interesting property is not that Passepartout can generate Mathematica's library. It is that each generated function becomes part of Passepartout's own capability. After generating the differentiation function, Passepartout uses it to generate the integration function. After generating linear algebra, it uses that to generate optimization algorithms. After generating formal proofs of real analysis, it uses those theorems to verify more complex deductions.
This is not a production pipeline. It is an autodidactic loop: the system generates math, then uses that math to generate more math. The acceleration is exponential in the early phases and linear in the later phases, limited by the rate at which the LLM can produce new correct specifications.
**The real barrier is not technical but oracular.**
At every step, Passepartout depends on the LLM's knowledge of existing mathematics. The LLM has seen most of human mathematical knowledge in its training data. It can propose correct implementations and proof strategies because it has seen them. But for genuinely new mathematics — theorems not present in the training data, algorithms that have not been discovered — the LLM has no signal. Passepartout would be limited by its oracle.
Stage 7 acknowledges this: oracular limits are fundamental. The verification subsystem can check correctness against a specification, but it cannot generate the specification itself. The LLM provides the what; ACL2 verifies the that. Neither provides the why that extends beyond existing knowledge.
**Conclusion.**
Recreating Mathematica's standard library: **days to weeks** of autonomous generation. The volume problem is solvable because the LLM already knows the answer space and ACL2 can verify each function. No human intervention required.
Recreating mathlib's formal proof corpus: **months** of continuous formalization. The neurosymbolic loop maps naturally onto the task of converting known theorems from one logical framework to another. The dependency graph of mathlib provides the optimal generation order.
Neither requires new research. Both are engineering throughput problems that Passepartout's architecture is designed to solve: generate, verify, reload, repeat. The only hard limit is the oracle — the system cannot generate mathematics that the LLM does not already know exists.
---
- [[id:7a8b9c0d-1e2f-3a4b-5c6d-7e8f9a0b1c2d][Viability of open-source Wolfram/Mathematica in Lisp]] — the human-driven assessment
- [[id:1c3ec48b-446c-50d2-b53e-126a81f5143f][Passepartout Architecture]] — the verification and self-modification systems

52
ideas/viability/schafmeister-clasp.org Normal file
View File

@@ -0,0 +1,52 @@
:PROPERTIES:
:CREATED: [2026-05-24 Sun]
:ID: f4e5d6c7-b8a9-0c1d-2e3f-4a5b6c7d8e9f
:END:
#+title: Christian Schafmeister
#+filetags: :ideas:lisp:nanotechnology:
Christian Schafmeister is a chemistry professor at Temple University in Philadelphia. He created [[https://github.com/clasp-developers/clasp][Clasp]], a Common Lisp implementation that interoperates with C++ libraries using LLVM compilation, specifically to solve a problem most Lisp implementers never face: designing molecules at the nanoscale.
* The problem.
Schafmeister's research focuses on spiroligomers — large, shape-programmable molecules built from synthetic monomers. These are programmable at the level of both shape and functional groups, meaning they can be designed to bind specific proteins as therapeutics or accelerate chemical reactions the way enzymes do. The goal is to create molecules that can do everything proteins do in nature, but that are designable and evolvable by human beings.
This is a computational problem of enormous complexity. Designing these molecules requires simulating their behavior, computing binding affinities, searching conformational space, and iterating designs rapidly based on experimental feedback. The compute pipelines involved typically live in the C++ ecosystem (a vast array of scientific computing libraries), but the workflow itself — rapid prototyping, interactive exploration, incremental development — demands the kind of environment that C++ alone cannot provide.
* Why Lisp won.
Schafmeister's argument for Common Lisp in computational nanotechnology mirrors the same reasoning that drives the knowledge-layers architecture:
- **Interactivity.** Molecular design requires exploration. A researcher needs to load data, inspect it, try a transformation, undo it, try another — all within a live environment. Lisp's REPL-driven development provides this in a way that compile-link-run cycles cannot match.
- **Incremental development.** The design space for spiroligomers is too large to simulate exhaustively. You need to build up models piece by piece, testing each step. Lisp's incremental compilation and hot-reloading make this natural.
- **Unified representation.** In Lisp, the code that describes a molecule and the code that simulates it share the same structure. There is no translation barrier between the design language and the simulation language.
But the scientific computing ecosystem lives in C++. Schafmeister could not afford to rebuild every computational chemistry library from scratch. So he built Clasp: a Common Lisp implementation that compiles to native code via LLVM and interoperates seamlessly with C++. Clasp can call any C++ library natively, and C++ can call back into Lisp. The result is that the entire scientific computing ecosystem becomes available from within a Lisp environment — programmable, interactive, introspectable.
* The architecture.
Clasp is not a wrapper or a bridge. It is a full Common Lisp implementation where the C++ interoperation is part of the language runtime itself. The clbind library provides declarative bindings — you describe how C++ classes and functions map to Lisp, and Clasp generates the glue code automatically. This is fundamentally different from CFFI-style foreign function interfaces, which require manual marshaling and are inherently fragile.
From the Lisp perspective, a C++ class appears as a CLOS class. You can subclass it, specialize methods on it, inspect its instances. The boundary between Lisp and C++ is transparent to the programmer.
* Funding and validation.
Clasp has been funded by the Defense Threat Reduction Agency (DTRA), the National Institutes of Health (NIGMS), and the National Science Foundation. These are agencies that fund computational chemistry and materials design, not programming language research. They funded Clasp because it solved a real problem in molecular design that no other approach addressed: making C++-scale scientific computing work within an interactive Lisp environment.
* Relevance to the knowledge-layers architecture.
Schafmeister's work is existence proof for two core claims:
1. Lisp is not a niche language for academic AI research or Emacs configuration. It is being used today to design therapeutic molecules that bind proteins, in environments funded by the NIH and NSF. The interactivity and homoiconicity that the knowledge-layers architecture relies on are the same properties that make this work possible.
2. The single-address-space model is not a limitation but an enabling constraint. Clasp proves that you can run C++ libraries inside a Lisp image, not alongside it. The Lisp machine is not a retro fantasy — it is a practical architecture being used today for computationally demanding scientific work.
The main difference in direction: Schafmeister brought C++ into Lisp to access the scientific computing ecosystem. The knowledge-layers architecture replaces C++ libraries with verified Lisp-native alternatives. The principle — one representation, one address space, no translation boundaries — is the same in both cases.
---
See also:
- [[id:329bd4fb-702a-4a2b-9c63-69281aacb83a][Knowledge Layers]] — the architecture that extends Schafmeister's principle to verification
- [[id:1c3ec48b-446c-50d2-b53e-126a81f5143f][Architecture]] — why Lisp is the choice for verified infrastructure