roadmap: Passepartout native Org-mode knowledge base
Adds roadmap item for Passepartout to ingest and query org files directly — no pandoc/gbrain bridge. Replaces the current org→md→gbrain pipeline with native org parsing, heading-level vector embeddings, property-based entity extraction, and org-id cross-references. Target v0.8.0-v0.9.0 after gate stack and Screamer planner.
This commit is contained in:
81
ideas/native-org-knowledge-base.org
Normal file
81
ideas/native-org-knowledge-base.org
Normal file
@@ -0,0 +1,81 @@
|
||||
:PROPERTIES:
|
||||
:ID: 7f4e6b9a-2c1d-5e8f-9a3b-6d7c4e5f2a1b
|
||||
:CREATED: [2026-05-23 Sat]
|
||||
:END:
|
||||
#+title: Passepartout Native Org-Mode Knowledge Base
|
||||
#+filetags: :passepartout:roadmap:knowledge:org:gbrain:
|
||||
|
||||
** What
|
||||
|
||||
Passepartout should be able to use Org-mode files directly as its
|
||||
knowledge base — no pandoc conversion, no markdown intermediary.
|
||||
|
||||
Currently gbrain provides vector search + entity linking over markdown,
|
||||
but we bridge via a conversion layer (org → pandoc → markdown → gbrain).
|
||||
This loses Org-mode semantics: properties drawers become flat YAML, tag
|
||||
inheritance is lost, file: links become relative markdown links, TODO
|
||||
states vanish, and the tree structure (headings with content subtrees)
|
||||
collapses into flat markdown headings.
|
||||
|
||||
** Why
|
||||
|
||||
Org-mode's data model is strictly richer than markdown's. A Passepartout
|
||||
that can ingest, index, and query org files natively has:
|
||||
- Property-based entity extraction (no separate links: frontmatter needed)
|
||||
- Tag-inheritance for automatic categorization
|
||||
- TODO/priority/timestamps for knowledge freshness signals
|
||||
- ID-based stable cross-references (org-id) that survive file moves
|
||||
- Heading-level chunking (one heading = one knowledge unit)
|
||||
- The same file format for everything — no split between "authoring format"
|
||||
and "knowledge base format"
|
||||
|
||||
** What it replaces
|
||||
|
||||
The current pipeline: org file → pandoc → markdown file → gbrain import →
|
||||
|
||||
gbrain embed → gbrain query. This is four serial steps with a conversion
|
||||
at each boundary that degrades the data model.
|
||||
|
||||
The target: org file → (Passepartout-native indexer) → query. Zero
|
||||
conversion, zero data loss.
|
||||
|
||||
** Architecture sketch
|
||||
|
||||
A Passepartout-native knowledge module that directly ingests
|
||||
ideas/*.org:
|
||||
|
||||
- Parser: extract each heading as a chunk. Preserve:
|
||||
- Heading path (H1 → H2 → H3) as a hierarchical path
|
||||
- Properties drawer as structured metadata
|
||||
- file: links as typed entity references
|
||||
- org-id as stable identifier
|
||||
- Tags (inherited from parent headings)
|
||||
- TODO state, priority, timestamps
|
||||
|
||||
- Embedder: vector-embed each heading chunk with metadata prefix
|
||||
|
||||
- Query: hybrid search over headings + full-text over content.
|
||||
Result includes the heading path + sibling headings for context.
|
||||
|
||||
- Cross-reference graph: build a typed entity graph from:
|
||||
- file: links → typed reference
|
||||
- org-id links → stable cross-doc reference
|
||||
- Tag co-occurrence → implicit relationship
|
||||
- Same-property values → attribute-based grouping
|
||||
|
||||
- Dream cycle: auto-discover entities from org properties and file:
|
||||
links. Enrich thin sections. Flag sections with stale timestamps.
|
||||
|
||||
** Priority
|
||||
|
||||
Below the gate stack and ACL2 planner (v1.0.0 dependencies) but above
|
||||
the Lisp Machine hardware. Target: v0.8.0-v0.9.0 range, once Screamer
|
||||
planner is stable enough to route queries through the knowledge base.
|
||||
|
||||
The short-term bridge (current) is gbrain with nightly org→md sync.
|
||||
This is adequate while the gate stack and planner are the priority.
|
||||
The native org module replaces gbrain entirely once built.
|
||||
|
||||
** See also
|
||||
[[file:../../concepts/compliance-framework-mapping.org][Compliance framework mapping]]
|
||||
[[file:../../ideas/passepartout-economics.org][Passepartout economics]]
|
||||
Reference in New Issue
Block a user