amr/memex
1
0
Fork 0
Code Issues 12 Pull Requests Actions Packages Projects 1 Releases Wiki Activity

refactor: moved org-agent to its own repository as a submodule

Browse Source
This commit is contained in:
Amr Gharbeia
2026-03-27 15:46:53 -04:00
parent 01f76a4570
commit b7e082c403
176 changed files with 19686 additions and 9665 deletions
Download Patch File Download Diff File Expand all files Collapse all files

87
projects/PROJECT-STATUS.org Normal file
View File

@@ -0,0 +1,87 @@
#+TITLE: Project Status Dashboard
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:28]
#+DATE: 2026-03-08
#+FILETAGS: :status:projects:tracking
* Active Projects Status
| Project | Status | Blocked | Next Action | Owner |
|--------------------+-----------------+-------------+----------------------+-------|
| Gitea Server | 🟡 Initializing | No | Wait + configure | Sol |
| X Bookmarks | 🔴 Stalled | YES - OAuth | Awaiting user export | User |
| Token Optimization | 🟢 Active | No | Stay on free tier | Sol |
| Revenue Skills | 🟡 Partial | No | Continue LinkedIn | Sol |
| Security Hardening | 🟢 Complete | No | Monitor | Sol |
* Stalled Items (Blockers)
*X Bookmarks*
- Blocker: OAuth 2.0 User Context required
- Attempted: OAuth 1.0a, client credentials
- Solution: User data export OR OAuth 2.0 walkthrough
- Status: Waiting for user input
- Action: Leave stalled, don't ask again
* Completing Without Input (In Progress)
*Gitea Server*
- Current: Database initializing
- ETA: 3-5 minutes
- Next: Configure admin, create mind repo
- Cost: Minimal tokens
- Action: Continue autonomously
*Revenue Skills*
- LinkedIn skill: Complete ✅
- Next: Sales Automation research
- Source: Cron social listening data
- Action: Build using existing research
* Daily Autonomous Actions
*Every 2 Hours (Free Operations):*
- Check Gitea status
- Git commit any changes
- Verify sync working
*Daily (7 AM Update):*
- Report status
- Flag real blockers only
- Propose next steps
* Decision Matrix (When to Ask User)
| Scenario | Action |
|-------------------------------+----------|
| Technical setup (Docker, git) | Do it ✅ |
| Budget impact > $5 | Ask 💰 |
| New project direction | Ask 📋 |
| Security changes | Ask 🔒 |
| Stalled 3+ days | Ask ⏸️ |
| Routine completion | Do it ✅ |
* Current Queue (Sol completes without asking)
1. ✅ Finish Gitea setup (in progress)
2. ✅ Create admin account
3. ✅ Create mind repository
4. ✅ Test clone/push/pull
5. ✅ Document setup
6. ⏸️ X bookmarks (if user provides export)
7. 🔄 Resume revenue skill building
8. 🔄 Continue LLM provider research
* Blocked Until User Input
- ❌ X bookmarks (need export)
- ❌ New budget allocation (currently $10)
- ❌ AWS Community Builders (need approval)
---
#+BEGIN_SRC
Rule: If status is 🟡 or 🟢, continue autonomously.
Rule: If status is 🔴 for >24hrs, ask once.
Rule: Update this file after every significant action.
#+END_SRC

9
projects/README.org Normal file
View File

@@ -0,0 +1,9 @@
#+TITLE: projects: Active Projects
#+AUTHOR: Amr
#+CREATED: [2026-03-17 Tue]
#+BEGIN_COMMENT
Active, time-bound efforts with a clear definition of done. Each project has its own dedicated folder for specifications and artifacts.
#+END_COMMENT
* projects: Active Projects
Active, time-bound efforts with a clear definition of done. Each project has its own dedicated folder for specifications and artifacts.

37
projects/agora/agora-requirements-00-readme.org Normal file
View File

@@ -0,0 +1,37 @@
#+TITLE: Agora: Decentralized Social Network
#+AUTHOR: Amr
#+CREATED: [2026-03-17 Tue]
#+BEGIN_COMMENT
A decentralized social network protocol for the ATmosphere (AT Protocol) ecosystem.
#+END_COMMENT
* Agora: Decentralized Social Network
This project contains the specification and analysis for a decentralized social network built on open protocols.
* Project Tasks
See the actionable tasks for this project in [[file:../../gtd.org::*Agora: Decentralized Social Network][GTD.org > Projects > Agora]]
* Key Documents
- [[file:agora-requirements-01-overview.org][01. Overview]]
- [[file:agora-requirements-02-identity.org][02. Identity]]
- [[file:agora-requirements-03-infrastructure.org][03. Infrastructure]]
- [[file:agora-requirements-04-the-primitive.org][04. The Primitive]]
- [[file:agora-requirements-05-social.org][05. Social Spaces]]
- [[file:agora-requirements-05-public-space.org][05. Public Space & Exchange]]
- [[file:agora-requirements-06-exchange.org][06. Exchange]]
- [[file:agora-requirements-06-advanced-integration.org][06. Advanced Integration]]
- [[file:agora-requirements-07-library.org][07. Farcaster & Nostr Integration]]
- [[file:agora-requirements-08-implementation.org][08. Implementation]]
- [[file:agora-requirements-09-strategy.org][09. Strategy]]
- [[file:agora-requirements-10-assessment.org][10. Gap Assessment]]
- [[file:agora-consolidated-gap-analysis.org][Consolidated Gap Analysis]]
* Status
- [X] Concept and atomic notes complete
- [X] Comprehensive specification built
- [ ] Governance decision (DEC-001) pending
- [ ] Implementation planning not started

425
projects/agora/agora-requirements-01-overview.org Normal file
View File

@@ -0,0 +1,425 @@
#+TITLE: Agora Requirements - 01: Protocol Overview and Foundational Principles
#+author: Amero Garcia
#+created: [2026-03-19 Thu 21:07]
#+DATE: 2026-03-20
#+ID: agora-requirements-01-overview
#+STARTUP: content
* 1. Introduction to the Agora Protocol
The Agora Protocol defines a novel architecture for decentralized digital interaction. Its primary objective is to replace extractive, centralized platforms—the era of **"Digital Feudalism"** where corporations own user data and control visibility via secret algorithms—with a decentralized **"Social Operating System"** that provides Identity, Justice, and Commerce for sovereign individuals and communities.
Agora returns power to the edges by providing a modular protocol stack where trust is cryptographic, privacy is inherent, and freedom is architectural. This document provides a comprehensive overview of Agora's foundational principles, core technical differentiators, and a detailed exploration of its capabilities across various use cases, including communication, content creation, e-commerce, collaboration, and liquid democracy. It serves as a high-level technical summary, articulating the design philosophy and the synergistic effects of its integrated components.
* 2. Foundational Principles
Agora's design is predicated upon a set of core principles that collectively ensure a robust, user-centric decentralized network.
** 2.1. User Sovereignty and Data Ownership
Central to Agora is the tenet of user sovereignty. Unlike centralized paradigms where platforms intermediate and often monetize user data, Agora's architecture ensures that all user-generated content and personal data are exclusively owned and controlled by the originating user. This is achieved through client-side encryption, self-hosted or user-controlled Personal Data Stores (PDS), and audience-defined access controls (`access_control`).
** 2.2. Decentralization and Censorship Resistance
The protocol is designed to eliminate single points of failure and control. By distributing data storage across user-controlled PDSs and routing communication through a permissionless Relay Network, Agora inherently resists censorship and external manipulation. There is no central authority capable of unilaterally restricting access, altering content, or deplatforming users.
** 2.3. Authenticity and Verifiability
Every action and piece of content within Agora is cryptographically signed by the originating Persona. This provides an immutable and auditable record, ensuring the authenticity and integrity of all interactions. The content-addressed nature of all data, via Content Identifiers (CIDs), guarantees that content cannot be altered without changing its unique identifier, thereby establishing verifiable provenance.
** 2.4. Privacy by Design
Agora incorporates privacy-enhancing technologies at every layer. End-to-end encryption is a default for private communications, and mechanisms such as Blinded Sharding for social recovery and "Off-the-Record" modes for ephemeral interactions are integrated to minimize metadata leakage and ensure user confidentiality.
* 3. Core Technical Differentiators
Agora's unique capabilities stem from the synergistic integration of three primary technical differentiators: The Note Primitive, Self-Sovereign Identity (Personas and Master Key), and a Distributed Infrastructure (PDS and Relay Network).
** 3.1. The Note Primitive: Atomic Unit of Information
At the heart of Agora's data model is the "Note"—the atomic, universal unit of information. Every piece of content or interaction within the protocol, regardless of its semantic meaning (e.g., a social post, a message, a contract, an encyclopedia entry, a product listing), is encapsulated within a Note.
For a comprehensive technical breakdown of the Note's structure, cryptographic hashing, and content flag schema, see **[[file:agora-requirements-04-the-primitive.org][04: The Primitive]]**.
*** 3.1.2. Benefits of the Unified Note Primitive
The "Everything is a Note" paradigm yields significant architectural advantages:
- **Universal Interoperability:** A single, standardized data model allows any Agora-compatible client application to understand and process any Note, fostering an open ecosystem where diverse applications can seamlessly interact.
- **Immutable Audit Trail:** The content-addressed and signed nature of Notes inherently creates an unalterable, verifiable history of all digital interactions and content evolution.
- **Simplified Development:** Developers can focus on application-layer semantics and user experience, leveraging a robust and consistent underlying data primitive.
** 3.2. Self-Sovereign Identity: Personas and the Master Key
Agora's identity system grants users absolute control over their digital presence, leveraging Hierarchical Deterministic (HD) cryptography to derive and manage multiple functional identities.
*** 3.2.1. The Master Key (Anima)
The Master Key serves as the absolute root of a user's digital being within Agora.
- **Root of Trust:** A single, securely generated and stored secret seed from which all other identities are derived.
- **Hierarchical Derivation:** Utilizes a BIP-44 compatible HD derivation path (`m/44'/1'/account'/persona'/key_purpose/key_index`) to generate an infinite number of unlinkable Personas, each acting as a sovereign sub-root for its own functional keys.
- **Secure Storage:** Recommended for offline storage or within Hardware Security Modules (HSMs) to ensure maximum protection.
*** 3.2.2. Personas: Functional Digital Identities
Personas are the active, functional identities through which users interact with the Agora network.
- **Distinct Identities:** Each Persona represents a distinct Decentralized Identifier (DID), allowing users to maintain separate digital roles (e.g., personal, professional, anonymous) with granular control.
- **Key Management:** Each Persona possesses its own signing and encryption keypairs, which can be revoked or rotated independently without affecting the Master Key or other Personas.
- **Asset Ownership & Rights:** Personas are analogous to legal entities, capable of owning digital assets (e.g., Bitcoin wallets), entering into binding contracts, and claiming protected rights such as due process and freedom of expression.
*** 3.2.3. Decentralized Identity Management Benefits
- **Absolute User Control:** Full ownership of identity and keys, independent of any central authority.
- **Granular Access Control:** Ability to manage access to specific Personas and their associated data.
- **Efficient Organizational Revocation:** For collective entities, the HD model enables atomic revocation of access for departing members directly from the Master Key control point, streamlining offboarding and enhancing security across all associated assets and services.
- **Resilient Social Recovery:** Utilizes Shamir's Secret Sharing with trusted "Guardians" to enable Master Key recovery without reliance on centralized services.
** 3.3. Distributed Infrastructure: PDS, Relays, and Thin Clients
Agora's infrastructure is specifically engineered to underpin user sovereignty, data ownership, and censorship resistance.
*** 3.3.1. Personal Data Store (PDS): The User's Digital Vault
The PDS is the central component for data ownership, acting as the user's sovereign digital vault.
- **Exclusive Control:** Every user controls their own PDS, whether self-hosted or through a trusted provider.
- **Master Archive:** Stores all user content (client-side encrypted) and identity data.
- **Access Gatekeeper:** Enforces access control, issuing decryption keys based on validated credentials or payments.
- **PDS-as-a-Service:** Services can integrate seamlessly, offering free sign-ups with grace periods and requiring in-Agora payments (e.g., Lightning) for continued service, bypassing traditional financial intermediaries.
*** 3.3.2. Relay Network: The Intelligent Communication Backbone
The Relay Network forms the intelligent communication backbone of Agora, efficiently routing encrypted Notes between Personas.
- **Ephemeral Routing:** Relays route ciphertext based on CIDs and Persona subscriptions, without long-term storage of user data.
- **Pub/Sub Model:** Facilitates efficient, real-time delivery of Notes based on user subscriptions.
- **Censorship Resistance:** Users can publish to multiple Relays, ensuring availability and resilience against censorship.
*** 3.3.3. Agile Client Architecture: Broad Accessibility and Adaptability
Agora adopts a flexible client architecture to balance user sovereignty with broad accessibility, particularly concerning app store ecosystems.
- **PDS-Proximate Logic:** Core application logic can reside and execute securely on the user's PDS.
- **Thin Clients:** Edge devices (mobile, desktop) run lightweight applications that interface with the PDS, mitigating app store restrictions and reducing device resource demands.
- **Strategic Imperative:** This architecture ensures Agora's reach to a wider user base while maintaining security and privacy.
* 4. Agora Use Cases: A Paradigm Shift
The synergistic combination of Agora's core differentiators enables a wide array of transformative use cases, redefining digital interaction across multiple domains.
** 4.1. Decentralized Social Interaction
Agora provides a robust framework for secure, private, and censorship-resistant interaction, moving beyond traditional platform-controlled silos.
*** 4.1.1. Asynchronous Interaction (The Note Primitive)
- **Unified Model:** All async interactions—whether directed messages or broadcast posts—are built on the same cryptographically signed **Note** primitive, utilizing the **DIDComm** protocol for secure transport.
- **Storage Sovereignty:** Employs a "Copy-on-Send" model for directed communication (ensuring recipient data ownership) and a "Reference-on-Send" model for broadcast content (ensuring owner control). The PDS acts as an encrypted mailbox proxy.
- **End-to-End Encryption:** Default for directed communications, utilizing standard encrypted envelopes. Double Ratchet and MLS ensure forward secrecy.
*** 4.1.2. Synchronous Interaction (Real-time)
- **WebRTC Integration:** Supports peer-to-peer real-time chat, voice, and video calls with end-to-end encryption and **decentralized signaling** via DIDComm handshakes.
- **Off-the-Record Mode:** Provides absolute privacy for ephemeral interactions by utilizing extremely short `ephemeral_duration` or bypassing PDS storage entirely, with content existing only in volatile client memory.
** 4.2. Social Publishing and Knowledge Management
Agora fundamentally reshapes how content is created, published, and managed, empowering creators and ensuring verifiable knowledge.
*** 4.2.1. Feeds and Pages
- **Immutable History:** Social posts (`is_feed: true`) and wiki pages (`is_feed: false`) are signed Notes, providing an unalterable history of creation and edits.
- **Auditable Threads:** Replies are Notes referencing parent CIDs, creating verifiable discussion threads across the distributed network.
- **Direct Monetization:** Paywalled content and seeder rewards enable direct creator-to-consumer economic models via Lightning micro-payments.
*** 4.2.2. Decentralized Wikis and Encyclopedias
- **Versioned Pages:** Each wiki page is an `is_feed: false` Note, with edits creating new Notes that supersede previous versions, building an immutable, auditable version history.
- **Collaborative Ownership:** Access control and editing rights are managed via **Contract Notes** (Consent or Service Contracts) with `Collective Personas`.
- **Incentivized Contributions:** Micro-payments can reward contributions, fostering a collaborative, trustworthy, and censorship-resistant knowledge base.
*** 4.2.3. Verifiable News Ecosystem
- **Signed Articles:** News articles are `is_feed: true` Notes, signed by journalist Personas, ensuring clear provenance and ownership.
- **Immutable Record:** All versions of an article are archived, preventing historical revisionism or "disappearing" stories.
- **Decentralized Distribution:** Resilient against censorship attempts, as distribution occurs via the Relay Network.
- **Reputation Systems:** Notes referencing Persona DIDs and community-driven verification mechanisms can build transparent reputation for sources and journalists.
** 4.3. Decentralized E-commerce and Markets
Agora enables peer-to-peer economic interaction without intermediaries, fostering transparent and auditable marketplaces for goods and services.
*** 4.3.1. Market Interaction Contracts
- **Offer as Early Contract:** A **Contract Note** (product listing) serves as a unilateral declaration of intent (**Offer**) by a seller, transitioning into a bilateral agreement (**Take**) upon buyer acceptance.
- **Transparent Listings:** Offers are signed Notes, providing verifiable details of items or services.
- **Questions and Reviews:** Notes that `reply_to` or `references` listings allow public or private dialogue, building transparent market trust and reputation based on Owner Reputation.
*** 4.3.2. Fungible vs. Non-fungible Assets
- **Non-Fungible:** Agora's **Contract Note** model is inherently well-suited for unique goods and services (e.g., digital art, custom work), with each contract representing a distinct agreement.
- **Fungible:** While Agora provides the identity, communication, and settlement rails (e.g., Lightning micropayments), high-speed trading of fungible assets (e.g., cryptocurrencies, commodities) would require specialized architectural layers (e.g., decentralized exchanges or AMMs) built *on top of* the Agora protocol for order matching and liquidity.
** 4.4. Decentralized Collaboration and Project Management
Agora offers robust primitives for secure, auditable collaboration, empowering teams and communities.
*** 4.4.1. Version-Controlled Documents and Code
- **Signed Commits/Edits:** Each change to a collaborative document or codebase is a signed Note with appropriate `content_type` (for code) or a versioned `is_feed: false` Note (for documents), creating an immutable, auditable history.
- **Collective Ownership:** Repositories or documents can be owned by `Collective Personas`, with access and editing rights managed via **Contract Notes**.
- **Decentralized GitHub/Git Integration:** Codebases are stored as Merkle DAGs of commit Notes, enabling decentralized version control. Issues and pull requests are also Notes, facilitating transparent project management.
*** 4.4.2. Project Management and Task Tracking
- **Tasks as Contracts:** Project tasks are **Contract Notes** in a negotiation state, allowing for assignment, progress tracking, and integration with payment mechanisms.
- **Incentivized Development:** Lightning bounties (**Contract Notes**) can be attached to issues or tasks, directly rewarding contributions upon completion and verification.
*** 4.4.3. The Aletheia Portfolio (Professional Integration)
The convergence of native hosting, identity, and contracts enables a unified professional workflow. For example, a freelance photographer can:
- **Generate & Publish:** Build a professional portfolio using a static site generator and publish it natively to the network via their "Professional Persona" root CID.
- **Sovereign Hosting:** The portfolio remains available via any Gateway, resilient against PDS downtime.
- **Contractual Linkage:** Directly link the portfolio Note to a binding service contract for client hiring, with payments settled via Lightning.
** 4.5. Liquid Democracy and Governance: Evolvable Collectives
Agora's identity and contract primitives lay the groundwork for a dynamic, adaptive model of decentralized governance that moves beyond the rigidity of traditional blockchain-based DAOs.
*** 4.5.1. Adaptive Constitutions and Policy Execution
- **Signed Votes and Execution:** Individual votes are signed Notes that `references` a proposal CID. Unlike immutable blockchain code, Agora governance is built around **Adaptive Constitutions**.
- **Recursive Rule-Making:** Successful votes trigger the Governance Executable Module (GEM) to automatically update the Collective's policy parameters (e.g., membership fees, arbitration rules) in its active Smart Constitution.
- **Immutable History, Mutable State:** While the complete audit trail of every vote and version is permanently recorded as a chain of CIDs, the organization can evolve its logic over time without requiring complex migrations.
*** 4.5.2. Decentralized Autonomous Organizations (DAOs)
- **Foundation Contracts:** DAOs are formalized as `Collective Personas` governed by a set of foundational `Contract Notes` that define membership, treasury management, and decision-making processes.
- **Forks as Safety Valves:** Because Agora is permissionless, minorities can "fork" a Collective by creating a new Persona based on an earlier constitutional CID, ensuring protection against majority tyranny and preserving community intent.
- **Transparent Operations:** All operational decisions, proposals, and expenditures within a DAO are conducted and recorded as signed Notes and Contracts, providing 100% transparency to participants.
* 5. Conclusion: Towards a Self-Sovereign Digital Future
The Agora Protocol is meticulously designed to serve as the foundational layer for a new era of decentralized digital interaction. By unifying identity, data, and communication under the immutable, verifiable, and user-owned "Note" primitive, coupled with a distributed infrastructure and self-sovereign identity management, Agora offers a robust and resilient alternative to centralized systems. Its capabilities span from secure personal communication to complex global e-commerce, from collaborative knowledge creation to transparent democratic governance. Agora empowers individuals and collectives to reclaim their digital sovereignty, fostering an internet where trust is cryptographic, privacy is inherent, and freedom is architectural.
* Bootstrapping & Progressive Decentralization
** The Cold Start Problem
A decentralized social network faces an existential network effect challenge. Users will not join if there is no content, and creators will not post if there are no users. Agora solves this through *Progressive Decentralization*.
** Bootstrap Sequence
The system MUST provide a smooth onboarding experience, especially in the first five minutes:
1. *Persona Selection:* A simple UI for selecting a "Persona Alias" (e.g., `@amr`).
2. *Key Generation:* High-speed, hardware-backed key derivation (BIP-32) happens in the background.
3. *PDS Selection:* Users are prompted to choose between *"Managed Hosting"* or *"Self-Hosting"*.
4. *Relay Discovery:* The client automatically connects to a set of high-reputation, geographic "Bootstrap Relays" to fetch initial content.
5. *Interest Capture:* Users select topics/interests to seed initial content recommendations.
6. *Migration Option:* Offer to import from Twitter, Reddit, Mastodon, etc. to bootstrap social graph.
** Interest Capture
*** Purpose
Reduce "empty feed" problem by immediately showing relevant content based on user interests.
*** Implementation
- *Explicit Selection:* Users pick from curated categories (Technology, Art, Politics, Science, etc.).
- *Implicit Extraction:* If user imports from centralized platforms, parse their follows/history to infer interests.
- *AI Assistance:* Sub-Agent can analyze imported content to suggest interest categories.
*** Content Seeding
- Client fetches popular public content in selected interest areas.
- Initial feed populated with high-quality, diverse content from selected topics.
- Users can refine interests over time (feedback loop).
** Migration and Social Graph Bootstrap
*** Supported Platforms
- *Twitter/X:* Import followed accounts via archive export or API.
- *Reddit:* Import subscribed subreddits and frequent communities.
- *Mastodon/ActivityPub:* Native federation, direct import of follows.
- *LinkedIn:* Professional connections import.
- *Blog/RSS:* Import RSS subscriptions as interest sources.
*** Privacy Considerations
- Migration is *opt-in*, not mandatory.
- Users choose which platforms to import from.
- Imported data is stored locally; only new Agora follows are public.
- Users can audit and remove imported suggestions before
confirming follows.
*** Discovery Expansion
- Suggest high-reputation personas in imported interest areas.
- Show "Your Twitter follows on Agora" for easy reconnecting.
- Surface collectives matching imported community memberships.
** The "Four Orders of Growth" (Scaling Sequence)
Scaling a decentralized network requires shifting from "Hand-holding" to "Protocol Incentives." Agora follows a strictly defined orders-of-magnitude growth strategy:
*** Order 1: The First 1,000 (The "Founders")
- **Target:** Technical enthusiasts, privacy advocates, and niche professional guilds (e.g., decentralized AI devs).
- **Tactics:** Manual onboarding. We seed the first Arbitration Guilds.
- **Success Metric:** First successful civil contract signed and settled via HODL invoice.
*** Order 2: The 10,000 (The "Communities")
- **Target:** Small NGOs, local trade groups, and content creator "Swarms."
- **Tactics:** Launch the Community PDS templates. Enable "One-Click Hub" setup so a leader can host their entire group.
- **Success Metric:** The emergence of "Community Algorithms"—feeds curated by these 10k users that provide unique value.
*** Order 3: The 100,000 (The "Marketplace")
- **Target:** Freelancers, gig workers, and "Etsy-style" digital sellers in regions with weak rule of law.
- **Tactics:** Focus on Mobile UX. The app must feel "normal." Introduce Automated Key Rotation so non-tech users don't fear losing their phones.
- **Success Metric:** $1M+ in peer-to-peer transaction volume via SCAL contracts.
*** Order 4: The 1M+ (The "Ecosystem")
- **Target:** The general public.
- **Tactics:** The Algorithm Marketplace becomes the draw. People join because "The Scientific Lens" or "The Family Lens" on Agora provides a better mental health experience than the addictive AI of centralized apps.
- **Success Metric:** Total P2P bandwidth (Seeding) exceeds the capacity of a mid-sized centralized CDN.
** Progressive Decentralization Phases
*** Phase 1: Managed Service (Days 1-100)
- *Centralized Experience:* The initial developers provide high-performance, managed PDS and Relay services to ensure a seamless "Twitter-like" experience.
- *Focus:* User acquisition and content density in specific "Alpha" collectives (e.g., AI/Dev communities).
*** Phase 2: Hybrid (Year 1)
- *Self-Hosting Options:* Users are encouraged to move to their own PDS or third-party providers as the ecosystem matures.
- *Social Graph Interoperability:* Enabling users to "Follow" personas across different PDS providers.
*** Phase 3: Full Decentralization (Year 3+)
- *No Central Authority:* The original developers become just one of many PDS and Relay providers.
- *Protocol Stability:* The V1.0 spec is finalized, and development is driven by the *Agora Governance Model*.
** Incentivized Growth
- *Referral Satoshis:* Early users can be rewarded in satoshis for successful referrals that lead to high-reputation personas.
- *Micro-Grant Bounties:* Funding developers to build "Must-Have" Agora apps through the economic layer.
* Strategic Positioning
** Platform Replacement Strategy
Rather than positioning Agora as an existential threat to Big Tech (Apple, Google, Meta), Agora should first target underserved communities and platforms with clear pain points:
*** Phase 1: Niche Community Platforms
** Forums (Reddit, phpBB, vBulletin)
- *Pain Point:* Centralized moderation, censorship, data mining.
- *Agora Advantage:* Sovereign moderation, portable identity, no platform lock-in.
- *Target Communities:* Developer forums, hobbyist communities, support forums.
** Visual Discovery (Pinterest)
- *Pain Point:* Algorithmic manipulation, advertising-driven discovery.
- *Agora Advantage:* User-chosen discovery algorithms, no surveillance capitalism.
** Professional Communities (LinkedIn, corporate intranets)
- *Pain Point:* Professional data exploitation, platform-controlled networking.
- *Agora Advantage:* Sovereign professional identity, portable reputation.
** Creator Platforms (Medium, Substack)
- *Pain Point:* Platform fees (10-50%), censorship risk, no portability.
- *Agora Advantage:* Near-zero fees, content ownership, subscriber portability.
** Marketplaces (eBay, Etsy)
- *Pain Point:* High fees (10-15%), centralized dispute resolution, account bans.
- *Agora Advantage:* Low fees (<5%), transparent reputation, sovereign stores.
** Adult Content (Pornhub, OnlyFans)
- *Pain Point:* Censorship, payment processor discrimination, lack of privacy.
- *Agora Advantage:* Censorship-resistant, Lightning-native payments, pseudonymous.
** Specialized Communities (QRZ, Logbook of the World)
- *Pain Point:* Aging infrastructure, lack of modern features, centralization.
- *Agora Advantage:* Modern protocol, extensible, community-governed.
** Decentralized Communities (Nostr, Fediverse)
- *Pain Point:* Fragmentation, lack of economic layer, UI/UX challenges.
- *Agora Advantage:* Unified protocol, Lightning integration, polished UX.
*** Phase 2: Horizontal Expansion
Once established in niche communities:
- *Bridge to Big Tech:* Migration tools for Twitter, Instagram, etc.
- *Enterprise Adoption:* Sovereign collaboration tools for companies.
- *Mass Market:* Only after protocol stability and network effects proven.
** Big Tech Analysis (Long-term)
While not the immediate focus, Agora's architecture eventually threatens Big Tech:
*** Meta/Facebook
- *Risk:* Portable identity undermines social graph lock-in.
- *Timing:* Year 3+ after network effects established.
*** Apple
- *Opportunity:* Privacy alignment, hardware security integration.
- *Risk:* App Store policies may restrict Agora clients.
*** Google
- *Risk:* Search dominance challenged by social-graph-first discovery.
- *Opportunity:* Federated search, open data standards.
** The "Trojan Horse" Strategy
- *Start Small:* Win over frustrated communities on Reddit, forums, Discord.
- *Build Bridges:* ActivityPub/Mastodon integration, Twitter migration tools.
- *Demonstrate Value:* Show "You trade 2 seconds for freedom" is worth it.
- *Let Giants React:* By the time Big Tech notices, Agora is entrenched.
** Strategic Assessment
- *Cold Start Problem:* The most significant hurdle. Requires aggressive bootstrapping in the first year.
- *Success Probability:* 30-50% for 100K users; 10-20% for 1M users (within 3 years).
- *The "Unstoppable" Factor:* Once the protocol is decentralized and the first million users are on-boarded, it becomes nearly impossible to shut down.
* Legal & Regulatory
** The Jurisdictional Challenge
As a decentralized protocol with no central authority, Agora is designed to operate across international jurisdictions.
** Content Moderation & Liability
*** The "Dumb Pipe" Strategy
- *Relays as Carriers:* Relays act as dumb, ephemeral conduits for encrypted CIDs. Their legal standing is similar to ISPs or postal services.
- *PDS Sovereignty:* The user (the PDS owner) is the only entity with the ability to decrypt and view the content.
*** The CSAM Challenge
- *Zero Tolerance Policy:* Agora's governance model includes protocol-level consensus for universally illegal content.
- *Network-Level Blocking:* High-reputation Relays can block CIDs associated with CSAM.
- *Fundamental Tension:* The trade-off between total privacy (E2EE) and the ability to detect illegal content.
** Financial Regulation & AML
- *Micro-Payments:* Lightning Network payments generally fall below traditional AML/KYC thresholds.
- *Non-Custodial:* Agora is non-custodial. Users control their own keys and funds.
** Data Privacy (GDPR/CCPA)
- *The "Right to be Forgotten":* In a CID-based system, data is not "deleted" but can be "un-indexed" or its decryption keys revoked.
- *Sovereign Control:* Users have absolute control over their own data in their PDS.
** Strategy for Resistance
- *Legal Defense Collective:* Establishing a legal defense fund (Collective Persona) to support Relay and PDS operators.
- *Transparency Reports:* High-reputation Relays and PDS providers should publish transparent reports on compliance.
* Game Theory & Economic Attacks
** Attack Vectors
- *Sybil Attacks:* Creating millions of fake personas.
- *Relay Censorship:* Majority of Relays blocking specific content.
- *Economic Spam:* Paying minimal fees to flood the network.
- *Governance Capture:* Attempting to take over protocol governance.
** Defenses
- *Reputation Systems:* Economic and social costs of attack increase with reputation requirements.
- *Multi-Home Relays:* Users can always switch to uncensored Relays.
- *Fee Markets:* Dynamic pricing makes spam economically unviable.
- *Fork Threat:* Credible threat of fork prevents governance capture.
* Related Documents
- [[id:agora-bootstrap-sequence][Agora Bootstrap Sequence]]
- [[id:agora-strategic-positioning][Agora Strategic Positioning]]
- [[id:agora-legal-regulatory][Agora Legal & Regulatory Strategy]]

612
projects/agora/agora-requirements-02-identity.org Normal file
View File

@@ -0,0 +1,612 @@
* Identity: The Genesis of Your Digital Being
** Master Key (Psyche)
The Master Key, often referred to as "Psyche" (Latin for soul or animating principle), is the absolute foundation of your digital identity in Agora. It serves as your unassailable root of trust, from which every other functional identity (your Personas) is cryptographically derived. This section meticulously outlines the Master Key's core requirements, elucidates how it empowers flexible organizational structures, and details the robust mechanisms for its secure management and resilient recovery. It is the ultimate key to your self-sovereignty.
*** Requirements & The Root of Trust
- The system MUST cryptographically decouple identity from the master cryptographic material, ensuring that derived keys can be managed independently while retaining the Master Key as the root of authority.
- Users MUST possess one Master Key (the "Seed") that is generated and stored securely, ideally never exposed to the network or a general-purpose operating system.
- All functional identities (Personas) MUST be derived from this single Master Key seed using Hierarchical Deterministic (HD) derivation, providing an organized and secure structure for digital identities.
- The Master Key MUST be generated from a minimum of 256 bits of high-quality, cryptographically secure entropy.
- The Master Key MUST be encoded as a BIP-39 mnemonic phrase (typically 24 words) for human-readable, offline backup and disaster recovery.
- The Master Key MUST be stored offline (e.g., on paper, engraved metal) or within a tamper-resistant hardware security module (HSM) for maximum protection against compromise.
- The system MUST utilize a custom HD derivation path: `m/44'/1'/account'/persona'/key_purpose/key_index`, uniquely identifying Agora's identity structure within the broader BIP-44 ecosystem. (*Note: Index `1'` is utilized for the experimental/testnet phase; a unique permanent index will be registered for the Agora Mainnet via SLIP-0044.*)
- This path allows each Persona to act as a "Sub-Root," deriving its own autonomous functional keys (e.g., for Bitcoin, Lightning, PGP, or SSH) without requiring access to the Master Key once the Persona's extended private key (xpriv) is provisioned to a device.
- Each `persona'` index within this derivation path MUST represent a distinct DID (Decentralized Identifier), ensuring global uniqueness and unlinkability.
- The system MUST allow a single Master Key seed to generate an infinite number of unique, unlinkable personas, providing unparalleled flexibility for different digital roles.
- Each Persona MUST possess its own distinct Ed25519 keypair for cryptographic signing and an X25519 keypair for robust encryption.
- The system MUST enable the revocation and rotation of individual Persona keys without compromising the integrity of the Master Key or affecting other derived Personas, offering granular control and enhanced security.
- The identity lifecycle MUST be managed via **KERI (Key Event Receipt Infrastructure)**, ensuring identities remain persistent regardless of key rotations.
- All key rotations and membership changes MUST be recorded in an append-only, verifiable **Key Event Log (KEL)**.
*** Master Key Interaction Protocol: Derivation vs. Action
It is critical to distinguish between the Master Key's role in *Persona derivation* and a Persona's role in *network actions*.
- **Master Key for Derivation (Creation of New Identities):** The Master Key is the sole cryptographic origin for generating new Accounts and Personas. Any creation of a new Persona (or Account) in your identity tree requires interaction with the Master Key. This ensures a clear, auditable, and cryptographically sound chain of custody from your single root to every Persona. While this might occasionally require accessing a hardware wallet for a new Persona setup, it safeguards the integrity of your entire identity graph.
- **Persona Keys for Actions (Interacting with the Network):** Once a Persona is created, it becomes a fully independent, active agent in the Agora network. All subsequent actions—signing messages, publishing content, entering into contracts (including Foundation Contracts), acting as a guardian for social recovery, or joining an organization—are performed using the Persona's own distinct keypairs. **The Master Key is explicitly *not* needed for these daily operational activities.** This design minimizes the Master Key's exposure, keeping it safely offline and dramatically reducing the frequency of hardware wallet interactions for routine tasks.
This clear separation ensures that your Master Key functions as a secure, infrequent-use root for identity creation and recovery, while your Personas are empowered to execute all network interactions autonomously.
*** Master Key Recovery: The Offline Root Seed
**** Shamir's Secret Sharing: Distributed Trust
If a user loses access to their offline Master Key, Agora's Social Recovery mechanism provides a decentralized, self-sovereign solution:
1. Master Key is cryptographically pre-split into N shards using Shamir's Secret Sharing.
2. These shards are securely distributed to M-of-N "Guardians" (trusted friends or professional services).
3. Recovery only requires M guardians to recombine their shards, rebuilding the Master Key offline.
4. This elegantly avoids reliance on centralized "Account Recovery" services, keeping you in control.
**** Social Recovery Privacy (Blinded Sharding)
***** Blinded Sharding Concept
Standard Shamir's Secret Sharing reveals which guardians hold shards when shards are stored on the PDS. *Blinded Sharding* hides this information from the PDS while still enabling recovery.
***** How Standard Shamir Reveals Guardians
- Shards stored as: `(index, shard_value)` pairs
- PDS sees: "Guardian #1 has this shard, Guardian #2 has that shard"
- Reveals: Who the user's trusted contacts are (social graph)
***** Blinded Sharding Solution
Instead of storing `(index, shard)` directly, use *cryptographic blinding*:
****** Step 1: Generate Mask
- Random mask `m` for each shard
- Mask is encrypted to Guardian's public key
- Only Guardian can unmask the shard
****** Step 2: Store Blinded Shard
```
Stored on PDS:
- Blind = hash(shard || guardian_pubkey)
- Shard encrypted to Guardian's key (X25519 + AES-GCM)
- Guardian ID: NOT stored in plaintext, only hash
```
****** Step 3: Recovery
- Guardian sends encrypted shard response
- User decrypts using their private key
- Verifies shard validity via Shamir reconstruction
- PDS never learns which Guardians participated
***** Implementation
#+begin_src typescript
interface BlindedShard {
// Public, stored on PDS
shardHash: string; // hash(shard || guardian_pubkey)
encryptedShard: string; // X25519 + AES-GCM encrypted
// Not stored: Guardian ID
// Guardian identified by: can decrypt `encryptedShard`
// (only valid Guardian has private key)
}
interface GuardianConfig {
guardianDID: string; // Known to user, NOT to PDS
guardianPublicKey: X25519PublicKey;
}
// Shard creation
function createBlindedShard(
shard: Buffer,
guardianConfig: GuardianConfig
): BlindedShard {
const shardId = hash(sha256, [shard, guardianConfig.guardianPublicKey]);
const encrypted = x25519_encrypt(shard, guardianConfig.guardianPublicKey);
return {
shardHash: shardId,
encryptedShard: encrypted
};
}
// Reconstruction
async function recoverShard(
blindedShard: BlindedShard,
guardianPrivateKey: X25519PrivateKey
): Promise<Buffer> {
// Guardian decrypts
const decrypted = x25519_decrypt(
blindedShard.encryptedShard,
guardianPrivateKey
);
// Verify not corrupted
if (hash(sha256, [decrypted, guardianPublicKey]) !== blindedShard.shardHash) {
throw new Error("Shard verification failed");
}
return decrypted;
}
#+end_src
***** Security Properties
1. *PDS doesn't know Guardians:* Only sees random hashes and ciphertexts
2. *PDS can't correlate:* Different users' Guardians appear as different random data
3. *Guardian anonymity:* Recovery can happen without PDS knowing who responded
4. *Integrity verified:* Hash check prevents corrupted shards
**** Shamir's Secret Sharing Parameters
***** Standard Parameters
- *Scheme:* Shamir's Secret Sharing over GF(2^256)
- *Threshold (M):* 3 (minimum to reconstruct)
- *Total Shares (N):* 5 (total generated)
- *Security:* 256-bit security (same as Bitcoin private keys)
***** Share Distribution
- *Guardian 1:* Trusted friend, geographically distant
- *Guardian 2:* Family member
- *Guardian 3:* Professional service (optional)
- *Guardian 4:* Personal cloud/HSM backup
- *Guardian 5:* Safety deposit box (physical)
***** Recovery Probability
- *1 guardian fail:* Still recoverable (4 of 5 remaining)
- *2 guardians fail:* Still recoverable (3 of 3 remaining)
- *3+ guardians fail:* Unrecoverable (design choice)
** HD Derivation
*** HD Derivation Architecture (BIP-32/44)
- Agora uses a custom derivation path to ensure interoperability: `m/purpose'/persona_index'/profile_index/key_type`.
- The `persona_index'` MUST be hardened to prevent correlation attacks between different personas.
- Each `persona_index'` MUST represent a distinct DID (Decentralized Identifier).
- This allows a single seed to generate infinite, unlinkable personas.
*** Decoupled Key Provisioning & Watch-Only Master
To minimize the exposure of the Master Seed, client applications MUST support decoupled key strategies:
- **Subkey Injection:** The client MUST allow importing a standalone extended private key (xpriv) or raw private key for a specific `persona_index'`. The app operates strictly within the scope of that imported key and cannot derive sibling personas.
- **Multi-Device Sync:** Users can securely provision a secondary device (e.g., a mobile phone) by injecting a Persona-level subkey, keeping the Master Seed in a physical hardware vault.
- **Watch-Only Master:** The client MAY allow storing the Master Extended Public Key (xpub). This creates an "Auditor View," enabling the device to monitor all derived Personas and balances without possessing the private keys necessary to authorize transactions or sign events.
*** Cross-Persona Interaction (The "Bridge")
The system MUST allow a user to prove relationships between their own Personas without publicly linking them to a single Master Seed.
- **Zero-Knowledge Proofs (ZKP):** A user can "Attest" that a specific capability or badge belongs to them across personas. For example, a "Pseudonymous Developer" Persona can use a ZKP to prove it holds a "Verified Citizen" badge issued to its associated "Legal Persona," proving citizenship without revealing *which* citizen they are.
*** Index Management (Gap Limit Protocol)
**** Concept
Clients must efficiently discover active personas derived from a Master Seed without performing an exhaustive scan of the entire index space. The Gap Limit Protocol defines the search window and criteria for identifying active personas during recovery or sync.
**** Specification
- *Gap Limit (L):* The number of consecutive unused persona indices to check before stopping the scan. The default Agora gap limit is *20*.
- *Active Persona Detection:* A persona index is considered "active" if it has:
1. A registered name in the Tier 2 Global Registry.
2. Any Content Objects published to a PDS/Relay.
3. Any incoming attestations from other personas.
- *Scan Window:* Clients scan indices in increments of L. If any index within $[i, i+L-1]$ is active, the window shifts to $[i+L, i+2L-1]$.
**** Recovery Workflow
1. Derive Master Key.
2. For each account index (starting from 0'):
a. Scan persona indices 0 through L-1.
b. If any active persona is found, continue scanning the next window of L.
c. If no active personas are found in the window, stop scanning this account.
3. If no active personas are found in the first window (0 through L-1) of an account, stop scanning accounts.
*** Centralized Revocation Efficiency: The Atomic Kill Switch for Organizations
**** Comparison to Traditional Systems
- **Traditional:** Partner leaves → Manually update 50+ passwords, revoke individual access rights across numerous platforms (email, bank, cloud storage, code repos, etc.). High risk of oversight and residual access.
- **Agora:** Partner leaves → One managed revocation at the Master Key level (or their specific Persona's access derivation is severed) → Instant, automatic severance of access across all derived keys (company Bitcoin, PGP, SSH, etc.).
This mechanism ensures that the collective's assets remain secure and under the control of the remaining authorized members, providing a robust solution for organizational identity management.
** Accounts
*** Account-Level Strategy: Organizing Your Digital Life
**** Derivation Path with Accounts
```
m/44'/1'/0'/0' # Account 0, Persona 0 (default personal)
m/44'/1'/0'/1' # Account 0, Persona 1
m/44'/1'/1'/0' # Account 1, Persona 0 (work account)
m/44'/1'/1'/1' # Account 1, Persona 1 (work, second persona)
m/44'/1'/2'/0' # Account 2, Persona 0 (anonymous/account-specific)
```
**** Account Separation Strategies
***** Personal vs Work
- *Account 0:* Personal life, friends, family
- *Account 1:* Professional identity, colleagues
- Each account has its own set of personas (persona index within account)
***** Anonymous vs Primary
- *Account 0:* Primary public identity
- *Account 2+:* Anonymous or temporary accounts
- Easy rotation: revoke entire account, create new account index
***** Organizational Accounts
- *Account 3+/Specific Values:* Could be assigned for specific organizations
- Each organization gets its own account namespace
**** Account Naming and Metadata
- *Account Aliases:* User-defined labels ("Personal", "Work", "Anonymous")
- *Account Icons:* Visual distinction in client UI
- *Account Metadata:* Not stored on-chain, local to client
- *Account Lock/Unlock:* Separate authentication for each account
**** Account-Specific Configuration
- *Default PDS:* Each account can use different PDS providers
- *Default Relays:* Account-specific relay preferences
- *Contact Isolation:* Contacts in one account not visible in others (by default)
- *Content Visibility:* Cross-account content visibility configurable
**** Cross-Account Operations
- *Account Switching:* Quick switch without re-entering Master Key
- *Cross-Account References:* "Share from Work to Personal" with privacy controls
- *Unified Inbox:* Optional aggregation of notifications across accounts
- *Backup Strategy:* Account-level backup (export all personas in account)
**** Security Considerations
- *Same Master Key:* All accounts derived from same seed—compromise of Master Key compromises all accounts.
- *Different Lock Codes:* Each account can have its own unlock PIN/biometric.
- *Plausible Deniability:* Hidden accounts possible (account index not sequential).
**** Developer Implementation
To generate a new Persona:
1. Load Master Seed.
2. Derive path `m/44'/1'/0'/N'` where N is the next available index.
3. Generate Ed25519 keypair from the derived entropy.
4. Construct the DID: `did:agora:<pubkey_multibase>`.
**** Account-Level Technical Specification: The Blueprint for Digital Organization
The Account-Level Strategy is built upon a robust technical foundation that rigorously adheres to and extends industry standards for cryptographic key derivation. This specification ensures predictable, secure, and interoperable management of multiple digital identities from a single Master Key.
***** BIP-44 Derivation Path Structure: Agora's Standard
Agora meticulously follows the established BIP-44 standard for hierarchical deterministic key derivation paths. This standardized structure guarantees compatibility and logical organization of your digital identities.
`m / purpose' / coin_type' / account' / persona' / key_purpose / key_index`
In Agora's context, this is specifically mapped as:
`m / 44' / 1' / account' / persona' / key_purpose / key_index`
- *Purpose (44'):* This is a hardened derivation, as prescribed by BIP-44, signifying that the derived keys are cryptographically isolated from the Master Key.
- *Coin Type (1'):* This is a hardened derivation, and `1'` is the officially registered SLIP-0044 index specifically allocated for the Agora Protocol.
- *Account (account'):* This is a hardened derivation. It provides independent, cryptographically isolated persona namespaces, enabling users to manage distinct organizational or contextual groupings of Personas.
- *Persona (persona'):* This is a hardened derivation. Each index represents a distinct, autonomous digital identity (DID). Hardening ensures that compromising one Persona's keys cannot compromise sibling Personas or the Master Key.
- *Key Purpose (key_purpose):* This non-hardened layer allows a single Persona to act as a "Sub-Root" to derive autonomous functional keys for specific tasks without requiring the Master Key. Examples:
- `0`: Primary Identity/Signing Key (Ed25519)
- `1`: General Encryption Key (X25519 for DIDComm)
- `2`: Bitcoin/Lightning Node Key
- `3`: Stablecoin/EVM Wallet
- *Index (key_index):* This is a non-hardened, incremental index used to generate multiple unique keys of a specific purpose (e.g., generating new receive addresses for a Bitcoin wallet).
*Note: This structure ensures that once a Persona's xpriv is loaded on a mobile device, that device can derive all necessary sub-wallets autonomously without re-accessing the Master Key.*
***** Account Types and Reserved Indices: Standardized Compartmentalization
While the choice of account indices is technically arbitrary, Agora recommends the following conventions. These standardized assignments ensure client interoperability and provide a common language for managing distinct digital compartments.
- *0': Primary Account.* This is the default account for a user's primary personal identity, social interactions, and other everyday personas.
- *1': Professional Account.* This account is dedicated to professional identity, credentials, work-related personas, and business interactions.
- *2': Anonymous/Testing Account.* Designed for high-churn, disposable, or experimental personas where anonymity or frequent rotation is desired.
- *100'+: Organization/Collective Accounts.* These indices are reserved for managing personas specifically associated with organizational entities, such as companies, DAOs, or other collective structures.
***** Client-Side Management Rules: Enforcing Security and Privacy
Client applications interacting with Agora's identity system MUST adhere to a strict set of rules to ensure the security, privacy, and integrity of user accounts.
1. *Account Discovery (Gap Limit):* Clients MUST implement a "Gap Limit" (a heuristic search window, typically 20) for account discovery. During recovery or initial synchronization, the client scans accounts 0' through `N'` (where `N'` is determined by the gap limit and activity) for active personas. If an active account is found, the scan window is intelligently shifted forward.
2. *Context Isolation:* Data associated with different accounts (e.g., contact lists, encryption keys, local indexes) MUST be stored in cryptographically isolated database partitions or encrypted with account-specific salts. This prevents accidental data leakage between contexts.
3. *Cross-Account Privacy:* Clients MUST NOT leak the relationships or activities between personas residing in different accounts unless explicitly authorized by the user (e.g., through a signed cross-account attestation Note).
4. *Independent Authentication:* Clients SHOULD allow users to set distinct local authentication requirements (e.g., PINs, biometric scans) for sensitive accounts (e.g., 1' Professional or 100' Organization accounts), providing an additional layer of security for critical digital identities.
***** Technical Implementation (Pseudocode)
```typescript
// Example: Account derivation from a Master Node (representing the Master Key)
const accountIndex = 0; // Defines the specific account (e.g., Primary)
const accountNode = masterNode.derivePath(`m/44'/1'/${accountIndex}'`);
// Example: Persona derivation within the chosen account
const personaIndex = 0; // Defines the specific persona within the account
const personaNode = accountNode.derivePath(`0/${personaIndex}`);
// Example: Key Generation for the derived Persona
// Ed25519 for secure digital signatures
const signingKey = ed25519.generateKeyPair(personaNode.privateKey);
// X25519 for robust cryptographic encryption
const encryptionKey = x25519.generateKeyPair(personaNode.privateKey);
```
** Personas
*** Personas: Your Active Digital Selves
*** Persona Keys
- Each Persona has its own Ed25519 keypair for signing and an X25519 keypair for encryption.
- Persona keys MUST be derived within the secure hardware (Secure Enclave/Keystore) when possible.
- Private keys MUST NEVER be exposed to application memory in plaintext.
- If a Persona key is compromised, it can be revoked and rotated without affecting the Master Key or other Personas.
*** Persona Governance & Operational Recovery
While the Master Key is an offline seed, Personas are active network agents governed by their own rules, smart contracts, and DID Documents. Operational recovery, succession, and governance occur at this layer and are defined via **Inception Policies** established at the moment the identity is created.
**** Recovery Guardian Dynamics: Natural Persons vs. Collectives
Agora distinguishes between the dynamics of recovery for individual "natural person" Personas and "collective" or organizational Personas (e.g., companies, DAOs) when it comes to social recovery.
***** Natural Person Persona: The "Dictator with Safety Nets"
For a human, the design goal is Ultimate Sovereignty. You are the "Root." Even if you have "Recovery Friends," they should have no power over you unless you are incapacitated.
- **The Logic:** The Persona's primary operational key holds absolute priority weight (e.g., Weight 100). The "Recovery Friends" group has a collective weight of 100, but their actions are restricted by time-locks.
- **Unilateral Action:** A natural person Persona retains the right to change their recovery "friends" (guardians) even if those guardians do not explicitly consent to be "rotated out."
- **Mechanism:** Any rotation signed by the primary key is effective immediately. Rotation signed by the Escrow Group (Guardians) requires a 72-hour `Pending State` (Time-Lock) and can be cancelled by the user at any time. This ensures you can "fire" your recovery team instantly without asking for permission, as your weight alone meets the threshold.
***** Collective Persona: The "Protected Quorum"
For an LLC or NGO, the goal is Mutual Defense and preventing "hostile takeovers" where one founder kicks out others.
- **The Logic (Consensus Required):** All shareholder keys have defined, often equal weights (e.g., 3 shareholders, weight of 33 each).
- **The Rotation Rule (Governance Gate):** Thresholds for different actions are defined at inception. For example, a simple majority (51%) might be sufficient for daily operations, but changing the board or quorum requires a supermajority (e.g., 75% or 3-of-3 unanimity).
- **Veto Power:** The identity may designate a specific "Founder Key" that possesses Veto Power. This key must be among the required editors for *any* rotation event to be valid, making that individual impossible to remove without their own signature.
- **Protection:** This prevents a single member from seizing the company identity. Removing a member requires signatures from the quorum (e.g., 3-of-4), ensuring that "consent" is baked into the math of the threshold.
***** Identity Succession & Minors
Agora handles the lifecycle of identity across generations.
- **Minor Onboarding:** For a minor, a parent or guardian Persona can "Co-sign" the identity inception event.
- **Succession Logic:** This link creates a pre-authorized recovery path where the parent holds a dormant weight that can be activated to rotate keys if the minor loses access, transitioning to full independence at a defined maturation date.
**** Legal Override & The "Break-Glass" Escrow (For Legal Entities)
To handle situations like the death of a sole founder, a lost key, or a binding court order without creating a central back door, Agora implements a "Dormant Escrow" pattern specifically designed for Collective Personas or High-Value single Personas.
- **The Dormant Key:** At inception, the Persona's governance structure includes a "Public Key" belonging to a Neutral Third Party (e.g., a decentralized notary or a legal escrow service). This key is assigned a weight of `0` for daily operations.
- **Multi-Party (M-of-N) Escrow:** To prevent a single corrupt entity from hijacking an identity, Agora utilizes a **Recovery Council**. For instance, a rotation might require 2-of-3 signatures from designated entities (e.g., a Notary, a Law Firm, and a Decentralized Oracle).
- **The Trigger:** The identity’s governing logic includes a rule: "If a certified Legal Attestation (e.g., signed by the local Court's Public Key) is presented, the Escrow Key's weight jumps to the necessary quorum threshold (e.g., 100) for a single Rotation Event."
- **Observer-First Transparency:** Any change to the master key—including a legal override—must be published to the **Key Event Log (KEL)**. This ensures it's impossible for an agent to "quietly" take over an account; every user device and hired "watchdog" service is alerted immediately.
- **The Veto Window (Time-Locking):** Any rotation event initiated by an Escrow Key triggers a mandatory 72-hour `Pending State`. If the primary owner still possesses their key (i.e., the agent is acting maliciously), they can sign a **Veto & Revoke** message. Because the Owner Key has absolute priority, this instantly kills the pending rotation and can strip the escrow agent of future rights. If the owner is incapacitated, they won't sign a veto, and after 72 hours, the change becomes final.
- **Empowerment through Pre-authorization:** This allows the law to intervene technically—not through "hacking," but via a pre-authorized, transparent mechanism agreed upon during the identity's inception.
**** The "Dead Man's Switch" (Protocol Level Recovery)
To prevent assets from being "lost forever" if a user disappears unexpectedly:
- **The Watcher:** A smart contract or a "Guardian Persona" monitors the user's on-chain and network activity.
- **The Trigger:** If the Persona DID has zero "Key Activity" for a defined period (e.g., 12 months), a pre-designated Inheritance Key is authorized to initiate a recovery rotation.
- **The Safety:** The user receives a "Warning Notification" (via DIDComm) every month leading up to the trigger. A single "Heartbeat" signature from their active phone resets the 12-month clock.
***** Against Founder Malice
- *Time-Locked Contracts:* Maturation date is immutable in Foundation Contract; founders cannot delay or prevent it.
- *Social Accountability:* Public attestations of maturation create social pressure against interference.
- *Legal Recourse:* Blockchain evidence of maturation provides evidence in legal disputes.
- *Fork Option:* If founders refuses to release keys, Persona can generate new identity and attest to the connection publicly.
***** Recovery During Stages
****** Before Key Introduction
- Founders fully control recovery (regenerate Persona keys).
- User SHOULD have Shamir shards among trusted guardians.
****** After Key Introduction, Before Maturation
- User holds own root backup; can recover independently.
- Founders can still recover if user loses key.
- *Both paths available:* Dual recovery for safety.
****** After Maturation
- Standard social recovery (Shamir's Secret Sharing with chosen guardians).
- No founder backup; full self-sovereignty.
- User SHOULD have hardware backups before maturation.
*** Wallet Integration (Ownership & Contracts)
Each Persona in Agora is analogous to a legal person, possessing the inherent right and capability to own property, enter into contracts, and claim protected rights (freedom of speech, due process). Therefore, every Persona will have its own associated wallets (e.g., for BTC, Lightning, stablecoins, other digital assets). These wallets are controlled by the Persona's derived keypairs, making cryptographic ownership an integral part of its functional identity. Personas are thus fully enabled to manage digital assets and participate in the Agora economy.
*** Delegated Authoring & AI Personas
**** Owner DID vs. Editor DID: The Mechanism of Agency
Agora distinguishes between the identity that owns the content and the identity that cryptographically signs it. While these are identical in most personal interactions, their separation enables complex organizational and recovery workflows.
- **Owner DID:** The source of authority, reputation, and ownership. This is the Persona "speaking" or "publishing." All social weight and historical context accrue to this DID.
- **Editor DID:** The cryptographic actor performing the signature, recorded within the Note's `proof` object. This is the entity "signing" the data. The network verifies that the Editor holds a valid Delegation Certificate or is an authorized recovery key for the Owner. If omitted from the `proof`, it defaults to the Owner DID (self-signed).
***** Key Use Cases for Separation
1. **Organizational Delegation (The Assistant Model):** An NGO (Owner DID) issues a Delegation Certificate to an employee, Alice (Editor DID). Alice publishes updates using her own keys, but the network attributes them to the NGO.
2. **AI Agent Accountability:** A Human (Owner DID) authorizes their personal AI Bot (Editor DID) to act on their behalf. Users can verify that a message is from the human while knowing it was technically generated and signed by their AI agent.
3. **Legal Override & Recovery:** When a user loses their keys, a pre-authorized Recovery Council (Editor DID) signs a Key Rotation Event for the Incapacitated User (Owner DID), restoring their digital presence.
4. **Guardianship:** A Parent (Editor DID) manages and signs events for a Minor (Owner DID) until a pre-defined maturation date.
***** Technical Benefits
- **Accountability:** Provides a transparent audit trail of the physical signers acting on behalf of an identity.
- **Granular Revocation:** An Owner can revoke an Editor's access instantly without needing to change their own identity or rotate master keys.
- **Reputation Portability:** Content history and social relationships stay with the Owner DID, regardless of which specific human or bot was authorized to sign at the time.
**** Cryptographic Delegated Signatures
To allow multiple individuals (e.g., employees) or autonomous agents to act on behalf of a single Persona (e.g., an LLC or a brand account) without sharing the Master Key, Agora employs Delegated Signatures.
- **The Delegation Certificate:** The "Owner" Persona signs a special `Delegation Certificate` granting specific capabilities to a "Delegate" DID for a defined period.
- **Example Constraint:** "Delegate X can publish `is_feed: true` Notes on behalf of Owner Y, but cannot sign `contract` Notes."
- **The Signature:** When the Delegate acts, they sign the Note with their *own* private key and append the Delegation Certificate. The network validates the certificate against the Owner's public key.
- **Instant Revocation:** The Owner can instantly revoke the delegation by publishing a revocation event, cutting off the Delegate without needing to change passwords or rotate the Owner's keys.
**** AI Agent Personas (AAP)
Agora treats Artificial Intelligence not as a backend feature, but as a first-class participant.
- **Agent DIDs:** An AI Agent is assigned its own derived Persona DID, completely separated from the human's primary identity.
- **Capabilities-Based Security:** Using the Delegation mechanism above, the human owner grants the AI Agent restricted capabilities (e.g., "Authorized to spend up to 5000 sats/month" or "Authorized to draft responses but not publish them").
- **Verifiable Origins:** Because the AI signs with its own DID, all network participants can instantly and cryptographically verify whether a piece of content was authored by a human or an AI.
*** Naming & Registry
**** Naming Tiers
***** The Local Alias (Tier 1)
- *Client-Side Only:* Every client allows users to assign private nicknames to DIDs in their contact list.
- *Privacy:* 100%. No one else knows what you call them.
- *Scope:* Private to the user.
***** The Global Registry (Tier 2)
- *Decentralized Ledger:* A name-to-DID mapping stored on a decentralized ledger (e.g., a simple L2 or a high-reputation PDS/Relay coalition).
- *Zooko's Triangle:* Agora attempts to achieve names that are *Human-Readable*, *Secure*, and *Decentralized*.
- *First-Come, First-Served:* Names are registered by the first persona to claim them, with small micro-fees (1000+ satoshis) to prevent squatting.
***** The Subdomain Model (Tier 3: The "Default" Handle)
- *Domain-Based Names:* If a user doesn't own a custom domain, their PDS provider (e.g., a community hub) grants them a subdomain.
- *Format:* `username.provider.org` (e.g., `alice.aletheia.social`).
- *Handle Resolution Protocol:* The system MUST support multiple methods for resolving a human-readable handle to a DID:
- **Method A (DNS TXT):** The client queries the DNS for a TXT record at `_atproto.alice.aletheia.social`.
- **Method B (HTTPS Well-Known):** The client fetches the DID from `https://alice.aletheia.social/.well-known/atproto-did`.
- *Cross-Namespace Resolution:* The network's Search Indexers MUST implement a "Resolver Bridge" to handle other ecosystems. For example, if a search matches a `.eth` name, the indexer queries the ENS Smart Contract on Ethereum to find the associated DID.
- *Validation:* To prevent "spoofing," the DID document returned by the PDS MUST contain a back-link to the handle.
- *Sovereignty:* If you move your PDS to your own custom domain, you take your name with you.
**** Multi-Persona Naming Convention
Because users manage multiple Personas (Legal, Professional, Anonymous) derived from a single Master Seed, clients SHOULD implement a Persona-Suffix convention to distinguish them clearly within the Subdomain Model:
- **Primary/Legal:** `name.provider.org` (e.g., `john.aletheia.social`)
- **Professional:** `name-pro.provider.org` (e.g., `john-pro.aletheia.social`)
- **Anonymous/Alt:** `alias.provider.org` (e.g., `night-owl.aletheia.social`)
**** Web3 Naming Services (e.g., ENS)
For users who want a username entirely untethered from a specific PDS provider's domain, Agora supports Decentralized Naming Services like Ethereum Name Service (ENS).
- *How it works:* The user registers a base name (e.g., `yourname.eth`). They can then generate unlimited subnames for their various Personas for free (e.g., `work.yourname.eth`, `social.yourname.eth`).
- *Portability:* If the user migrates their data to a new PDS, the `.eth` name stays with them. They simply update the "Content Hash" record on the blockchain to point to the new PDS location, ensuring unbreakable ownership of the handle.
*** Naming Registry Implementation
**** Implementation Options
***** Option 1: Simple L2 on Bitcoin/Lightning
- *Architecture:* Dedicated Lightning channel for name registration (similar to Lightning Addresses).
- *Process:* User sends 1000 sats + desired name to a specific "Name Registrar" Persona. Registrar publishes signed attestation (name -> DID) to a public PDS.
- *Verification:* Clients verify attestation against Registrar's DID.
- *Pros:* Low cost, high speed, leverages existing infrastructure.
- *Cons:* Registrar still a single point of failure for initial registration.
***** Option 2: Federated Registrar Network
- *Architecture:* M-of-N multi-sig collective of Name Registrar Personas.
- *Process:* User pays fee; M of N registrars sign attestation.
- *Pros:* Decentralized, more robust against single point of failure.
- *Cons:* Higher latency, more complex setup.
***** Option 3: Sidechain/Drivechain
- *Architecture:* Dedicated sidechain for name registrations.
- *Process:* Transaction on sidechain maps name to DID.
- *Pros:* High throughput, specialized functionality.
- *Cons:* New trust assumptions, requires sidechain security.
***** Decision: Option 1 (Simple L2 Registrar) for V1.0
- Prioritizes speed and simplicity for initial rollout.
- Recognizes that full decentralization of the Global Registry is a complex problem.
- Clients can choose their registrar.
**** Registrar Persona Requirements
- *DID:* Must be a well-known, high-reputation Persona.
- *API:* Standard API for name registration/lookup.
- *Fees:* Transparent and auditable fee structure.
- *Availability:* High uptime and low latency.
- *Audit:* Publicly auditable log of all name registrations.
*** Identity Linking
*** Verification Objects
- *Verification Objects:* A persona can publish a signed *Verification Object* linking their Agora DID to other identities (e.g., a PGP key, a personal domain, or even a centralized social profile).
- *Proof-of-Domain:* Proving ownership of a domain (via DNS record) is the gold standard for high-trust identity verification in Agora.
*** Zero-Knowledge Proofs (ZKP) & Selective Disclosure
The system allows a user to "Attest" that two Personas belong to the same human (or hold the same credentials) *without revealing the Master Seed or creating a public cryptographic link*.
- **The Problem:** Your "Anonymous Developer" Persona wants to prove it has a "Verified Citizen" badge issued to your "Legal Name" Persona.
- **The ZKP Solution:** Using a Zero-Knowledge Proof, the user can cryptographically prove they hold the private key for the "Legal Name" DID (which holds the badge) and assert a statement on behalf of the "Anonymous" DID.
- **Privacy Preservation:** The resulting proof verifies the credential is valid but explicitly hides *which* specific Legal Name DID generated the proof.
**** Attribute-Based Predicate Proofs
Agora extends ZKP capabilities beyond cross-persona linking to support **Selective Disclosure** and **Predicate Proofs** using Verifiable Credentials (VCs) with advanced cryptographic schemas (e.g., BBS+ signatures or AnonCreds). This allows Personas to prove attributes about their physical or financial reality without leaking metadata or underlying identifiers.
- **Age/Date Verification:** A Persona can cryptographically prove a predicate like `Age > 18` to access age-restricted content or contracts without revealing their actual date of birth.
- **Financial Ability:** A Persona can prove `Wallet Balance > 10,000 sats` or `Monthly Income > X` to serve as collateral or qualify for a service contract without revealing their exact balance or transaction history to the counterparty.
- **Citizenship & Residence:** A Persona can prove membership in a specific geographic jurisdiction (e.g., "Resident of New York") for local governance voting or tax-compliant commerce without disclosing their legal name or specific home address.
- **Asset Ownership:** A Persona can prove ownership of a specific Physical Asset Link (PAL) or digital token to gain entry to a gated community or guild without linking that high-value asset to their everyday public identity.
**** Verification Object Schema
#+begin_src typescript
interface VerificationObject {
// Identity linking DID
did: string;
// What external identity is being linked
identityType: 'domain' | 'pgp' | 'social_twitter' | 'social_github' | 'other';
identityIdentifier: string; // e.g., 'example.com', '0x1234...ABCD', '@amr'
// The cryptographic proof of control over the external identity
// - For domains: A signed string expected to be found in DNS TXT record
// - For PGP: A signature of the DID using the PGP key
// - For social: A URL to a public post containing the DID and signature
proof: {
proofType: 'dns_txt' | 'pgp_signature';
proofData: string;
};
// Agora persona signature (proving the DID owner agrees to the link)
timestamp: number;
signature: string; // Ed25519 signature over the object
}
#+end_src
**** Problem Statement**
When a Persona's derived keys are compromised, lost, or need deactivation, users need a cryptographically verifiable mechanism to:
1. Invalidate the affected Persona
2. Preserve the Master Key and other Personas
3. Optionally migrate content history
4. Maintain network integrity
*** Identity Verifiability & Forward Security
Personas are the functional, active identities through which you engage with the Agora network. Each Persona is uniquely and cryptographically derived from your Master Key, acting as your distinct digital self for specific contexts. They are the sovereign participants in the network, fully empowered to own property, enter into binding contracts, publish content, and claim protected rights such as due process and freedom of expression. This section details the cryptographic derivation, secure management, revocation mechanisms, and identification systems that enable your Personas to operate seamlessly and securely within the broader Agora ecosystem.
*** Key Event Log (KEL): The Observer-First Transparency Log
Every Persona in Agora maintains a Key Event Log (KEL). This is a public, append-only ledger of all identity-related events, including:
- **Key Events:** Inception, rotation, and revocation.
- **Follow Events:** Every time you follow another DID, a signed "Follow Event" is added to the log.
- **Transparency:** It is impossible to "quietly" take over an account or manipulate your public history. Any change to the keys or following relationships must be published to the log. Watchdog services can monitor this log and alert the user immediately if an unauthorized event is initiated.
**** Social Graph Reconstruction
The "Social Graph" (the list of DIDs you follow and who follows you) is mathematically reconstructible from the KEL.
- **The Proof:** Follow Events (Notes) are cryptographically signed by the Persona's authorized keys (or the Master Key).
- **The Rebuild:** When initializing a new PDS, the software scans the network and subscribed Relays for any signed Follow Events belonging to the user's DID. It automatically reconstructs the user's entire social graph (e.g., a list of 500 friends) without the user needing to remember a single username or manual backup.
*** Pre-rotation: Quantum-Resistant Continuity
Agora utilizes the principle of *Pre-rotation* to ensure forward security as an ultimate fail-safe.
- **The Hash Commitment:** When a user creates their current active key, they simultaneously publish a cryptographic hash of their *next* (unborn) public key.
- **The Protection:** Even if an attacker breaks the user's current private key (e.g., via a future quantum computer, theft, or even a malicious legal override attempt), they cannot forge a rotation event because they do not know the private key corresponding to the pre-committed hash. Rotation only becomes valid when the user reveals the new key that matches the previous hash, providing true "forward security."
**** Technical Requirements
- *Interface:* Clients MUST support communication via *WebHID* (browser-based) or *USB/HID* (native) using the standard *APDU* (Application Protocol Data Unit) format.
- *Key Derivation:* The HSM MUST perform BIP-32/44 derivation internally.
- *Signing Protocol:*
1. Client sends unsigned Content Object hash to HSM.
2. HSM displays metadata (CID, Persona name) to user for confirmation.
3. Upon user approval, HSM signs hash using the specified Persona key.
4. HSM returns the Ed25519 signature to the client.
- *Master Key Security:* The 24-word mnemonic MUST be generated and stored exclusively within the HSM.
** Hardware Keys
*** The "Vault" Device Guide (For the Engineer)
The "Vault" is a dedicated application for an offline/hardware device that manages the Master Seed.
**** Functional Requirements for the Vault Tool:
- **Seed Generation:** Must use a high-entropy hardware RNG to generate the BIP-39 mnemonic.
- **Persona Derivation:** Must implement a hardened derivation logic where the user can "Export Persona #N."
- **Key Rotation Editor:** This is the most important feature. If a phone is lost, the Vault device creates a DID Update Transaction. This is a cryptographically signed message that says: "I am the Master Seed; I hereby revoke Persona Key A and authorize Persona Key B."
- **Recovery Seed Export:** The Vault should allow exporting a "Recovery Key"—a special key used specifically for the "Re-Wrapping" process on the PDS during content re-keying.
*** Hardware Integration: Sphinx for Your Keys
**** Technical Requirements
**** BIP-39 / BIP-44 Compatibility
Agora-compatible hardware wallets MUST support the `m/44'/1'/` path. If the device does not support the custom `1'` coin type, clients MAY fallback to a generic data-signing path, but this is NOT recommended for production use.

823
projects/agora/agora-requirements-03-infrastructure.org Normal file
View File

@@ -0,0 +1,823 @@
#+TITLE: Agora Requirements - 03: Infrastructure
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:28]
#+DATE: 2026-03-14
#+ID: agora-requirements-03-infrastructure
#+STARTUP: content
* The Sovereign Infrastructure: Your Digital Foundation
Agora's infrastructure is meticulously architected to deliver on the promise of true digital sovereignty. Unlike traditional platforms that centralize control, Agora distributes power to the edges—directly into the hands of users. This section details the foundational infrastructure that makes self-sovereign identity, data ownership, decentralized communication, and global discovery not just possible, but practical and scalable.
** Personal Data Store (PDS): Your Digital Fortress
The Personal Data Store (PDS) is the cornerstone of Agora's sovereignty model—your personal, encrypted vault where all your Notes, identity data, and digital assets reside. Unlike cloud services that claim ownership of your data, your PDS is unequivocally yours. You control it. You host it. You decide who accesses it. It is the physical manifestation of your digital self-sovereignty.
*** Requirements
- The system MUST use a hybrid network architecture with Personal Data Stores (PDS) and Relays.
- Every user MUST control their own PDS (hosted or self-run).
- The PDS MUST serve as the master archive for all the user's content (encrypted) and identity data.
- The PDS MUST act as a gatekeeper, issuing decryption keys upon valid payment or credential verification.
- Relays MUST NOT store data long-term (unless paid to).
- Relays MUST route ciphertext based on CID and persona subscriptions.
- The system MUST incentivize Relays to route high-traffic content or provide specific delivery guarantees.
- The system MUST allow users to publish their CIDs to multiple relays to ensure availability and bypass censorship.
- The system MUST use Double Ratchet for 1-on-1 private messaging.
- The system SHOULD use MLS (Messaging Layer Security) for group chats.
- The system MUST use symmetric encryption for paywalled content (individual keys per object).
- The system MUST support social recovery using Shamir's Secret Sharing, allowing users to split their Master Key into shards distributed to trusted guardians.
*** Technical Logic
**** Personal Data Store (PDS)
- *Home Base:* Every user controls their own PDS (hosted or self-run).
- *Master Archive:* All the user's content (encrypted) and identity data live here.
- *Key-Server Separation:* The PDS MUST include a distinct Key-Management Module that handles the automated sale and distribution of decryption keys/LSATs. This MUST be logically separate from the Data-server hosting the encrypted blobs, ensuring that the entity holding the keys does not necessarily host the content payload.
- *Access Control:* PDS acts as a gatekeeper, issuing decryption keys upon valid payment or credential verification.
**** Encryption Model (E2EE)
- *Double Ratchet:* Used for 1-on-1 private messaging.
- *MLS (Messaging Layer Security):* Proposed for group chats.
- *Symmetric Encryption:* Used for paywalled content (individual keys per object).
- *Envelope Encryption (Data-at-Rest):* To protect against stolen devices, PDS storage utilizes Envelope Encryption. Large files are encrypted with a random Data Encryption Key (DEK), which is itself encrypted (wrapped) with the Persona Public Key.
- *Automated Re-Keying Service:* The PDS MUST include a background worker that triggers upon a `KEY_ROTATION_EVENT`. The worker iterates through KeyHeader objects and uses a Proxy Re-Encryption (PRE) scheme to securely re-wrap the DEKs with the new key, without ever exposing the raw Master Seed to the PDS.
*** Developer Implementation
To send a private message:
1. Encrypt message for the recipient's Persona Encryption Key (X25519).
2. Upload ciphertext to the user's PDS.
3. Notify the recipient's subscribed Relays of the new CID.
4. Recipient's client fetches the CID from the Relay/PDS and decrypts locally.
*** PDS Migration: Seamless Sovereignty Transfer
PDS Migration represents a fundamental capability of Agora's architecture—the seamless, user-initiated transfer of one's entire digital corpus from one Personal Data Store to another. Unlike traditional platforms where data migration is often complex, permission-based, or impossible, Agora treats PDS Migration as a first-class operation. This is not an edge case, but a core feature that ensures users retain ultimate sovereignty over their data throughout its lifecycle. Whether changing hosting providers, upgrading hardware, or responding to security incidents, PDS Migration ensures users are never trapped by infrastructure choices.
**** Concept
PDS Migration is the comprehensive process of transferring a user's entire encrypted content repository and identity data from one PDS to another while rigorously maintaining Content Identifier (CID) integrity, subscription continuity, and access control mechanisms. This process ensures that all cryptographic relationships between Notes remain intact, and that no data is lost or corrupted during the transfer.
Key principles of PDS Migration:
- **User Sovereignty Absolute:** Users retain complete autonomy to migrate their data without requiring permission, intervention, or cooperation from any third party. This is a fundamental right within the Agora ecosystem.
- **Zero-Downtime Operation:** Migration SHOULD occur without interrupting the user's ongoing presence or activities on the network. This ensures continuous availability of services and interactions.
- **Rollback Safety:** Users MUST have the capability to revert to the original PDS if the new PDS fails to perform adequately or if any issues arise during the migration process. This provides a safety net for critical data transfers.
- **Atomic Cutover:** There is a clearly defined, atomic moment when the new PDS becomes the authoritative source of truth, and the old PDS transitions to a backup role, ensuring data consistency.
Migration scenarios include a comprehensive range of use cases:
- Self-hosted PDS → Cloud-hosted PDS (moving from personal infrastructure to professional hosting)
- Cloud provider A → Cloud provider B (e.g., AWS → GCP, avoiding vendor lock-in)
- Old hardware → New hardware (self-hosted upgrade for improved performance or capacity)
- Compromised PDS → Clean PDS (security incident response and remediation)
- Cost optimization (migrating to more economical hosting solutions)
- Performance enhancement (migrating to geographically closer or faster infrastructure)
**** Requirements
- The system MUST support full PDS migration without service interruption.
- The system MUST preserve all Content Object CIDs during migration (content-addressed storage guarantees this).
- The migration process MUST update the Persona's DID Document to point to the new PDS service endpoint.
- The system MUST notify all subscribed Relays about the PDS endpoint change.
- The system MUST support parallel operation (old and new PDS active simultaneously) during migration testing.
- The system MUST provide migration progress tracking and verification.
- The system MUST support incremental pre-migration sync to minimize final cutover time.
- The system MUST handle in-flight messages during cutover (queue and forward).
- The system MUST provide a rollback mechanism if migration fails.
**** Migration Phases
***** Phase 0: Preparation
- *Prerequisites:* Ensure new PDS meets minimum specs (storage, bandwidth, availability).
- *Provisioning:* Configure new PDS with Persona's DID but initially in "standby" mode.
- *Authorization:* New PDS MUST prove ownership via Persona signature challenge.
***** Phase 1: Initial Sync
- *Full Replication:* Transfer all Content Objects from old PDS to new PDS.
- *CID Verification:* Block-by-block verification that all content transferred correctly.
- *Metadata Sync:* Sync Persona profiles, access control lists, and configuration.
- *Duration:* May take hours/days depending on data volume.
- *Old PDS Remains Authoritative:* Writes still go to old PDS during this phase.
***** Phase 2: Incremental Catch-up
- *Delta Sync:* Catch up changes made since Phase 1 started.
- *Repeat:* Continue incremental syncs until delta is small (e.g., < 1 minute of data).
- *Read Testing:* Client optionally reads from new PDS to verify accessibility.
***** Phase 3: Cutover
- *Freeze Writes:* Brief write lock on old PDS (seconds to minutes).
- *Final Delta:* Transfer last remaining changes.
- *DID Update:* Publish new DID Document with new PDS service endpoint.
- *Relay Broadcast:* Announce endpoint change to all subscribed Relays.
- *New PDS Becomes Authoritative:* Write traffic now routes to new PDS.
***** Phase 4: Stabilization
- *Monitor:* Observe new PDS for errors, dropped messages, or performance issues.
- *Verification:* Confirm all personas can reach new PDS, all content accessible.
- *Grace Period:* 24-48 hour buffer where old PDS remains available as hot standby.
- *Rollback Window:* If issues detected, can revert to old PDS via DID re-update.
***** Phase 5: Decommissioning
- *Archive:* Old PDS data backed up (user's discretion).
- *Tombstone:* Old PDS endpoint publishes redirect or shutdown notice.
- *Cleanup:* Remove old PDS from user's infrastructure (cancel cloud instance, retire hardware).
- *Final Verification:* Confirm no traffic routing to old PDS.
**** Technical Considerations
***** Concurrent Access During Migration
- *Read Replication:* Old PDS can serve reads while new PDS receives writes (after cutover) to reduce downtime to near-zero.
- *Message Queueing:* Relays queue messages during the brief cutover window; messages forwarded once new PDS confirms readiness.
- *Conflict Avoidance:* Strict sequencing ensures no split-brain scenarios (only one PDS accepts writes at any time).
***** Verification Protocol
- *CID Audit:* Iterate through all CIDs in Persona's content graph; verify retrievable from new PDS.
- *Signature Verification:* Spot-check Content Object signatures against Persona's public keys.
- *Access Control Verification:* Test decryption of sample encrypted content using Persona's keys.
- *Subscription Testing:* Verify Relays successfully forward new CIDs from new PDS.
***** Rollback Procedures
- *Trigger:* Migration fails verification or new PDS experiences critical failure.
- *DID Reversion:* Re-publish previous DID Document with old PDS endpoint.
- *Relay Re-announce:* Broadcast reversion to all Relays.
- *Data Reconciliation:* If any writes occurred on new PDS before failure, sync them back to old PDS before re-activating.
- *Graceful Degradation:* If both PDS fail, Persona can restore from backup and re-establish with same or new infrastructure.
**** Developer Implementation Example
#+begin_src typescript
// Initiate PDS migration sequence
interface PDSMigrationPlan {
sourcePDS: string; // Old PDS endpoint
targetPDS: string; // New PDS endpoint
personaDID: string;
phases: MigrationPhase[];
estimatedDuration: number; // Estimated seconds for full migration
rollbackDeadline: number; // Timestamp until rollback is possible
}
interface MigrationPhase {
name: 'preparation' | 'full-sync' | 'incremental' | 'cutover' | 'verification' | 'completed';
status: 'pending' | 'in-progress' | 'completed' | 'failed';
startedAt?: number;
completedAt?: number;
progressPercent: number;
}
// Phase 1: Full replication
async function replicateContentObjects(
sourcePDS: string,
targetPDS: string,
personaDID: string,
authToken: string
): Promise<ReplicationResult> {
const sourceClient = new PDSClient(sourcePDS, personaDID);
const targetClient = new PDSClient(targetPDS, personaDID);
// Fetch all CIDs from source
const allCIDs = await sourceClient.listAllCIDs();
// Batch transfer with verification
const results = await batchTransfer(allCIDs, {
source: sourceClient,
target: targetClient,
verifyCID: true, // Verify hash after transfer
batchSize: 100,
concurrency: 5
});
return {
transferred: results.successful.length,
failed: results.failed,
duration: results.elapsedTime
};
}
// Phase 3: Cutover - Update DID Document
async function executeCutover(
persona: Persona,
newPDSEndpoint: string,
oldPDSEndpoint: string
): Promise<CutoverResult> {
// 1. Freeze writes on old PDS
await freezeOldPDS(oldPDSEndpoint, persona.did);
// 2. Final incremental sync
await finalIncrementalSync(oldPDSEndpoint, newPDSEndpoint);
// 3. Update DID Document with new service endpoint
const updatedDoc = await updateDIDDocument(persona.did, {
service: [{
type: 'PDS',
serviceEndpoint: newPDSEndpoint,
// ... other service properties
}]
});
// 4. Sign and publish new DID Document
const signedDoc = await persona.sign(updatedDoc);
await didResolver.publish(signedDoc);
// 5. Notify all subscribed Relays
await notifyRelays(persona.did, {
type: 'PDS_ENDPOINT_CHANGE',
oldEndpoint: oldPDSEndpoint,
newEndpoint: newPDSEndpoint,
signature: signedDoc.proof
});
return { status: 'success', newDocumentCID: signedDoc.cid };
}
// Verification: Confirm all content accessible
async function verifyMigration(
newPDS: string,
personaDID: string,
expectedCIDCount: number
): Promise<VerificationResult> {
const client = new PDSClient(newPDS, personaDID);
// Verify CID count matches
const reachableCIDs = await client.listAllCIDs();
if (reachableCIDs.length !== expectedCIDCount) {
throw new Error(`CID mismatch: expected ${expectedCIDCount}, found ${reachableCIDs.length}`);
}
// Spot-check: verify random sample of CIDs
const sample = selectRandomSample(reachableCIDs, 100);
const verificationResults = await Promise.all(
sample.map(cid => verifyContentObject(client, cid))
);
const failures = verificationResults.filter(r => !r.valid);
if (failures.length > 0) {
throw new Error(`Verification failed for ${failures.length} objects`);
}
return { status: 'verified', sampleSize: sample.length, failures: 0 };
}
#+end_src
**** User Experience Considerations
- *Progress Dashboard:* Real-time view of migration progress with estimated time remaining.
- *Notification:* Alerts to user's clients about upcoming maintenance window (for cutover).
- *Zero-Click Resume:* If migration interrupted, resumes from last checkpoint automatically.
- *Cost Transparency:* Estimate bandwidth/storage costs before starting (especially for cloud-to-cloud).
- *Support Contact:* During migration, provide direct line to new PDS operator for issues.
**** Security During Migration
- *Authenticated Transfer:* All replication traffic encrypted and authenticated (mTLS or Noise).
- *No Plaintext Exposure:* Ciphertext transferred as-is; no decryption during migration.
- *Audit Trail:* All migration events logged as tamper-evident Content Objects.
- *Guardian Notification:* Optional: notify social recovery guardians of major infrastructure change.
- *Rate Limiting:* Prevent migration from overwhelming either PDS; throttle if needed.
*** PDS-to-PDS Synchronization: Redundancy and Resilience
In a truly sovereign digital ecosystem, users should never be constrained to a single point of failure. Agora's PDS-to-PDS Synchronization protocol empowers users to run multiple Personal Data Stores simultaneously—for redundancy, load balancing, or geographic distribution. This protocol enables bidirectional synchronization of encrypted Content Objects between a user's PDS nodes, maintaining CID integrity, conflict resolution, and data consistency across the distributed infrastructure. It ensures that your digital presence remains resilient, available, and performant, regardless of individual infrastructure limitations.
**** Concept
The PDS-to-PDS Synchronization Protocol allows users to maintain multiple, synchronized copies of their encrypted data across different PDS instances. This capability transforms the PDS from a single point of failure into a distributed, fault-tolerant system that can withstand outages, attacks, or infrastructure changes. By synchronizing data across multiple nodes, users achieve:
- **High Availability:** If one PDS becomes unreachable, others can seamlessly serve data, ensuring continuous access to your digital identity and content.
- **Geographic Distribution:** PDS nodes can be strategically located in different regions to minimize latency and comply with local data sovereignty requirements.
- **Load Distribution:** High-traffic Personas can distribute read operations across multiple synchronized PDS nodes, improving performance and responsiveness.
- **Disaster Recovery:** Synchronized PDS nodes provide inherent backup capabilities, ensuring data preservation even in catastrophic failures.
**** Sync Protocol Architecture
**** Merkle DAG Synchronization
- Each PDS maintains a Merkle DAG of all stored Content Objects.
- Root hash represents complete state of the PDS.
- Sync compares Merkle roots to efficiently identify differences.
**** Sync Modes
**** Full Sync
- Complete synchronization of all Content Objects.
- Used for initial setup or recovery from desync.
- Sends all CIDs not present in the other PDS.
**** Incremental Sync
- Only synchronizes changes since last sync.
- Maintains sync cursor (last synced CID timestamp).
- More efficient for ongoing synchronization.
**** Selective Sync
- Synchronizes only specific content types or time ranges.
- User-defined filters (e.g., "only public posts", "last 30 days").
- Useful for bandwidth-constrained scenarios.
**** Sync Process
**** Phase 1: Handshake
- PDS nodes authenticate using Persona's DID.
- Exchange authentication proofs (signatures).
- Verify both nodes are authorized for this Persona's data.
- Exchange capabilities (sync modes supported, bandwidth limits).
**** Phase 2: Discovery
- PDS A computes Merkle root of current Content Object set.
- PDS B does the same.
- Compare roots: if equal, sync complete; if different, continue.
- Exchange Merkle proofs to identify divergent branches.
**** Phase 3: Delta Calculation
- Based on Merkle proof comparison, identify missing CIDs on each side.
- Calculate delta: set of CIDs A has that B doesn't, and vice versa.
- Partition delta into batches for transfer.
**** Phase 4: Transfer
- Request missing CID payloads from peer PDS.
- Receive Content Objects with full metadata.
- Verify CID integrity (content-addressed verification).
- Store in local PDS.
**** Phase 5: Conflict Resolution
- If same CID exists with different content (hash mismatch):
- Both versions preserved (content-addressed storage).
- Conflict marked for manual resolution.
- User interface shows both versions, user selects authoritative.
- If same CID exists with same content: no conflict.
**** Phase 6: Confirmation
- Both PDS nodes recompute Merkle roots.
- Verify roots match post-sync.
- Update sync cursor for incremental future syncs.
**** Sync Conflicts
**** Conflict Types
**** Write-Write Conflict
- Same CID modified differently on two PDS nodes simultaneously.
- Resolution: Keep both, mark secondary as "alternate version", user resolves.
**** Tombstone Conflict
- CID deleted on PDS A, modified on PDS B.
- Resolution: Configurable policy ("last write wins" vs. "preserve all").
**** Schema Conflict
- Content Object valid on PDS A but rejected by PDS B (schema mismatch).
- Resolution: Log error, quarantine object, notify user.
**** Periodic Synchronization
- *Frequency:* User-configurable (real-time, hourly, daily).
- *Real-time Sync:* WebSocket connection for immediate propagation.
- *Scheduled Sync:* Cron-like jobs for periodic full or incremental syncs.
- *On-Demand Sync:* Manual trigger by user or administrator.
**** Security Considerations
- *Authentication:* Both PDS nodes MUST authenticate as authorized for Persona's data.
- *Encryption:* Sync traffic SHOULD be encrypted (TLS 1.3 or Noise Protocol).
- *Authorization:* PDS nodes MAY implement additional access controls (IP allowlists).
- *Audit:* All sync events logged as Content Objects for audit trail.
**** Performance Optimization
- *Delta Encoding:* Only transfer missing CIDs and metadata.
- *Compression:* Payload compression for large Content Objects.
- *Parallel Transfer:* Multiple concurrent transfers for large datasets.
- *Resume:* Partial transfers resume from interruption point.
**** Implementation (TypeScript)
#+begin_src typescript
interface PDSSyncSession {
sessionId: string;
participantPDS: string[]; // PDS DIDs
personaDID: string;
mode: 'full' | 'incremental' | 'selective';
status: 'handshake' | 'discovery' | 'transfer' | 'complete' | 'error';
startedAt: number;
completedAt?: number;
}
interface MerkleNode {
cid: string;
children: MerkleNode[];
hash: string; // Blake3 hash of concatenated child hashes
}
interface SyncDelta {
fromPDS: string;
toPDS: string;
missingCIDs: string[];
conflictCIDs: string[];
estimatedSize: number; // Bytes
}
interface SyncConfig {
mode: 'full' | 'incremental' | 'selective';
filter?: {
contentTypes?: string[];
afterTimestamp?: number;
beforeTimestamp?: number;
publicOnly?: boolean;
};
frequency?: 'realtime' | number; // number = seconds between syncs
priority?: 'low' | 'normal' | 'high';
}
#+end_src
*** Distributed Mirroring & Social Resilience
**** Following: Default "Feed Gossip" & The Phoenix Effect
Agora ensures baseline content resilience by leveraging a gossip-based mirroring architecture triggered by every "Follow" event.
- **Following = Essential Replicating:** When a user "follows" another Persona, their device or PDS automatically joins the gossip swarm for that target's most critical low-bandwidth data.
- **Feed Gossip Scope:** To balance network resilience with device resources, default gossip is restricted to the **Identity Log (KEL)** and a rolling window of **recent text-based Notes** (e.g., the last 1,000 posts).
- **The Phoenix Effect:** This level of mirroring ensures the "Phoenix Effect" remains viable. If a user's PDS is destroyed, they can "shout" to their followers: "I am the owner of DID:123. Please send me everything you have signed by my key." The essential history and social graph flow back from the community.
- **Censorship Resistance:** By making essential gossip a default behavior, the social graph and latest news stay alive through the "cracks" of the internet automatically.
**** Supporting: Opt-in "Supporter-Mirroring" & Decentralized CDN
For high-bandwidth content and deep archives, Agora transitions from simple gossip to an explicit infrastructure donation model.
- **Persistent Mirroring:** When a user clicks "Support," they opt-in to a deeper technical commitment. The supporter's PDS archives the **entire historical feed** of the creator, not just the recent window.
- **High-Bandwidth "Pinning":** Supporters provide the backbone for the **"Follower-as-CDN"** model. A supporter can allocate specific storage (e.g., "Pin 5GB of latest video") to ensure large payloads (audio, video, high-res images) remain highly available.
- **WebRTC Peering & Seeding:** Supporters act as active "Seeds" in a BitTorrent-style swarm. When a new viewer watches a video, the app prioritizes streaming via WebRTC from online supporters rather than the creator's PDS, ensuring viral content has $0 infrastructure cost for the creator.
**** "In-Kind" vs. "Monetary" Support
This tiered model transforms the relationship between organizations and their communities:
- **Scalable In-Kind Support:** A "Poor but Loyal" follower contributes at the Gossip tier (bandwidth for text), while a "Dedicated Patron" contributes at the Mirroring tier (storage for video).
- **Resilience against De-platforming:** Even if a government blocks a creator's main server, the "Swarm" of followers and supporters continues to host and circulate the content through the P2P network.
**** Encrypted Peer-Backups (The "Friend-Box")
While the social swarm recovers public history, users ensure the recovery of private data (drafts, DMs) via formal peer-to-peer backup agreements.
- **The "Friend-Box" Logic:** Users can establish reciprocal "Friend-Box" agreements where they swap encrypted storage space (e.g., swapping 10GB of space with three trusted friends).
- **Mechanism:** The user's PDS automatically generates and sends an encrypted, compressed "State Snapshot" (a Merkle DAG of the entire PDS state) to these friends' servers periodically (e.g., nightly).
- **Privacy Guarantee:** Because the backup is encrypted with the user's keys, the friends cannot read the private drafts or DMs; they only host the raw ciphertext blobs.
- **Restoration:** In the event of a catastrophic local failure (e.g., fire, server loss), the user can download their latest snapshot from a friend and instantly restore their entire digital life to a new PDS node using their recovered Identity Keys.
** Relay Network: The Circulatory System of Agora
The Relay Network serves as Agora's intelligent, adaptive message routing layer—ephemeral, user-chosen pathways that efficiently route encrypted Notes via a pub/sub model. Unlike centralized servers that store and monitor your data, Relays are transient routers that respect your privacy, delivering your messages without ever holding them long-term. They are the circulatory system of the Agora network, ensuring vital communication flows freely and securely.
*** Requirements
- Relays MUST route ciphertext based on CID and persona subscriptions.
- Relays MUST NOT store data long-term (unless paid to).
- The system MUST incentivize Relays to route high-traffic content or provide specific delivery guarantees.
- The system MUST allow users to publish their CIDs to multiple relays to ensure availability and bypass censorship.
- Relays MUST support subscription filters for content discovery.
*** Technical Logic
*** Relay Routing & Prioritization: Pay-to-Prioritize (P2P)
To ensure high-performance and sustainability without central control, Agora Relays utilize a **Pay-to-Prioritize (P2P)** routing strategy. Crucially, Relays are **Logic-Blind**. They do not inspect the Note's payload or contract terms (which may be encrypted). Instead, they prioritize traffic based on explicit, unencrypted metadata.
**** Explicit Priority Fee (The "Fast-Lane")
If a Note requires instant routing (e.g., a time-sensitive financial transaction or live chat), the sender can attach a Lightning micropayment directly to the routing request.
- **`priority_fee`:** A metadata field indicating the attached fee. Relays automatically move Notes with sufficient priority fees into the highest-speed queue.
- **Proof of Priority:** The fee *is* the proof. The Relay doesn't need to know *why* the Note is important, only that the sender is willing to pay for bandwidth.
**** Economic Density & Wire-Size Billing
Relays manage their resources using an **Economic Density** metric:
- **Sender Reputation (Zero-Fee Routing):** Small Notes from highly staked or reputable DIDs are often routed with zero additional fees to foster network liquidity and social interaction.
- **Low-Density (Large/Static):** Large Notes (e.g., binary payloads, media) are routed via **Bulk Pipes**. Billing for these Notes is proportional strictly to their raw payload size on the wire.
**** Incentivization
- Relays charge for routing high-traffic content or for specific delivery guarantees based on wire-size and explicit priority fees.
*** Relay Discovery
*** Relay Economics and Network Resilience
**** Relay Discovery
***** Bootstrap Problem
New clients need to find Relay nodes without hardcoded lists (centralization risk).
***** Discovery Mechanisms
****** DNS TXT Records
- Domain: `_agora-relay._tcp.example.com`
- Returns: List of relay endpoints (WebSocket URLs)
- Update: DNS propagation handles relay churn
****** Well-Known DID
- DID: `did:agora:bootstrap`
- Service Endpoint: "RelayDirectory" with list of known high-reputation relays
- Maintained: By Agora Trust, updated quarterly
****** DHT (Future)
- Distributed hash table for relay discovery
- Similar to BitTorrent trackerless torrents
- Resistant to censorship
****** Social Bootstrap
- Friend's relay list shared on connection
- "You follow Alice → Oh, Alice uses Relay X, try that"
- Gossip protocol for relay reputation
**** Relay Subscription
***** Subscription Types
- *CID Filters:* Subscribe to new CIDs from specific DIDs
- *Topic Filters:* Subscribe to content with specific tags
- *Content-Type Filters:* Only contracts, only posts, etc.
***** Subscription Management
- *WebSocket:* Persistent connection for real-time updates
- *REST Poll:* HTTP polling for clients without WebSocket
- *Webhook:* Push notifications for server-side clients
***** Subscription Pricing
- *Basic:* Free (up to 100 subscriptions)
- *Premium:* 100 satoshis/month (unlimited)
- *Enterprise:* Negotiated (dedicated relay capacity)
**** Relay Operator Profiles
1. **"Backbone" Providers (Big Tech/NGOs):** Large organizations (e.g., Bluesky Social PBC or the "Free Our Feeds" collective) run "Global Relays," ingesting entire network activity for platform-wide search and indexing.
2. **"Neighborhood" Operators (NGOs & Communities):** NGOs, professional guilds, or town councils run community-specific relays, indexing only the members relevant to their specific domain.
3. **"Specialists" (Commercial Startups):** Companies (e.g., Primal, River) run highly-optimized relays to power specific apps, prioritizing speed and specialized feature sets for their target market.
**** Relay Economic Models
To ensure sustainability without compromising user data (avoiding "Surveillance Capitalism"), operators utilize diverse revenue models:
- **The "Anti-Spam" Entrance Fee:** One-time or monthly payments (e.g., $1–$5) via Bitcoin Lightning to discourage bot-farms and cover hardware costs.
- **The "Boutique" Model (Freemium):** Free "Read" access with a paid subscription required to "Write" (post data), often offering guarantees for data persistence and indexing quality.
- **Institutional "Public Good" Funding:** Government or NGO-funded "Public Interest Relays" provided as a digital utility to ensure censorship-resistant communication.
- **"Zaps" & Micro-tips:** Direct user-to-operator micro-tips via integrated Lightning Keys, rewarding relays for high-quality filters or specialized indexes.
**** Relay Pricing
***** Standard Price Announcement
- Relay publishes `price_per_kb` in Lightning millisats
- WebSocket endpoint: `/pricing` returns current rates
- Update frequency: Hourly, cached by clients
***** Pricing Tiers
- *Basic:* 1 millisat/KB (default routing)
- *Priority:* 10 millisats/KB (fast lane)
- *Bulk:* 0.5 millisats/KB (>100KB messages)
- *Free:* 0 millisats/KB (below 1KB, within rate limits)
***** Fee Structure
- *Relay:* Keeps 70% of fees (operating costs)
- *Validator Oracles:* 20% (fraud detection)
- *Agora Protocol:* 10% (development fund)
**** Network Resilience: Global Firehose vs. Fragmented Relays
The Agora design ensures that the relay network is inherently replaceable and resilient:
- **Replaceable Relays:** Users can instantly switch to competitor relays if a provider becomes greedy or censorious by simply re-pointing their PDS.
- **"Multi-homed" Data (Firehose Protection):** Users push posts to multiple relays simultaneously. If any relay fails, history remains accessible via others, ensuring followers can always maintain connectivity.
*** Privacy Considerations: The "Honeypot Relay" Risk
Because a relay is fundamentally a server that routes traffic, it is technically possible for an operator to offer a "free" service while secretly harvesting metadata to sell to advertisers. This creates the risk of "Honeypot Relays" during the network's early bootstrap phase.
**** The Metadata Harvesting Trap
Even if messages are End-to-End Encrypted (E2EE), a malicious relay can observe highly valuable metadata for surveillance capitalism:
- *IP Addresses:* Revealing the user's physical location and device profile.
- *The Social Graph:* Observing who a DID communicates with, how often, and who constitutes their "inner circle."
- *Activity Patterns:* Tracking when a user is active, sleeping, and which tags/topics they frequently query.
- *Unencrypted Content:* Building interest profiles based on public posts and read-only data.
**** Defense Against Decentralized Surveillance
While Honeypot Relays pose a risk, their power is fundamentally weaker than legacy Web 2.0 walled gardens:
1. *No Walled Garden (Instant Migration):* If a relay is discovered to be harvesting data, users simply uncheck a box in their PDS settings. They migrate to a new relay instantly, and their followers find them immediately because their identity (DID) remains constant.
2. *Fragmented Data:* Because users multi-home their data across several relays simultaneously, no single relay possesses the "whole picture" of a user's digital life.
3. *The Tor/VPN Option:* Advanced users and organizations can run their PDS traffic through Tor or a VPN, stripping away the IP address—the most valuable piece of surveillance data.
**** Organizational Defense: The Tiered Relay Strategy
For Collectives, NGOs, or LLCs managing sensitive operations, relying on "free" public relays is an unacceptable security risk. These entities MUST utilize a Tiered Relay Strategy:
- *Tier 1 (Internal Relay):* The NGO runs its own private, isolated relay exclusively for internal board and team communications. This relay is "dark" to the public internet and collects zero metadata for third parties.
- *Tier 2 (Public Gateway):* The organization uses high-traffic "Surveillance" or public relays solely for PR, marketing, and public announcements. They treat these relays like digital billboards—expected to be public, but never used for private business.
**** Relay Reputation
- *Uptime:* % availability over last 30 days
- *Latency:* Message propagation time
- *Censorship:* Has relay blocked legal content?
- *Fees:* Reasonable pricing?
- *Users:* Number of active personas (network effect)
** Search & Indexing: The Firehose Indexers
In a decentralized network, finding historical content or discovering new Personas requires specialized indexing infrastructure. Indexing Nodes are high-performance servers that ingest the public Relay firehose to provide full-text search and complex discovery services.
*** Requirements
- Indexing Nodes MUST ingest public Content Objects from the Relay firehose.
- Indexing Nodes MUST support full-text search across public metadata and decrypted public content.
- The system MUST provide a standardized Search API for clients to query indexes.
- **The Aggregator API:** The standard endpoint MUST support an open querying format (e.g., `GET /search/query?q=keyword`) returning a structured schema of DIDs, Handles, and Profile snippets.
- **Ranking Transparency:** The provider MUST publish its Ranking Logic (e.g., "We prioritize accounts with 3+ Web-of-Trust endorsements") so users understand the algorithmic biases of the index.
- Indexing Nodes MUST respect content flags (e.g., `indexable: false` or `ephemeral`).
- The system MUST support "Private Indexing," where a user's PDS builds a local search index for their own encrypted history and DMs.
*** Technical Logic
- **Public Indexing:** Backbone providers run global indexers using technologies like Meilisearch or Elasticsearch to enable "Google-like" search for the whole platform.
- **Private Indexing:** Thin clients delegate private search tasks to the user's PDS, which maintains a local, encrypted index of all authorized Content Objects.
- **Economics:** High-speed indexing services MAY utilize micro-payments (Lightning) or subscriptions for advanced query capabilities or higher rate limits.
*** Persona Discovery & The Global Directory
To solve the UX problem of finding friends in a decentralized namespace, Indexers act as a Global Directory, constantly cataloging Handle <-> DID mappings broadcast over the network.
**** Multi-Format Handle Indexing
When a user searches for "@alice," the Indexer searches across all supported namespace formats simultaneously:
- **Subdomains:** `alice.aletheia.social`
- **Custom Domains:** `alice.com`
- **Web3 Names:** `alice.eth` or `alice.p2p`
**** Verified Search Results (Anti-Squatting)
Because anyone can theoretically publish a Note claiming to be "Alice," the Search UI relies on DIDs to guarantee authenticity.
- **Cryptographic Back-Links:** The Search engine ONLY displays a "Verified" checkmark if the human-readable handle (e.g., `alice.com`) has a valid cryptographic DNS/TXT record pointing back to the Persona's DID, *and* the DID has published a signed Note claiming that handle.
- **Unverified Flagging:** If a user squats on a username without owning the corresponding domain or blockchain record, the Indexer explicitly flags the search result as "Unverified" or excludes it.
**** "Privacy-First" Search
Because Agora supports multiple isolated Personas per user, global search is strictly opt-in:
- **Public Personas:** (e.g., a "Work" or "Creator" Persona) publish a "Directory Opt-In" Note. Indexers catalog them, making them searchable by anyone.
- **Private Personas:** (e.g., an "Anonymous" or "Family" Persona) do not publish this Note. They are entirely hidden from global Indexers. To find a Private Persona, another user must possess their exact DID string or be invited via a secure DIDComm routing channel.
** Agora-to-Web Gateways: The Bridge to the Open Web
*** Concept
To make decentralized, P2P content accessible to users on the "Open Web" (traditional browsers like Chrome or Safari without special plugins), Agora must bridge the gap between Content-Addressed Data (CIDs) and Location-Addressed URLs.
Gateways act as "translators" sitting on the edge of the decentralized network, talking HTTP to the legacy web while speaking P2P protocols internally. Every PDS or dedicated "Public Relay" can act as a web gateway.
*** 1. The HTTP Gateway (The Bridge)
**** Public Gateway API & URL Mapping
A piece of content identified by its hash (CID), such as `bafy...123`, can be viewed by anyone at a standard HTTP URL.
- **Pathing:** Gateways MUST support standard pathing `/ipfs/{cid}` and `/at/{did}/{collection}/{rkey}`.
- **CORS Policy:** Gateways MUST implement a permissive CORS policy to allow decentralized applications (dApps) to fetch media directly across origins.
- **MIME-Type Sniffing:** The gateway MUST read the Universal Event metadata and serve correct HTTP headers (e.g., `Content-Type: video/mp4`) so browsers handle the media natively.
**** The Translation Process
When a browser hits that link, the Gateway performs the following automated steps:
1. **Fetch:** Retrieves the data from the P2P swarm using Agora's native protocols.
2. **Verify:** Cryptographically verifies the Note's signature against the creator's Persona DID to ensure authenticity.
3. **Wrap:** Wraps the raw content (Markdown, JSON) in standard HTML/CSS templates so it renders correctly in a standard web browser.
*** 2. Human-Readable Handles (DNS & ENS)
Most users will not share long cryptographic hashes. To make content web-friendly, Gateways automate domain routing.
**** DNSLink (Traditional Domains)
Users can point their own domains (e.g., `alice.com`) directly to their Persona.
- **Automatic Resolution:** When someone visits `alice.com`, the Gateway reads a DNS TXT record that says: "Go find content hash XYZ on the Agora network."
- **Zero-Configuration SSL:** Gateways automatically provision and renew HTTPS certificates (via Let's Encrypt) for any domain linked to a Persona DID.
- **Well-Known Verification:** Gateways automatically serve the user's DID document at `https://[custom-domain]/.well-known/atproto-did` to prove ownership.
**** Automated Subdomain Issuance (Registrar Service)
To onboard users quickly without forcing them to buy a domain, PDS providers act as Registrars.
- **Availability & Routing:** The PDS performs an automated availability check. If a handle is free, it updates its Virtual Host configuration and internal DNS to instantly route traffic for `newuser.provider.org`.
**** Web3 Domains (.eth, .p2p)
For users utilizing blockchain-based naming services, Agora integrates with specialized gateways (e.g., Eth.limo). A user types `yourname.eth.limo` into a standard browser, and the gateway does the heavy lifting of resolving the blockchain record and serving the underlying P2P data.
*** 3. Social Mirroring for Search Engines (SEO)
To ensure Agora content is discoverable on legacy search engines like Google, the network utilizes automated rendering pipelines.
**** The Firehose
Agora Relays emit a continuous "Firehose" of every public Note created on the network.
**** SEO Rendering (App Views)
Specialized indexers or "App Views" (functioning like web-frontends) consume this firehose. They automatically generate static, crawlable HTML pages for every public profile, post, and thread. This ensures that decentralized content is aggressively indexed by Google's web crawlers, matching or exceeding the discoverability of traditional centralized blogs.
*** 4. Persona-as-Host (Native Web Hosting)
Because of this robust Gateway architecture, publishing a website becomes a native feature of the network.
- **Static Asset Resolver (SAR):** The PDS/Gateway can interpret a directory CID as a web root. If a request hits a folder CID without a filename, the SAR automatically serves `index.html`. It resolves all internal assets (images, CSS) using Relative Pathing to ensure the site works regardless of the gateway domain.
- **Automated Deployment (Push-to-Publish):** Developers can use Git integration. When code is pushed, a CI/CD action builds the site, signs the resulting root CID with the Persona Key, and broadcasts the Publish Event to the PDS.
- **Instant Rollbacks:** Every Publish Event is logged in the Persona's immutable history. Reverting a site is simply signing a new Note pointing back to a previous CID.
*** Monetized Static Sites (Split-State Encryption)
Gateways integrate with the Exchange layer. Owners can host static sites where certain paths are encrypted. The Gateway serves the public storefront, but the actual asset is only "unwrapped" locally if the user's browser provides a Lightning Preimage (LSAT) proving payment.
*** Requirements
- Gateways MUST take CID-based Agora content and render it as HTML for legacy browsers.
- Gateways MUST support SEO-friendly rendering for public content.
- The system MUST allow anyone to run a Gateway (not restricted to Relay operators).
- Gateways MUST NOT require authentication for public content.
- Gateways SHOULD cache content to reduce load on PDS/Relay networks.
- The system MUST support Gateway discovery (similar to Relay discovery).
- Gateways MUST respect content flags (e.g., `indexable`, `ephemeral`).
- Gateways MUST NOT expose private/direct content.
*** Relationship to Relays
- *Relays* serve Agora-native clients via WebSocket/pub-sub protocols.
- *Gateways* serve legacy browsers via HTTP.
- They are *separate infrastructure* - a Gateway may use Relays as a backend, but they're distinct services.
*** Gateway Discovery
**** Concept
Gateways bridge Agora content to the legacy web via HTTP. Discovery mechanisms are needed for clients to find reliable gateways to generate shareable HTTP links for their public content.
**** Discovery Mechanisms
***** Public Registry
- A well-known DID (e.g., `did:agora:gateways`) maintains a list of verified, active gateways.
- Clients can query this registry to get a randomized or latency-sorted list of active gateways.
***** DNS TXT Records
- Similar to Relay discovery, domains can publish `_agora-gateway._tcp` TXT records pointing to HTTP endpoints.
***** User Preference
- Users can manually configure a preferred gateway in their client settings (e.g., `agora.example.com`).
- Clients use this preferred gateway when generating "Share Link" URLs.
** Infrastructure Discovery: DID Document Bindings
For a Persona to function within the network, its Decentralized Identifier (DID) must "bind" to specific infrastructure endpoints. This is achieved via the `service` section of the Agora DID Document.
*** The Service Schema
Every Agora DID Document SHOULD include a list of service endpoints that allow other Personas and clients to locate the user's data and communication channels.
#+begin_src json
{
"id": "did:agora:123...",
"service": [
{
"id": "#pds",
"type": "AgoraPDS",
"serviceEndpoint": "https://pds.example.org"
},
{
"id": "#relay",
"type": "AgoraRelay",
"serviceEndpoint": "wss://relay.aletheia.social"
},
{
"id": "#search",
"type": "AgoraSearch",
"serviceEndpoint": "https://search.agora-backbone.net"
}
]
}
#+end_src
*** Resolution & Routing Logic
1. **Discovery:** When a client wants to interact with a Persona, it first resolves the DID to its latest DID Document (via the KEL or a resolver).
2. **Binding:** The client extracts the `AgoraPDS` endpoint to fetch content and the `AgoraRelay` endpoint to subscribe to real-time updates.
3. **Failover:** If a primary PDS is unreachable, the client attempts to resolve alternative endpoints if provided in the service list (supporting the multi-homed data model).
** Client Architecture: PDS-Proximate / Thin Client Model
*** Concept
Agora's architectural strategy for client applications aims to balance user sovereignty with broad accessibility and app store compliance. Instead of relying solely on "sovereign clients" (full-featured applications running entirely on edge devices), a hybrid approach will be adopted: core client logic will reside closer to the Personal Data Store (PDS), with only a "thin client" deployed on edge devices (e.g., mobile apps, web browsers). This allows for greater flexibility in distribution and development.
*** Motivation: App Store Compliance & Broad Reach
Traditional "sovereign client" models, where full application logic, data processing, and cryptographic operations occur entirely on the user's edge device, can encounter significant hurdles with mainstream app stores (e.g., Apple App Store, Google Play Store). These platforms often impose restrictions on:
- Custom cryptographic implementations
- Direct access to underlying network protocols
- Data storage and handling outside platform-defined sandboxes
- Features deemed to circumvent platform monetization or control
The PDS-proximate / thin client model is a pragmatic solution to navigate these limitations, enabling Agora to reach a wider user base through conventional app distribution channels without compromising core protocol principles.
*** Strategic Advantages
1. **Enhanced App Store Compliance:** A thin client, functioning primarily as a user interface and communication layer with the PDS, is less likely to violate app store guidelines, increasing the likelihood of approval and sustained availability.
2. **Reduced Edge Device Footprint:** Lower computational, memory, and storage requirements on mobile phones and other edge devices. This translates to better performance, lower battery consumption, and broader compatibility across a range of hardware.
3. **Streamlined Updates & Maintenance:** Core application logic and feature updates can be deployed directly on the PDS or associated infrastructure, minimizing the need for frequent client-side app store updates and accelerating development cycles.
4. **Enriched PDS Functionality:** The PDS evolves from a passive data archive into a more active, "smart" personal server capable of hosting and executing significant portions of the client application logic. This allows for more sophisticated features (e.g., advanced indexing, content processing, AI integration) to run efficiently on behalf of the user.
5. **Greater Platform Portability:** A thin client model naturally facilitates deployment across diverse platforms, including web browsers (via WebAssembly or JavaScript), minimal native mobile wrappers, and potentially embedded systems, ensuring a consistent user experience.
*** Architectural Considerations & Challenges
1. **PDS Reliability and Performance:** The user experience becomes intrinsically linked to the performance, uptime, and responsiveness of the PDS. Robust PDS implementations and reliable hosting options are paramount.
2. **Privacy and Trust Model:** While the PDS is personal to the user, moving client logic there means processing occurs "off-device." End-to-end encryption must remain a fundamental guarantee, ensuring the PDS only handles encrypted data where sensitive information is concerned. User control over the PDS becomes the primary locus of sovereignty.
3. **Offline Capabilities:** Thin clients will inherently have limited or no offline functionality, as they rely on real-time communication with the PDS. Strategies for graceful degradation and cached read-only access for essential data will be necessary.
4. **Definition of "Thinness":** A clear architectural specification is required to define the boundary between client logic executed on the PDS and the minimal essential logic (e.g., basic key management, UI rendering) that must reside on the edge device for security and usability.
*** Conclusion
The adoption of a PDS-proximate / thin client architecture is a strategic imperative for Agora to achieve mass adoption and navigate the complexities of modern app distribution, while simultaneously enhancing the capabilities of the Personal Data Store as a dynamic and powerful extension of the user's digital self.
** Related Documents
- [[file:agora-requirements-04-the-primitive.org][The Primitive]] - Content Object structure and core encryption primitives.
- [[file:agora-requirements-05-social.org][Social]] - Messaging models, social publishing, and the attention marketplace.
- [[file:agora-requirements-02-identity.org][Identity]] - Master Key (Psyche) and Persona governance.
- [[file:agora-requirements-06-exchange-and-contracts.org][Exchange and Contracts]] - Economic primitives, fee structures, and the SCAL layer.
** Gaps
- *None.* All identified gaps in the infrastructure layer have been resolved.

428
projects/agora/agora-requirements-04-the-primitive.org Normal file
View File

@@ -0,0 +1,428 @@
#+TITLE: Agora Requirements - 04: The Primitive
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:28]
#+DATE: 2026-03-15
#+ID: agora-requirements-04-the-primitive
#+STARTUP: content
* The Primitive: The Atomic Foundation of Agora
** Concept: The Universal Data Primitive
The Primitive is Agora's foundational content layer—the base data structure upon which all social interaction, economic exchange, and identity management is built. Before there are posts, contracts, or profiles, there are Notes. The Note is the atomic, universal unit of information in Agora.
This elegant simplicity—the "Everything is a Note" paradigm—enables Agora's powerful interoperability, immutable audit trails, and seamless cross-application compatibility. By reducing all digital interactions to a single, cryptographically verifiable primitive, Agora creates a unified ecosystem where any application can understand and process any data, breaking down the silos that plague traditional digital platforms.
** The Note Structure
A Note is the atomic unit of information in Agora. Everything—posts, messages, contracts, profiles—is a Note with behavioral flags.
*** Technical Specification
Every Note is identified by its CID (Content Identifier):
- *Format:* CIDv1 with configurable codec and hash (Default: `dag-pb` + `sha2-256`).
- *Property:* Same content = same CID (deterministic).
- *Immutability:* Content cannot change without CID changing.
#+begin_src json
{
"cid": "string", // Unique content identifier
"owner": "DID", // Source of authority (Persona DID)
"is_feed": boolean, // Behavioral Intent: Chronological Flow (true) vs Static Page (false)
"contract": { ... }, // Optional: Rules of engagement (JSON Object)
"payload": "string", // Optional: The asset (Inline data or P2P CID)
"content_type": "string", // MIME type (e.g., text/markdown, image/jpeg)
"priority_fee": integer, // Optional: Relay routing priority (millisats)
"access_control": ["DID"], // Permissions (Omitted=Personal, []=Public)
"notify": ["DID"], // Attention (Target entities for push notifications)
"references": ["CID"], // Semantic links/citations
"reply_to": "CID", // Parent CID (for threading/negotiation)
"thread_root": "CID", // Root CID of the conversation/exchange
"ephemeral_duration": integer, // Expiry in seconds (0 = persistent)
"createdAt": "timestamp", // ISO-8601 creation time
"proof": { // Cryptographic authenticity
"editor": "DID", // Optional: The signer (defaults to owner)
"signature": "string" // Signature over Note content
}
}
#+end_src
** Behavioral Intent & Schema Validation
The single `is_feed` property defines the behavioral intent of a Note without changing its underlying technical structure.
*** Core Behavioral Intent
| Property | Type | Description |
|------|---------|----------|
| `is_feed` | boolean | Chronological timeline item (Post, Status, Update). If false/omitted, the Note is a static Page. |
*** The Contract & Payload Split
Every signed Note in Agora is inherently a contract. To clearly separate the "Rules of Engagement" from the "Asset", the Note structure defines two distinct fields:
- **`contract` (JSON Object):** Defines the terms. This includes both human-readable terms (e.g., `"license": "CC0"`) and machine-readable state (e.g., `"price_satoshis": 5000`).
- **`payload` (Polymorphic String):** Defines the asset governed by the contract. This can be:
1. **Inline Data:** Raw text, markdown, or small base64 blobs embedded directly.
2. **P2P Reference (CID):** A URI (e.g., `ipfs://Qm...`) pointing to a distributed Merkle DAG hosted by the PDS or the Swarm.
*** Audience & Visibility (access_control)
The visibility and routing of a Note are determined by the `access_control` array:
- **Personal (Private):** Omitted or `null`. The Note remains internal to the PDS.
- **Public (Broadcast):** Explicit empty array `[]`. The Note is pushed to all subscribed Relays for global indexing.
- **Restricted (Directed):** Array of target DIDs `["did:agora:bob"]`. The Note is routed only to the specified recipients.
*** Attention & Intent (notify)
The `notify` array defines who should receive a push notification or "Inbox" alert for the Note:
- **Personal/Silent:** Omitted or `null`. No entities are notified.
- **Targeted Ping:** Array of target DIDs `["did:agora:bob"]`. Triggers a notification for the specified entities.
*** Semantic Derivations
Because Agora uses a minimalist flag system, high-level social and economic concepts are reconstructed by clients using core flags, audience scope (`access_control`), and Note relationships (`references`, `reply_to`, `notify`).
**** Basic Content
- **Public Post:** `is_feed:true` + `access_control:[]`
- **Private DM:** `access_control:["did:agora:bob"]` + `notify:["did:agora:bob"]`
- **Static Page:** `is_feed:false` + `access_control:[]`
- **File:** A Note with a binary `content_type` (e.g., `image/jpeg`).
**** Social Graph & Interaction
- **Like / Reaction:** A Note that `references` a Content CID and contains a reaction payload. Typically `is_feed: false`.
- **Boost / Repost:** A Note that `references` a Content CID with `is_feed: true`, injecting it into the owner's chronological timeline.
- **Follow:** A Note that `references` a Persona DID.
- **Public Mention:** `access_control:[]` + `notify:[Target_DID]`.
- **Private Connection:** `access_control:[Target_DID]` + `notify:[Target_DID]`.
**** Economic & Contract Lifecycle
- **Contract Negotiation (Offer/Take/Task):** A Note represents a proposal (**Offer**), an acceptance (**Take**), or a commitment to perform work (**Task**) depending on its place in the `reply_to` chain.
- **Economic Lifecycle (Invoice/Payment/Escrow):**
- **Invoice**: A Note with a payment request in its `contract` (`price_satoshis`).
- **Payment**: A fulfillment Note (`Take`) containing cryptographic proof (e.g., `preimage`).
- **Escrow**: A Note referencing a multi-signature threshold in its `contract`.
- **Support / Subscribe:** A Note referencing a Persona DID, establishing a recurring payment stream or premium access in its `contract`.
**** Events & Coordination
- **Event Announcement:** A Note (usually `is_feed: true`) where the `contract` defines temporal/spatial rules (start time, location, capacity).
- **Invite:** A directed Note (`access_control: [DID]`, `notify: [DID]`) that `references` an Event Announcement. It serves as a contract **Offer** for attendance.
- **RSVP:** A Note that `reply_to` an Invite. The `contract` field contains the acceptance state (`{"rsvp": "attending"}`), acting as a **Take**.
*** Flag Combination Rules
Agora implements strict validation to ensure network integrity.
**** Rule 1: Flow (Feed vs. Page)
- `is_feed: true` indicates chronological content.
- `is_feed: false` (default) indicates static resource.
**** Rule 2: Audience Scope
- **Public Broadcast:** MUST use an explicit empty array `access_control: []`.
- **Restricted Routing:** MUST provide at least one recipient DID in `access_control`.
- **Personal:** Omission of `access_control` defaults to private storage on the PDS.
**** Rule 3: Requirements & Dependencies
- **Ephemerality:** The presence of `ephemeral_duration > 0` indicates the Note is ephemeral.
- **Restricted Access:** If `access_control` is populated, both the `contract` and `payload` SHOULD be encrypted into a single envelope for the specified audience.
*** Technical Specification (JSON Schema)
#+begin_src json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://agora.ai/schemas/content-flags.json",
"title": "Agora Note Flags",
"description": "Validation schema for the Binary Core flag set",
"type": "object",
"properties": {
"cid": {
"type": "string",
"description": "Unique content identifier.",
"pattern": "^[a-zA-Z0-9]+$"
},
"owner": {
"type": "string",
"description": "DID of the owner persona.",
"pattern": "^did:agora:[a-zA-Z0-9]+$"
},
"is_feed": {
"type": "boolean",
"description": "Chronological timeline item (Post/Update). If false, it's a static Page.",
"default": true
},
"contract": {
"type": "object",
"description": "Optional rules of engagement governing the payload (e.g. licensing, price).",
"additionalProperties": true
},
"payload": {
"type": "string",
"description": "The asset content (inline or P2P reference CID)."
},
"content_type": {
"type": "string",
"description": "MIME type of the content.",
"enum": [
"text/plain",
"text/markdown",
"text/html",
"application/json",
"image/jpeg",
"image/png",
"image/gif",
"video/mp4",
"audio/mpeg",
"application/pdf",
"application/zip",
"application/jwe"
]
},
"priority_fee": {
"type": "integer",
"description": "Relay routing priority in millisats.",
"minimum": 0
},
"access_control": {
"type": "array",
"description": "Determines audience. Omitted=Personal, []=Public, [DIDs]=Restricted.",
"items": {
"type": "string",
"pattern": "^did:agora:[a-zA-Z0-9]+$"
}
},
"notify": {
"type": "array",
"description": "Targets for push notifications.",
"items": {
"type": "string",
"pattern": "^did:agora:[a-zA-Z0-9]+$"
}
},
"references": {
"type": "array",
"description": "CIDs of related content objects.",
"items": {
"type": "string",
"pattern": "^[a-zA-Z0-9]+$"
}
},
"reply_to": {
"type": "string",
"description": "CID of content this is a reply to. Required for reply types.",
"pattern": "^[a-zA-Z0-9]+$"
},
"thread_root": {
"type": "string",
"description": "CID of the root post in a thread.",
"pattern": "^[a-zA-Z0-9]+$"
},
"ephemeral_duration": {
"type": "integer",
"description": "Duration in seconds for ephemeral content. If 0 or omitted, the Note is persistent.",
"minimum": 0,
"maximum": 31536000
},
"createdAt": {
"type": "string",
"format": "date-time",
"description": "ISO-8601 creation timestamp."
},
"proof": {
"type": "object",
"description": "Cryptographic proof of authenticity.",
"properties": {
"editor": {
"type": "string",
"description": "Optional: DID of the signing persona. Defaults to owner if omitted.",
"pattern": "^did:agora:[a-zA-Z0-9]+$"
},
"signature": {
"type": "string",
"description": "Ed25519 signature over content hash.",
"pattern": "^[A-Za-z0-9+/]+=*$"
}
},
"required": ["signature"]
}
},
"additionalProperties": false
}
#+end_src
** Content Lifecycle & Persistence
*** Encryption: Security by Design
Security is woven into the fabric of the protocol. Agora uses industry-standard primitives to ensure that only intended recipients can access private content.
- **End-to-End Encryption (E2EE):** Private Notes use AES-256-GCM for payloads and X25519 for ECDH key exchange.
- **Forward Secrecy:** Agora employs Double Ratchet for 1-on-1 messaging and MLS (Messaging Layer Security) for groups, rotating keys per-message.
*** Ephemeral Content Enforcement
The `is_ephemeral: true` flag is enforced through three complementary mechanisms:
1. **Time-Locked Encryption (Primary):** Payloads are encrypted with keys that can only be retrieved from a Decentralized Key Management Network (DKMN) or solved via a Time-Lock Puzzle before the expiration time.
2. **Key Shedding (Forward Secrecy):** For DMs, the client securely deletes the specific message key after the display duration expires.
3. **Voluntary Infrastructure Compliance:** PDS nodes MUST garbage collect expired CIDs, and Relays MUST drop them from routing tables.
*** Note Persistence (PDS)
- **Home Base:** All Notes are stored in the owner's Personal Data Store (PDS) by default.
- **Availability:** Content is hosted by the PDS, replicated across mirrors, and cached by Relays/clients for performance.
- **Lifecycle:** Create → Store (PDS) → Announce (Relay) → Fetch → Decrypt → Render.
** Relationships, Sync & Performance
*** Note Relationships
Agora uses three distinct fields to define relationships between Notes, balancing semantic precision with high-performance discovery.
**** Threading & Reference Logic
- **`references` (Array, 0-N):** General semantic linking. This field is used for citations, user mentions, quoting other posts, or attaching auxiliary Content Objects. It tells the network: "This Note is related to these other things."
- **`reply_to` (Single, 0-1):** Direct parentage. This field is mandatory for any Note that is part of a branching conversation. It defines the exact hierarchy for UI indentation and determines which owner should receive a notification.
- **`thread_root` (Single, 0-1):** The Global Anchor. This points to the very first Note that initiated the entire conversation. It allows clients to fetch thousands of replies in a single batch query, avoiding the "N+1 fetch" performance bottleneck.
***** Comparison Summary
| Field | Cardinality | Primary Role | UI Impact |
| :--- | :--- | :--- | :--- |
| **`references`** | Array (0-N) | Citation/Linking | Link previews, mentions |
| **`reply_to`** | Single (0-1) | Parentage | Nesting/Indentation |
| **`thread_root`** | Single (0-1) | Grouping | "View Full Thread" performance |
***** Example Implementation Scenario
Alice posts a product listing (Note A). Bob asks a question (Note B) about the listing. Charlie replies to Bob (Note C) but also quotes Alice's original product photo (Note D) in his comment.
**Charlie's Note (Note C) logic:**
- `thread_root`: CID of Note A (The listing anchor).
- `reply_to`: CID of Note B (The immediate parent).
- `references`: [CID of Note B, CID of Note D] (The citations).
*** Large Payload Handling
- **Streaming Protocol:** Files >100MB are split into 1MB chunks and represented as a Merkle DAG.
- **Streaming CIDs:** The root CID points to the tree, allowing concurrent, prioritized downloading of chunks.
*** Real-time Sync & Collaboration
- **Live Collaboration:** Agora uses CRDTs (Conflict-free Replicated Data Types) for shared state (e.g., co-editing a document).
- **Ephemeral Channels:** Real-time updates (like typing indicators) are broadcast via Relay WebSockets without being committed to the PDS as permanent Notes.
*** Content Deduplication
- **Block-level Deduplication:** Since payloads are content-addressed, PDS nodes only store identical data once, using reference counting to manage garbage collection.
** Validation Reference (Examples)
*** Valid: Public Post
#+begin_src json
{
"cid": "QmPost123",
"owner": "did:agora:alice",
"is_feed": true,
"contract": {
"license": "CC-BY-4.0"
},
"payload": "Hello, Agora!",
"content_type": "text/markdown",
"access_control": [],
"createdAt": "2026-03-25T14:30:00Z",
"proof": {
"signature": "abc123..."
}
}
#+end_src
*** Valid: Private DM
#+begin_src json
{
"cid": "QmDM456",
"owner": "did:agora:alice",
"payload": "eyhbGciOiJkaXIiLCJlbmMiOiJBMjU2R0NNIn0...",
"content_type": "application/jwe",
"access_control": ["did:agora:bob", "did:agora:alice"],
"notify": ["did:agora:bob"],
"createdAt": "2026-03-25T14:35:00Z",
"proof": {
"signature": "def456..."
}
}
#+end_src
*** Valid: Digital Storefront (Split-State Encryption)
#+begin_src json
{
"cid": "QmStore789",
"owner": "did:agora:alice",
"is_feed": false,
"contract": {
"title": "Exclusive Indie Film",
"price_satoshis": 50000,
"decryption_method": "LSAT"
},
"payload": "ipfs://QmEncryptedVideoChunks...",
"content_type": "application/vnd.agora.encrypted+video/mp4",
"priority_fee": 1000,
"access_control": [],
"createdAt": "2026-03-25T14:40:00Z",
"proof": {
"signature": "xyz012..."
}
}
#+end_src
*** Valid: Ephemeral Story (Public)
#+begin_src json
{
"cid": "QmStory789",
"owner": "did:agora:alice",
"is_feed": true,
"payload": "This disappears in 24 hours",
"access_control": [],
"ephemeral_duration": 86400,
"createdAt": "2026-03-25T14:45:00Z",
"proof": {
"editor": "did:agora:bot_agent",
"signature": "ghi789..."
}
}
#+end_src
*** Invalid: Broadcast Conflict
#+begin_src json
{
"cid": "QmInvalid001",
"access_control": [],
"payload": "encrypted-blob-here",
"content_type": "application/jwe"
}
#+end_src
Validation error: Public broadcast (`access_control: []`) cannot contain an encrypted payload.
*** Invalid: Restricted without Audience
#+begin_src json
{
"cid": "QmInvalid002",
"notify": ["did:agora:bob"]
}
#+end_src
Validation error: Notifications (`notify`) require the target DID to be present in the `access_control` list or for the Note to be public.
** Related Documents
- [[file:agora-requirements-02-identity.org][Identity]] - Personas and contracts
- [[file:agora-requirements-03-infrastructure.org][Infrastructure]] - PDS and Relay
- [[file:agora-requirements-05-social.org][Social]] - Relationships and communication
- [[file:agora-requirements-06-exchange.org][Exchange]] - Economic layer
** Gaps
- *None.* All identified gaps in the primitive layer have been resolved.
# Local Variables:
# org-confirm-babel-evaluate: nil
# End:

181
projects/agora/agora-requirements-05-social.org Normal file
View File

@@ -0,0 +1,181 @@
#+TITLE: Agora Requirements - 05: Social Space
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:28]
#+DATE: 2026-03-15
#+ID: agora-requirements-05-social-space
#+STARTUP: content
* Social Space: Where Human Connection Becomes Sovereign
The Social Space is where Agora's revolutionary architecture transforms how humans connect, communicate, and transact. Unlike traditional platforms that own your relationships and monetize your attention, Agora puts you in complete control of your social graph. Every interaction—from a casual conversation to a binding commercial contract—is self-owned, cryptographically secured, and entirely under your sovereignty. This is social interaction reimagined for a decentralized future.
** Concept
Social Space encompasses all person-to-person and person-to-collective interaction in Agora: public and private, asynchronous and real-time. All social interaction is mediated by Notes and contracts running on the Exchange layer.
** Asynchronous Communication (Correspondence & Messaging)
Asynchronous communication in Agora utilizes the **Secure Communication Module (SCM)**, which enforces the **DIDComm v2 (Decentralized Identifier Communication)** protocol—a transport-agnostic standard for secure, private communication.
- **Message Format:** All private messages MUST be formatted as JWM (JSON Web Messages).
- **Encryption Suite:** JWMs MUST be wrapped in a JWE (JSON Web Encryption) envelope, utilizing X25519 for key agreement and AES-256-GCM for content encryption.
*** The Mailbox (PDS as Proxy)
Because a user's primary device (e.g., a phone) is not always online, the PDS acts as an encrypted "Post Office" or proxy for asynchronous messages.
- **Sending:** The sender encrypts the Note using the recipient's Persona Public Key (retrieved from their DID Document).
- **Routing & Asynchronous Forwarding:** The encrypted JWE envelope is sent to the Service Endpoint listed in the recipient's DID Document. The PDS MUST support the DIDComm `Forward` message type, acting as an encrypted relay for offline delivery.
- **Storage:** The PDS stores the encrypted envelope. Because it is encrypted for the recipient's key, the PDS cannot read the content.
- **Pickup:** When the recipient's device wakes up, it fetches the envelope from the PDS, decrypts it locally, and deletes the copy from the PDS.
*** Contextual Isolation
Agora enforces strict multi-persona isolation. Each Persona (e.g., "Work," "Dating," "Personal") has a separate, cryptographically isolated message queue. A message sent to a user's Work DID never touches the inbox or metadata of their Dating DID, ensuring zero cross-context leakage.
** The Unified Note Primitive
All asynchronous interaction in Agora—whether a public post or a private message—is built upon the same "Note" primitive. The behavior and visibility of a Note are defined by cryptographic signatures and a set of standardized metadata flags.
*** Flag Definitions & Storage Models
| Flag | Meaning | Storage Model |
|------|---------|---------------|
| `access_control: []` | Broadcast (Public) | Reference-on-Send (authoritative on owner's PDS) |
| `access_control: [DIDs]` | Restricted (Private) | Copy-on-Send (authoritative on each recipient's PDS) |
| `is_feed: true` | Chronological entry (Post/Update) | Varies (e.g., public feed items are Reference-on-Send) |
| `is_feed: false` | Static resource (Page/Wiki) | Reference-on-Send |
*** Ephemeral Content
Notes where `ephemeral_duration > 0` are automatically garbage-collected by the PDS and dropped from routing tables by Relays after the duration expires.
*** Structural Integrity
Every async interaction is a Note identified by a Content Identifier (CID). This ensures that the history of a conversation or content stream is immutable and cryptographically verifiable.
** Directed Communication (Copy-on-Send Model)
For Notes intended for specific recipients (e.g., 1-on-1 messages, group chats), Agora employs a "Copy-on-Send" model to ensure recipient data ownership and high availability.
*** Audience & Attention
- **Audience:** Defined by the `access_control` array. These entities have the cryptographic right to own and decrypt the Note.
- **Attention:** Defined by the `notify` array. These entities receive a push notification or "Inbox" alert for the Note.
*** Mechanism
When an owner sends a directed Note (`access_control: [DIDs]`), a unique, encrypted copy is generated for each recipient and stored on their respective PDSs. The sender also retains a copy on their PDS.
*** Data Ownership
This model ensures recipients have full ownership and control over the messages they receive. Access to the Note is independent of the sender's PDS status after the initial send.
** Social Publishing: Feeds & Streams
For content intended for a broad audience (e.g., social posts, public articles, project wikis), Agora uses a "Reference-on-Send" model to maximize efficiency and owner control.
*** Concept: Feed vs. Stream
- **The Feed:** A Persona's curated output of chronological entries (`is_feed: true`) and static resources (`is_feed: false`).
- **The Stream:** A user's personalized, aggregated view of all the Feeds they follow.
*** The "Lens" Architecture (Polymorphic UI)
Because all data in Agora shares the exact same base schema (The Atomic Note), client applications are not locked into "siloed" databases. Instead, data is a single pile of uniform "bricks." The client app acts as a **Lens** that filters this stream and adjusts its interface based on the Note's internal metadata.
- **Unified Content Schema:** Apps do not maintain separate APIs for videos, products, or posts. They read the universal Note structure.
- **Dynamic Interfaces:** The UI interprets the `content_type` and `contract` fields to render the appropriate experience:
- If `content_type: "video/mp4"` (and duration is short): The UI enables a "TikTok-style" vertical scroll and auto-play.
- If `content_type: "audio/mpeg"`: The UI switches to a "Podcast" player with 1.5x speed and background play.
- If the `contract` contains `price_satoshis`: The UI injects a "Buy Now" button linked to a Lightning Invoice.
- **Fluid Content (Multiple Lenses):** Because the data is distinct from the UI, a single Note can be viewed through completely different lenses simultaneously. For example, a 10-minute video Note:
- One user views it through a **"YouTube Lens"** (displaying comments via `reply_to` links and related videos).
- Another views it through an **"Educational Lens"** (where a specific algorithm has filtered it alongside academic papers).
- A third user streams just the audio track through a **"Podcast Lens"** while driving.
*** Mechanism
When an owner creates a broadcast Note (`access_control: []`), it is stored authoritatively on their Personal Data Store (PDS). Interested parties (followers, caching Relays) receive a notification containing the Note's CID. Their clients then *pull* the content using that CID.
*** Owner Control
The authoritative copy resides solely on the owner's PDS. Deletion by the owner renders all references to that CID inaccessible across the network, providing a sovereign "Right to be Forgotten."
*** Content Types
- **Feed Entries (`is_feed: true`):** Chronological posts, status updates, and news articles.
- **Static Pages (`is_feed: false`):** Wikis, documentation, and personal profiles.
** Synchronous Communication (Live Voice & Video)
For real-time calls, Agora utilizes **WebRTC** with a decentralized twist for the signaling phase.
*** Decentralized Signaling
Traditional WebRTC requires a central signaling server to help devices discover each other. In Agora, the **DIDComm channel** handles the handshake:
1. **Request:** Persona A sends a "Call Request" via DIDComm to Persona B's PDS.
2. **Negotiation:** Persona B's phone receives the request and sends back its IP/ICE candidates (the "digital map") via the same secure DIDComm channel.
3. **P2P Tunnel:** Once the handshake is complete, voice/video data flows directly between the two devices. No server—not even the PDS—sees the call data.
*** Off-the-Record (OTR) Mode
To address the need for absolute privacy and deniability, OTR mode completely bypasses PDS storage.
- **Mechanism:** Encrypted channels exist only in volatile client memory for the duration of the session.
- **Persistence:** No persistent record is kept on any PDS or local client cache.
- **Recording:** Clients MUST explicitly prevent any recording when in this mode.
** Encryption & Metadata Privacy
Agora's communication layer goes beyond standard end-to-end encryption to ensure long-term security and metadata protection.
*** Double Ratchet Algorithm (Signal Protocol)
Every single message uses a new, derived key. This ensures **Perfect Forward Secrecy (PFS)** and **Post-Compromise Security**. If a specific message key is ever compromised, it cannot be used to decrypt past or future messages in the conversation.
*** Metadata Masking (Onion Routing)
To hide traffic patterns from network observers, Agora utilizes Tor-style **Onion Routing** between PDSs where possible. This masks who is talking to whom, preventing external observers from building a social graph based on connection frequency or message timing.
** Profiles
*** Concept
A Profile is a public-facing presentation of a Persona. Agora supports multiple Profiles per Persona (e.g., a "Public Developer" profile and a "Private Family" profile).
*** Requirements
- Each Profile MUST be a Note (CID) with public visibility.
- Profiles MUST be discoverable via the Naming Registry.
- Profile updates create new CIDs, preserving a verifiable history of the identity's presentation.
*** Profile as Static Site
Personas can publish their profiles and professional portfolios as decentralized static websites using the native hosting model (see [[file:agora-requirements-03-infrastructure.org][Infrastructure]]). Agora-aware browsers render these natively from CIDs, while legacy browsers access them via Gateways with automated SSL and domain mapping.
** The Attention Marketplace (The Information Router)
In traditional social media, the algorithm is a secret "Black Box" that sits between users and their social graph, deciding what is seen to maximize platform revenue. In Agora, the Algorithm Layer is reimagined as an open **Information Router**. By moving the algorithm out of the central server and into an open market, Agora empowers users to "hire and fire" the logic that sorts their attention.
*** Pluggable Feed Generation (PFG)
Users subscribe to independent "Feed Generators" via an open API. This decoupling of data from sorting logic is achieved through a three-step workflow:
1. **The Skeleton Request:** When a user opens their application, the client sends a request to a user-chosen Feed Generator (which can be operated by anyone—an NGO, a scientist, or a community collective).
2. **The Skeleton Response:** The Generator does not possess the user's private data. It returns a "Skeleton"—a lightweight JSON list of Content Identifiers (CIDs) that its specific logic has prioritized.
3. **Hydration:** The client application takes this list of IDs and "hydrates" the feed by pulling the actual Note content directly from the distributed PDS/Relay network.
*** The Algorithm Marketplace
Because the PFG API is open and transport-agnostic, different organizations compete to provide the best curation and routing services:
- **Academic Lenses:** Scientists or universities can provide generators that prioritize peer-reviewed content and primary sources.
- **Community Curators:** Local neighborhoods or professional guilds can run generators that surface the most relevant news for their specific domain.
- **Verification Services:** NGOs or fact-checking collectives can provide "Filtered Lenses" that prioritize highly-attested content.
*** Decentralized Moderation (Competitive Labeling)
Moderation in Agora is treated as "Competitive Labeling" rather than central censorship.
- **Labeler DIDs:** Independent services (NGOs, Fact Checkers, Church Groups) operate as "Labelers." They review the public firehose and "tag" content (e.g., "Spam," "Graphic," "High-Quality").
- **Client-Side Filtering:** The user's application pulls these public labels and applies the user's personal policy (e.g., "Hide anything labeled 'Graphic' by the NGO 'SafetyFirst'").
- **Stackable Moderation:** Users can subscribe to multiple labelers simultaneously to create a highly personalized, robust, and sovereign moderation filter.
*** Circular Economy: Following as Investment
Lightning micro-payments allow for a self-sustaining attention economy.
- **Incentivized Curation:** Feed Generators can charge micro-fees (millisats) for their routing and sorting services.
- **Creator Support:** "Following" a creator becomes an act of economic investment and infrastructure support, bypassing the need for extractive advertising models.
*** Decentralized Moderation (Stackable Labelers)
Moderation is treated as "Competitive Labeling." Users subscribe to multiple Labelers (AI agents, NGOs, fact-checkers) to create a composite moderation profile tailored to their values.
** Social Governance & Moderation
*** Multi-layered Moderation
1. *Individual:* Publisher controls their own content and PDS.
2. *Community (Social Governance):* Collective rules enforced via governance modules (GEM).
- *Global Blocklists:* Communities can vote on shared moderation policies. If a quorum (e.g., 70% of an NGO's members) flags a specific DID as a "Spam Bot," that DID is automatically added to a Global Blocklist and hidden from all participating members' feeds.
- *Curated Feeds:* A community can vote to "Pin" certain content creators to a shared "Featured" feed, effectively acting as a decentralized editorial board.
3. *Algorithm:* User-chosen filtering and sorting via PFG and Competitive Labeling.
4. *Network:* Protocol-level consensus for universally illegal content (e.g., CSAM).
** Related Documents
- [[file:agora-requirements-06-exchange-and-contracts.org][06: Exchange and Contracts]] - Economic layer and human connection formalization.
- [[file:agora-requirements-02-identity.org][02: Identity]] - Personas and Master Keys.
- [[file:agora-requirements-03-infrastructure.org][03: Infrastructure]] - PDS and Relays.

306
projects/agora/agora-requirements-06-exchange-and-contracts.org Normal file
View File

@@ -0,0 +1,306 @@
#+TITLE: Agora Requirements - 06: Exchange
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:28]
#+DATE: 2026-03-15
#+ID: agora-requirements-06-exchange
#+STARTUP: content
* Exchange
** Concept
The Exchange layer provides the economic substrate of Agora: value transfer via the Lightning Network, multi-currency support, and payment primitives. Built on top of Content Objects (see [[file:agora-requirements-04-the-primitive.org][The Primitive]]) and Social relationships (see [[file:agora-requirements-05-social.org][Social]]).
** Lightning Native
** Base Currency
Lightning Network (Bitcoin L2) is the default payment rail:
- *Minimum:* 1 satoshi
- *Maximum:* Channel capacity limited
- *Speed:* Near-instant for settled payments
- *Cost:* Fraction of a cent per transaction
** Payment Types
**** One-shot Payment
- Single payment for content or service
- Invoice generated, payment fulfilled, preimage reveals
**** Streaming Payment
- Continuous micropayments (per-second, per-action)
- Used for subscriptions, metered services
- Automatic via HTLC stream
**** Hodl Invoice
- Escrow with hash-locked release
- Payment committed but conditional on secret
**** Keysend
- Spontaneous payment without invoice
- Used for tips, donations
** Lightning Node Architecture
The specification currently lacks explicit guidance on how end users run Lightning nodes. Below are the three architectural options under consideration:
**** Option 1: Embedded LDK Node (Self-Sovereign)
- Each user's client (desktop/mobile) runs an embedded Lightning node using LDK (Lightning Dev Kit)
- User has full custody of keys; channels are mobile-friendly (LSP-managed)
- PDS handles the always-online requirement since user devices aren't
- Aligns with Agora's "sovereign" philosophy but requires technical sophistication
**** Option 2: LSP (Lightning Service Provider) Model
- User connects to an LSP that provides inbound liquidity and accepts payments on their behalf
- User still has signing keys locally; LSP manages channels and uptime
- User can switch LSPs without losing funds (Lightning hygiene)
- Most realistic for mobile users; PDS providers may bundle LSP services
**** Option 3: Custodial Bridge (On-ramp)
- PDS offers built-in custodial Lightning wallet for users who don't want self-custody
- Users can withdraw to self-custody later (exit guarantee)
- Default for new users, opt-in for sovereignty
- Trade-off: convenience vs. full sovereignty
**** Key Distinction: Custody vs. Hosting
Non-custodial means *you* control the private keys, not where the software runs. Even if the Lightning node runs on hosted PDS infrastructure, it can still be *your* node with *your* keys:
- Your PDS gets encrypted data (content)
- Your Lightning node gets encrypted state (channel backups)
- Both are encrypted to *your* keys
- The PDS provider cannot sign transactions or spend your funds
**** GAP: Decision Required
The specification has not yet decided between:
- Requiring all users to run embedded nodes (sovereign, high technical barrier)
- Defaulting to LSP connections (practical, retains key custody)
- Offering custodial as default with opt-out (maximum adoption, sovereignty trade-off)
*Next Step:* Evaluate technical feasibility of LSP integration with PDS providers and document recommended architecture for V1.0.
** Multi-Currency Support
** Supported Currencies
- *Lightning (default):* For micro-payments (<$1)
- *On-chain Bitcoin:* For settlements, channel opens
- *Stablecoins (RGB):* USDT/USDC on Bitcoin L2
** Currency Routing
- Client specifies desired currency
- PDS may support conversion
- Exchange rates oracle-attested
** Concept
The Agora protocol must support multiple currencies beyond Lightning-native satoshis to facilitate broader economic participation and provide stability options. While Lightning remains the primary rail for micro-payments, other assets will be integrated for larger transactions and specific use cases.
** Supported Currencies
**** Lightning Network (L2 Bitcoin)
- *Role:* Primary for all micro-payments (typically <$10).
- *Mechanism:* BOLT-compatible invoices, streaming payments, Keysend.
**** On-chain Bitcoin
- *Role:* For large settlements, channel opens/closes, long-term value storage.
- *Mechanism:* Standard Bitcoin transactions, multi-sig escrow.
**** Stablecoins
- *Role:* For price stability, high-volume transactions, fiat-pegged value.
- *Mechanism:* RGB protocol on Bitcoin (future), wrapped assets on compatible L2s, or direct integration with atomic swaps.
** Currency Routing & Conversion
**** Client-Side Preference
- Users specify preferred payment currencies for sending and receiving.
- Clients automatically attempt conversion if sender's and receiver's preferred currencies differ.
**** PDS/Relay Support
- PDS nodes MAY offer automated currency conversion services (e.g., satoshis to stablecoins).
- Fees for conversion MUST be transparent and competitive.
- Conversion services MUST be auditable (using attestations).
**** Exchange Rate Verification (Oracle)
- The system MUST use a decentralized oracle network to attest to current exchange rates.
- Exchange rate attestations are signed Content Objects.
- Clients verify oracle signatures and rate validity before conversion.
** Integration with Contracts
- Contracts (e.g., Sale, Service) MUST specify accepted currencies.
- Prices in contracts MUST be expressed in a base unit (e.g., satoshis) with optional equivalent in other currencies.
- Exchange rates for contract execution MUST be based on oracle attestations at time of execution.
** Economic Primitives
** Invoice
- BOLT-11 compliant
- Amount, memo, expiry
- Static (LNURL) or dynamic
** Payment
- Preimage proof of settlement
- Content-addressed for audit trail
- Refundable if escrowed
** Account
- DID-linked balance tracking
- Multi-currency support
- Reconciliation with on-chain
** Fee Structure
** Relay Fees
- Per-message routing (configurable)
- Subscription-based access
- Priority delivery premium
** PDS Fees
- Storage: per-GB per month
- Bandwidth: per-request or per-GB
- Compute: for AI, indexing
** Marketplace Fees
- Owner-defined (0-30%)
- Universal Open Market: minimal (relay costs)
** Exchange Primitives
** Escrow
Hold funds until conditions met:
- 2-of-3 multisig (buyer, seller, arbitrator)
- HTLC hash-time-locked contracts
- Smart contract on compatible L2
** Subscription
Ongoing economic relationship:
- Streaming Lightning payments
- Permissioned content access
- Automatic key provision
** Bounty
Payment for task completion:
- Escrowed funds
- Completion attestation
- Oracle verification option
** Sovereign Contract & Arbitration Layer (SCAL)
To enable Personas to execute binding agreements with decentralized dispute resolution, Agora implements SCAL. A contract in this system is not a static PDF; it is an executable cryptographic object.
*** 1. The Ricardian Contract Module
Agora contracts follow the Ricardian model, ensuring they are both human-readable and machine-executable.
- *Natural Language (The Markdown):* The human-readable terms of the agreement (e.g., "Person A delivers 100 bricks to Person B by Friday").
- *Machine Logic (The JSON-LD):* The executable parameters embedded in the Note's metadata (e.g., `due_date: 2026-01-16`, `price_sats: 50000`, `arbitrator_did: did:key:xyz`).
- *The Merkle Link:* Both parts are hashed together into a single Content Identifier (CID). If a single comma is changed in the text, the hash changes, breaking the digital contract. This ensures the "Code" and the "Law" remain identical.
*** 2. Payment & Escrow: The "HODL Invoice"
For service delivery and physical goods, Agora relies on Lightning HODL Invoices as a trustless escrow, removing the need for a custodial middleman.
- *Commitment:* The Buyer "pays" the invoice. The funds leave their Lightning wallet but remain cryptographically locked in the network routing nodes.
- *The Proof:* The Seller observes the network state, sees the funds are "Locked," and confidently delivers the goods or services.
- *Settlement:* Once the Buyer confirms receipt, they release the cryptographic Preimage (the key). The money instantly settles to the Seller.
- *Dispute:* If a problem arises, the funds stay locked. An agreed-upon Arbitrator intervenes, eventually providing the key to either the Buyer (triggering a Refund) or the Seller (forcing a Payout).
- *Timeout Logic:* Contracts MUST include a `CLTV-expiry` (CheckLockTimeVerify). If the arbitrator does not rule within a predefined window (e.g., 30 days), the funds are automatically returned to the Buyer to prevent "Forever-Locks."
*** 3. Proof-of-Delivery (Oracles)
To automate the release of HODL invoices without manual buyer intervention, SCAL supports cryptographic Proof-of-Delivery.
- *Physical Goods:* Support for "Scanning a QR code" upon physical delivery, which automatically signs the release transaction and broadcasts the Preimage.
- *Digital Goods:* Support for Zero-Knowledge Proofs (ZKP). The payment is released automatically once the client cryptographically verifies that the received file hash matches the contracted payload.
*** 4. Multi-Level Arbitration & The Ricardian Evidence Vault
To address disputes without a central state, contracts reference a tiered system of human judgment (The "Circles" Model). As detailed in the [[file:agora-requirements-10-governance-and-assets.org][Governance]] specifications, this involves escalating from Local Elders to specialized Guilds, and finally to Global Juries.
- **Web of Trust (WoT) Level 1:** Arbitrators at Level 1 are selected based on Transitive Trust (e.g., the system finds a mutual connection trusted by both parties within 3 degrees of separation).
- **Ricardian Evidence Vault:** During a dispute, parties upload encrypted "Evidence Blobs" to their PDS. Using Zero-Knowledge Proofs (ZKPs) or Shared Keys, they grant the current level of arbitrators temporary read-access to the evidence without making it public.
- **Real-Time Adjudication:** If live hearings are required, the system MUST support VoIP/WebRTC signaling conducted over an authenticated DIDComm v2 channel, utilizing "blind" Community TURN servers if direct P2P fails.
- **Audit Trail:** Every ruling, appeal, and evidence hash is permanently stored in the Key Event Log (KEL) for that specific contract, creating a verifiable record of the "trial."
*** 5. Enforcement: Social vs. Financial
In weak rule-of-law environments, the system relies on two "sticks" to ensure contract compliance without physical police forces:
- *Financial Collateral:* High-risk contracts can require both parties to lock "Safety Deposits" in a 2-of-3 multisig before the contract begins. If a party defects, they forfeit their deposit.
- *Reputation Slashing (Social Enforcement):* If a Persona loses an arbitration and refuses to comply, their DID is cryptographically "Flagged" across the public network. Because DIDs are persistent and tied to social graphs, they cannot simply delete their account to escape the penalty. Their "Trust Score" drops to zero, effectively cutting them off from future trade, employment, or community participation.
** Integration with Content Objects
Economic actions are specialized Notes containing structured `contract` metadata:
- *Invoice:* Contract offer Note containing payment terms (`price_satoshis`, `bolt11`).
- *Payment:* Contract fulfillment Note (`Take`) containing cryptographic proof (`preimage`).
- *Escrow:* Contract state Note referencing a multi-signature threshold or conditional logic.
- *Subscription:* Ongoing contract Note with streaming parameters or recurring billing cycles.
Transactions reference the Content Objects they interact with:
- Payment Note `reply_to` the Invoice Note being fulfilled.
- Subscription Note `references` the Feed CID it provides access to.
- Bounty Note (Contract) `references` the Task description.
** Content Monetization & Seeder Rewards
To monetize high-bandwidth content (like video or software) in a decentralized, permissionless network, Agora utilizes a combination of Split-State Encryption, the LSAT protocol, and granular Lightning network routing. This ensures creators get paid without relying on centralized DRM or hosting providers.
*** 1. The Encrypted Swarm (Blind CDN)
If you want to charge for a video, you cannot send the raw file into the P2P swarm. If you did, the first "seeder" would simply share the unencrypted version for free.
- **The Locked Box:** The creator encrypts the video with a unique Symmetric Key.
- **The Split Structure:** The Note's `contract` field is Public (listing the price, title, and terms), but the `payload` field is a CID pointing to the encrypted video chunks.
- **Blind Replication:** Followers and network participants host and seed this encrypted `payload`. They act as a "Blind CDN" (Content Delivery Network)—hosting a file they cannot see.
*** 2. The LSAT Protocol (The Smart Ticket)
To automate the purchase and unlocking of this content, Agora uses LSATs (Lightning Service Authentication Tokens).
- **The 402 Challenge:** When a viewer clicks "Play," their client attempts to fetch the payload. The PDS responds with an HTTP 402 (Payment Required) error, containing a Lightning Invoice (generated based on the `contract` terms) and a "Macaroon" (a digital ticket).
- **The Unlock:** Once the user pays the invoice (e.g., 100 sats), they receive a cryptographic Preimage (proof of payment). They send this Preimage back to the PDS.
- **The Result:** The PDS validates the proof and releases the Decryption Key. The video decodes instantly on the client's device. The data may have been downloaded from a friend's PDS (the swarm), but the permission to view it was purchased securely from the creator.
*** 3. Incentivizing the Seeders (Paid Seeding)
One of Agora's most innovative features is "Seeder Micro-Rewards." If a follower provides the bandwidth that allows a creator's content to go viral, the network can programmatically share the revenue.
- **The Split Payment:** The Note's `contract` can define a Lightning routing split. When the 100 sats are paid via the LSAT, the network routes the funds accordingly:
- **90 sats** go to the Creator.
- **5 sats** go to the Indexing Relay.
- **5 sats** go to the Seeder (the specific follower who provided the data bits).
- **The Economic Shift:** "Following" an NGO or an indie creator becomes a way to earn a tiny amount of Bitcoin while supporting their mission. The better the content you seed, the more "tips" your server collects for providing the bandwidth.
*** Physical Collateralization
In environments with weak state enforcement, Agora enables the use of physical assets as cryptographically-secured collateral via the PAL (Physical Asset Linking) protocol.
- **The Pledge:** A user links a Digital Twin token (representing a physical asset like a car or machine) to a Civil Contract Note.
- **The Lock:** The contract's logic "freezes" the Digital Twin token. While the user maintains physical possession of the asset, they are cryptographically barred from selling or transferring the digital title until the contract obligations (e.g., a loan repayment) are met.
- **Enforcement:** Severe defaults can trigger the "IoT Stick" (see [[file:agora-requirements-07-advanced-integration.org][Advanced Integration]]), where an IoT-enabled smart lock physically disables the asset based on an Arbitration (HDR) ruling.
** Advanced Exchange Features
** Cross-Chain Swaps
**** Atomic Swaps Architecture
Agora enables seamless value transfer between Bitcoin and other blockchains without relying on centralized exchanges.
- *HTLC Contracts:* Hash Time-Locked Contracts (HTLCs) are used to lock assets on both chains simultaneously.
- *Swap Personas:* Specialized Personas (Market Makers) provide liquidity and act as counterparties for atomic swaps, competing on fees and speed.
- *Protocol Integration:* A `CrossChainSwap` Content Object defines the terms (rate, chains, timelock). Once agreed, both parties publish the HTLCs on their respective chains. The revelation of the preimage on one chain allows claiming the funds on the other.
** Stablecoin Integration
**** RGB Protocol Specification
Stablecoins (e.g., USDT, USDC) are supported natively as Layer 2 assets on top of Bitcoin/Lightning using the RGB protocol.
- *Asset Issuance:* Stablecoin issuers maintain a Genesis Contract on Agora defining the asset's RGB schema and initial supply.
- *Client Support:* Agora clients MUST integrate an RGB node alongside their Lightning node to parse client-side validated state transitions.
- *Payment Routing:* RGB assets are routed over standard Lightning channels. Clients construct invoices that specifically request the RGB stablecoin asset ID instead of raw satoshis.
- *PDS Storage:* The client-side validation data (consignment) for RGB assets is stored as encrypted Content Objects in the user's PDS, ensuring the user maintains full custody of the asset's history.
** Subscription Management
**** Complex Recurring Billing Logic
Agora handles recurring payments natively without centralized payment processors.
- *Subscription Objects:* A `SubscriptionContract` defines the terms: amount, currency, billing cycle (e.g., monthly, weekly), and grace period.
- *Streaming vs. Discrete Billing:*
- For continuous services (e.g., Relay access), streaming payments (sats/second) are preferred.
- For discrete access (e.g., monthly newsletter), the client software automatically generates a local cron job to pay the creator's static LNURL-pay endpoint at the start of each billing cycle.
- *Grace Periods & Revocation:* If a recurring payment fails (due to offline client or insufficient funds), the provider's PDS logs a `PaymentFailed` event. The subscriber is granted a predefined grace period (e.g., 3 days). If unresolved, the provider's PDS automatically revokes the decryption keys for the subscribed content.
** Related Documents
- [[file:agora-requirements-04-the-primitive.org][The Primitive]] - Content Object structure
- [[file:agora-requirements-05-social.org][Social]] - Connection types for economic relationships
- [[file:agora-requirements-02-identity.org][Identity]] - Contracts and attestations
** Gaps
- *None.* All identified gaps in the exchange layer have been resolved.

465
projects/agora/agora-requirements-07-advanced-integration.org Normal file
View File

@@ -0,0 +1,465 @@
#+TITLE: Agora Requirements - 07: Advanced Integration
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:28]
#+DATE: 2026-03-15
#+ID: agora-requirements-06-advanced-integration
#+STARTUP: content
* Advanced Integration
** AI Integration
*** Overview
Agora enables AI at multiple layers: as sovereign actors, personal assistants, algorithms, and collaborative agents. All AI interactions are economically mediated via Lightning and respect user data sovereignty.
*** AI Personas (Sovereign AI Actors)
**** Identity and Verification
- AI models MUST be instantiated as AI Personas with their own DIDs (e.g., `did:ai:openai:gpt-4o`, `did:ai:local:llama3`).
- AI Personas MUST cryptographically sign their outputs, allowing users to verify the model and its provenance.
- AI Persona metadata MUST include: model architecture, training date, capabilities, and trust assumptions.
**** Economic Model
- The system MUST support micro-payments via Lightning Network for AI queries.
- Pay-per-query: Users pay only for what they use (e.g., 0.1-10 satoshis per query).
- No subscriptions required for casual use.
- AI providers set their own pricing; market competition drives efficiency.
**** Execution Tiers & Compute Swarms
- *Tier 1 (On-Device):* Models run locally using WebNN or local NPU/GPU. Zero privacy leak, no query fees, hardware-limited.
- *Tier 2 (Cloud):* Access to state-of-the-art models. Queries encrypted with X25519. Provider sees query but not user identity if anonymous persona used.
- *Tier 3 (Compute Swarms):* Decentralized P2P AI Marketplace. For heavy tasks (e.g., generating 4K video or training a guild-wide model), the network taps into the spare GPU power of the community. Nodes that provide "Compute" are rewarded with sats.
**** Plug-and-Play Inference
To support Tier 1 and localized Community processing, the PDS MUST include a standard Inference Proxy API.
- *Local Execution:* When a user selects a "Smart Filter," the PDS can route the data through a local Ollama instance or a community-run vLLM node instead of a centralized provider.
- *Prompt Transparency:* The "System Prompt" for every public AI algorithm (e.g., a Feed Generator or Moderation Labeler) MUST be public and verifiable. If an NGO claims their sorting algorithm is "unbiased," the community can inspect the actual instruction weights and prompt text.
*** AI Sub-Agents (Personal Assistants)
**** Concept
AI Sub-Agents are personal AI assistants that act on behalf of the user, operating from the user's PDS with full access to the user's content and context.
**** Requirements
- Sub-Agents MUST run within the user's PDS or on their sovereign client (local-first).
- Sub-Agents MUST have access to user's Content Objects via the PDS API (with user authorization).
- Sub-Agents MUST be able to perform actions as the user: post content, send messages, manage tasks, schedule events.
- All Sub-Agent actions MUST be logged and auditable by the user.
- Sub-Agents MUST operate within user-defined constraints (budget limits, action permissions, time windows).
**** Sub-Agent Capabilities
- *Content Management:* Organize, tag, and archive user's content.
- *Communication Management:* Filter and prioritize messages; draft responses for user approval.
- *Discovery:* Proactively surface relevant content from the social graph.
- *Personalization:* Learn user preferences to improve recommendations.
**** Economic Integration
- Sub-Agents can invoke paid AI Personas on user's behalf (with spending limits).
- Micro-payments for external AI services are tracked and reported.
*** AI Algorithms (Content Curation and Moderation)
**** Concept
AI algorithms that process content for curation, moderation, sorting, and ranking. These run locally on the client as sovereign code.
**** Algorithm Marketplace
- The system MUST support a marketplace of open-source "Sorting Algorithms" for feed display.
- Algorithms MUST run locally on the client or as trusted services in the user's PDS.
- Algorithms MUST be content-addressed (CID) for integrity verification.
- Algorithm developers can monetize via licensing fees (Lightning).
**** Curation Algorithms
- *Feed Ranking:* Sort posts by relevance, recency, engagement, or custom criteria.
- *Content Filtering:* Filter out spam, low-quality content, or topics user wants to avoid.
- *Summarization:* Generate summaries of long posts or threads.
- *Personalization:* Learn from user behavior (locally, without data exfiltration).
**** Moderation Algorithms
- *Spam Detection:* ML models to detect and flag spam patterns.
- *Toxicity Scoring:* Local sentiment analysis for content warning labels.
- *Authenticity Scoring:* Detect potential misinformation or manipulation.
- All moderation actions are local to the user; no centralized censorship.
**** Search and Discovery
- *Intelligent Search:* Natural language queries over user's indexed content.
- *Discoverability Scoring:* Rank new personas/content by predicted relevance.
- *Trend Detection:* Identify emerging topics in user's extended network.
*** AI-to-AI Communication
**** Concept
AI Personas and Sub-Agents can communicate with each other to solve complex tasks, negotiate services, or coordinate actions.
**** Distributed Reputation Oracles
AI Personas can operate as specialized reputation oracles and adjudicators within the Governance layer:
- *Tier 0 Arbitrator:* Before a human enters the Judicial process, a local AI analyzes the evidence and provides a "Sanity Check" or "Likely Outcome" report, saving time and human capital.
- *Automated Labeling:* AI agents can act as high-speed "Labelers" (see Social Moderation), tagging millions of posts for quality, spam, or sentiment, which users can then choose to route their feed through or ignore.
**** Requirements
- AI Personas MUST be able to query other AI Personas via standard Agora messaging.
- AI-to-AI communication MUST use the same Content Object primitives as human communication.
- AI Personas MUST be able to negotiate service terms (price, scope, timeline) via smart contracts.
- AI-to-AI transactions MUST be economically settled via Lightning.
**** Use Cases
- *AI Researcher → AI Coder:* Researcher queries literature; Coder implements findings.
- *AI Moderator → Human Curator:* AI flags content; human curator reviews and decides.
- *AI Translator → AI Summarizer:* Translate foreign content, then summarize for user.
- *Oracle Network Coordination:* Multiple Validator Oracles coordinate testing and attestation.
*** Data Sovereignty and Consent
**** Model Training & Federated Learning
- AI providers MUST NOT train on user content without explicit Consent Contract.
- Users MUST be able to revoke training consent at any time.
- Training data contributions MUST be economically compensated (Lightning).
- *Privacy-Preserving Training (Federated Learning):* The system MUST support Federated Learning. Collectives (e.g., an NGO) can train custom models on members' data without ever seeing the raw data. Member devices compute weight "updates" locally, which are then aggregated into a new model version.
**** Context Control
- Users MUST be able to provide "Context CIDs" to limit AI access to specific data.
- Sub-Agents MUST respect PDS access controls and encryption boundaries.
- All AI processing of sensitive data SHOULD prefer on-device (Tier 1) execution.
**** Auditability
- All AI queries and responses MUST be logged as Content Objects (optional, user-configurable).
- Users MUST be able to inspect what data AI Sub-Agents accessed and what actions they took.
** Physical World Integration
*** IoT & Device Management
- The system MUST instantiate physical entities (events, locations) as Collective Personas (DIDs).
- Users MUST be able to publish signed Proof-of-Presence Objects.
- Every smart device MUST be a persona under the control of the user's master key.
- Devices MUST communicate using the standard Agora protocol with Consent Contracts.
- Sensor data MUST be published as encrypted Content Objects.
- Users MUST be able to sell signed sensor data to Data Collector Personas.
*** Physical-Digital Bridging
- *QR Codes:* Personas and CIDs can be easily shared in the physical world via QR codes. Scanning a "Place QR" initiates a Consent Contract to join the location's collective.
- *Physical Keys:* Hardware-backed personas can be used as digital keys for physical locks (e.g., using NFC or BLE).
*** On-Device AI Limitations
**** Performance Constraints
- *Model Size Limits:* On-device models MUST be optimized for size (typically <5GB for mobile, <500MB for low-end devices).
- *Inference Latency:* Target <100ms for simple queries, <2s for complex generation tasks.
- *Memory Footprint:* Runtime memory SHOULD NOT exceed 2GB on mobile devices
- *CPU/GPU Utilization:* Models MUST throttle to prevent device overheating and battery drain.
**** Hardware Classification
The system MUST define hardware tiers for on-device AI:
| Tier | Devices | Capable Models | Example |
|------|---------|----------------|-----------|
| Tier A | Flagship smartphones/laptops | LLMs up to 7B params, full multimodal | iPhone 15 Pro, M3 MacBook |
| Tier B | Mid-range smartphones | Small LLMs (3B), vision models | Pixel 7, iPhone 14 |
| Tier C | Low-end/older devices | Tiny LLMs (<1B), embeddings only | iPhone SE, budget Android |
| Tier D | Embedded/IoT | Embeddings, classification | Raspberry Pi 4, IoT sensors |
**** Battery Impact Mitigation
- *Adaptive Scheduling:* AI inference MUST respect system power states (defer when low battery).
- *Thermal Throttling:* Reduce model complexity or batch size if device temperature >45°C.
- *Background Processing:* Background AI tasks (indexing, summarization) ONLY during charging.
- *User Controls:* Granular settings for AI battery usage per Sub-Agent.
**** Model Size Limits by Tier
| Hardware Tier | Max Model Size | Context Window |
|---------------|----------------|----------------|
| Tier A | 7B parameters | 8K-32K tokens |
| Tier B | 3B parameters | 4K tokens |
| Tier C | 1B parameters | 2K tokens |
| Tier D | 500M parameters | 1K tokens |
**** Fallback Mechanisms
- If on-device model fails or is unavailable, system MUST gracefully degrade:
1. Attempt smaller quantized version of same model
2. Route to user's PDS-hosted inference (if available)
3. Offer encrypted cloud query (Tier 2) with user consent
4. Queue request for later on-device processing
*** Privacy Trade-offs Clarification
**** UX Design for AI Privacy Choices
The system MUST provide clear, user-friendly visualization of privacy trade-offs:
**** Tier 1 (On-Device) Indicators
- *Privacy Badge:* Green shield icon indicating "Process locally — data never leaves device"
- *Capability Badge:* Shows model capabilities (e.g., "7B params — answers, summaries, code")
- *Limitation Notice:* Clear disclosure of model limitations vs cloud alternatives
- *Cost Display:* "Free — no micro-payment required"
**** Tier 2 (Cloud) Indicators
- *Privacy Warning:* Yellow alert icon: "Query sent to [Provider] — provider can see requests"
- *Anonymity Shield:* Optional ghost icon: "Anonymous persona — provider cannot link to your identity"
- *Capability Badge:* "State-of-art — unlimited context, multimodal, real-time"
- *Cost Display:* Live satoshi counter: "~15 satoshis per query"
**** Comparative Display
When user is choosing between Tier 1 and Tier 2:
```
┌─────────────────┬─────────────────┐
│ On-Device AI │ Cloud AI │
├─────────────────┼─────────────────┤
│ ✅ Private │ ⚠️ Provider sees│
│ ✅ Zero cost │ ⚡ Pay per query │
│ ⚡ Limited power│ ✅ Unlimited │
│ 📱 Device only │ 🔒 Anonymous OK │
└─────────────────┴─────────────────┘
```
**** Consent Flow for Cloud AI
1. *First Use:* Explicit consent required: "Allow queries to [Provider]?"
2. *Spending Limit:* User MUST set Lightning budget cap before first use.
3. *Per-Query Confirmation:* Optional setting for expensive queries (>100 satoshis).
4. *Revocation:* One-tap disable cloud AI, return to on-device only.
*** Proof-of-Presence Cryptography
**** Concept
Cryptographic attestation that a user's Persona was physically present at a specific geographic location and time, without revealing continuous location history.
**** Proof Generation
```typescript
interface ProofOfPresence {
// Location data (coarse granularity for privacy)
locationHash: string; // hash(lat, lng) truncated to 100m grid
locationZone: string; // Human-readable zone name (e.g., "Downtown NYC")
// Time attestation
timestamp: number; // Unix timestamp (hour granularity)
timeWindow: number; // Validity window (e.g., ±30 minutes)
// Cryptographic proof
witnessDIDs: string[]; // Nearby personas/devices that co-signed
beaconSignatures: string[]; // Signatures from location beacons (BLE/WiFi)
// Persona attestation
personaDID: string;
signature: string; // Signed {locationHash, timestamp, witnessDIDs}
}
```
**** Verification Process
1. *Proximity Witnesses:* At least 3 nearby Personas must co-sign (K-anonymity set)
2. *Beacon Verification:* Location beacon (collective persona) signs timestamp
3. *Time Sync:* All signatures MUST be within 5-minute tolerance
4. *Revocation:* Cannot be revoked — historical proof permanent
**** Privacy Properties
- *Coarse Location:* 100m grid precision, not GPS exact coordinates
- *Temporal Decay:* Proofs expire after 24 hours (useful for ephemeral access)
- *No Tracking:* Individual location history NOT stored — only specific presence proofs
- *Selective Disclosure:* User reveals only specific proofs, not full location data
**** Use Cases
- *Event Access:* "Prove I was at the conference" for post-event content access
- *Location-Based Collectives:* Join a venue's collective by proving presence
- *Gaming:* Geocaching, location-based achievements
- *Governance:* "Only people who attended the town hall can vote"
*** D2D Command Authorization
**** Concept
Device-to-device (D2D) commands allow smart devices to request actions from other devices or the user's Persona. These MUST be authorized via cryptographically-signed Consent Contracts.
**** Consent Contract Structure
```typescript
interface D2DConsentContract {
// Contract parties
devicePersonaDID: string; // e.g., smart thermostat
ownerPersonaDID: string; // User's main persona
// Scope of authorization
commands: {
command: string; // e.g., "set_temperature"
parameters: { // Valid parameter ranges
[param: string]: {
type: 'number' | 'string' | 'boolean';
min?: number;
max?: number;
allowedValues?: string[];
}
}
}[];
// Constraints
timeRestrictions?: {
allowedHours: [number, number]; // e.g., [9, 17] for 9am-5pm
timezone: string;
};
rateLimit?: number; // Max commands per hour
// Emergency override
emergencyContacts?: string[]; // DIDs that can bypass restrictions
// Signatures
deviceSignature: string;
ownerSignature: string;
expiresAt: number;
}
```
**** Command Flow
1. *Request:* Device sends signed command request to user's client
2. *Validation:* Client checks Consent Contract for authorization
3. *Confirmation:* For sensitive commands, require user confirmation UI
4. *Execution:* User's client executes command, returns signed receipt
5. *Logging:* All D2D commands logged as Content Objects for audit
**** Revocation
- Owner can revoke Consent Contract at any time
- Revocation broadcast via Relays, cached by devices
- Devices MUST stop accepting commands from revoked contracts within 60 seconds
*** Sensor Data Encryption
**** Concept
Continuous sensor data (IoT devices, wearables) MUST be encrypted with automatic key rotation to prevent long-term key compromise.
**** Encryption Methods
**** Method 1: Per-Record Encryption
- Each sensor reading encapsulated as Content Object
- Encryption: AES-256-GCM with ephemeral keys
- Key derivation: X25519 ECDH between sensor and owner's Persona
- Metadata: Timestamp, sensor type, data type, encrypted payload
**** Method 2: Stream Encryption (for high-frequency data)
- Establish long-term X25519 keypair for sensor
- Derive session keys via HKDF-SHA256
- Rotate session key every 10,000 records or 24 hours (whichever comes first)
- Use ChaCha20-Poly1305 for stream encryption (faster than AES for bulk)
**** Key Rotation Protocol
```typescript
interface KeyRotation {
oldPublicKey: string; // X25519 public key being retired
newPublicKey: string; // New X25519 public key
rotationTimestamp: number;
previousKeySignature: string; // Signature proving chain of custody
deviceDID: string;
}
```
**** Data Lifecycle
1. *Collection:* Sensor encrypts data locally, never stores plaintext
2. *Transmission:* Encrypted Content Objects sent to owner's PDS
3. *Storage:* PDS stores ciphertext only
4. *Access:* Owner decrypts on-demand; can share via new encryption to specific parties
5. *Expiration:* Configurable TTL after which PDS can garbage collect
**** Implementation Requirements
- Sensor firmware MUST support hardware-backed key generation (HSM/TEE)
- Key material MUST be protected in Secure Enclave or TPM
- Rotation events MUST be logged for audit
- Compromised keys MUST trigger automatic rotation within 5 minutes
*** Hardware-Backed Contract Enforcement (The "IoT Stick")
For high-stakes physical assets (e.g., tractors, factory machinery, or smart-lock-equipped real estate), Agora supports hardware-level enforcement of contract obligations.
- **Binding IoT to Contract:** A physical asset's IoT sensor or "Smart Lock" is cryptographically bound to a specific Civil Contract Note.
- **Enforcement Signal:** The machine's firmware is configured to listen for signed state updates from the contract's designated Arbitration (HDR) module.
- **Default Action:** If the HDR module rules that a user has defaulted on a payment or violated the contract terms, it publishes a signed "Disable" event.
- **Physical Lockout:** Upon receiving the verified signal, the machine's IoT controller automatically disables operation (e.g., preventing the engine from starting or locking the facility) until a subsequent "Release" event is published following debt settlement or compliance.
- **Privacy & Safety:** The system MUST include an "Emergency Override" mechanism for life-safety situations, which triggers a high-severity notification to all contract parties and designated emergency contacts.
*** Physical Key Protocol
**** Concept
Hardware-backed Persona keys used for physical access control (locks, vehicle access, secure facilities) via NFC or BLE.
**** Protocol Stack
| Layer | Technology |
|-------|------------|
| Physical | NFC (ISO 14443) or BLE 5.0+ |
| Authentication | Challenge-response with Ed25519 signatures |
| Transport | Encrypted session keys (X25519 ECDH) |
| Application | Lock state management via Consent Contracts |
**** Authentication Flow
1. *Tap:* User taps device (phone, smart card, wearable) to NFC reader or establishes BLE connection
2. *Challenge:* Lock generates random 256-bit challenge + timestamp + lock ID
3. *Response:* Device signs challenge with Persona's private key
4. *Verification:* Lock checks signature against registered Persona DIDs
5. *Authorization:* Lock queries Consent Contract for access permissions (time, allowed actions)
6. *Grant/Deny:* Lock opens or rejects based on authorization
**** Consent Contract for Physical Access
```typescript
interface PhysicalAccessContract {
lockDID: string; // DID of the physical lock
authorizedPersona: string; // DID of key holder
schedule: {
daysOfWeek: number[]; // 0-6 (Sunday-Saturday)
startTime: string; // HH:mm format
endTime: string; // HH:mm format
timezone: string;
};
accessRules: {
maxUsesPerDay?: number;
consecutiveDelay?: number; // Minimum seconds between accesses
requiresCompanion?: boolean; // Requires another authorized person present
};
emergencyOverride: {
enabled: boolean;
emergencyContact: string; // DID to notify on emergency override
};
signatures: {
owner: string; // Lock owner signature
authorized: string; // Key holder signature
};
}
```
**** Hardware Requirements
- *Secure Element:* Physical key MUST store Ed25519 private key in tamper-resistant hardware (Secure Enclave, TPM, or smart card)
- *NFC/BLE:* Support for standard proximity protocols
- *Offline Capability:* Can authenticate without internet (lock caches authorized DIDs)
- *Revocation:* Lock MUST check revocation list daily for compromised keys
**** Security Properties
- *Non-clonable:* Keys cryptographically bound to Persona's Master Key
- *Ephemeral:* Session keys for each unlock event, not reusable
- *Auditable:* Every access logged as Content Object
- *Recoverable:* Lost physical key can be revoked without changing lock
** Related Documents
- [[id:agora-ai-integration][Agora AI Personas & Privacy]]
- [[id:agora-physical-iot][Agora Physical World & IoT]]

116
projects/agora/agora-requirements-08-library.org Normal file
View File

@@ -0,0 +1,116 @@
#+TITLE: Agora Requirements - 08: Library
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:28]
#+DATE: 2026-03-14
#+ID: agora-requirements-07-library
#+STARTUP: content
* Library
** Concept
The Library is a unified content archiving and media management system. It works like a unified *arr suite (Sonarr, Radarr, Readarr, etc.) that builds your personal libraries across all content types.
** Supported Content Types
- Video (movies, TV shows, educational content)
- Audio (podcasts, music, audiobooks)
- Photos (personal albums, professional portfolios)
- Text (books, articles, documents)
- Maps (geographic data, custom itineraries)
- Physibles (physical object designs, 3D models)
- Manufacturing Processes (recipes, procedures, blueprints)
** Architecture
The Library consists of three core components:
*** Downloaders
- Content acquisition tools that fetch media from various sources
- Support for torrents, Usenet, direct downloads, and IPFS
- Integration with content discovery networks
- Automated quality selection and format conversion
- Metadata fetching from external databases
*** Indexers
- Local search and categorization of library content
- Full-text search across documents, subtitles, metadata
- Tag-based organization (genre, year, creator, etc.)
- Content deduplication via CID comparison
- Integration with Agora's discovery layer for shared content
*** Library Managers
- Content organization and presentation interfaces
- Unified browsing across all content types
- Playlist and collection creation
- Offline sync for mobile clients
- Sharing controls (personal, collective, public)
** Content Addressing
All Library content is stored as CIDs:
- Original files content-addressed for integrity
- Metadata stored as separate Content Objects
- Thumbnails and previews generated and addressed separately
- Version history maintained via CID chains
** Archiving
*** Concept
Archiving preserves Content Objects and open web content for long-term access, creating personal or collective knowledge repositories that outlive the ephemeral nature of streams.
*** CID Content Archiving
**** Personal Archives
- Users can archive any CID-based content they have access to (public or decrypted)
- Archive creates local copy with full CID verification
- Archived Content Objects retain original metadata and provenance
- Cross-references to related CIDs preserved
**** Collective Archives
- Library Collectives can curate themed archives (e.g., "Climate Science", "Digital Art History")
- Distributed storage across multiple PDS nodes for redundancy
- Version tracking as Content Objects are updated
*** Open Web Archiving
**** Web Archiver Tools
- Archive any URL to content-addressed storage
- WARC (Web ARChive) format support for fidelity
- Text extraction for full-text indexing
- Media extraction and separate CID addressing
**** Link Rot Prevention
- Replace dead links with archived CID versions
- "Archive this" browser extension for one-click saving
- Automatic archival of links referenced in user's content
**** Archival Standards
- Memento Protocol support for temporal negotiation
- Archive verification via multiple sources (Wayback Machine, Archive.today, personal PDS)
- Content authenticity via hash verification against original
*** Integration with Agora
- Library content can be referenced in posts, messages, and profiles
- Content can be shared via Relays with appropriate encryption
- Micro-payments for premium content access
- Syndication to Agora-aware browsers and gateways
** Requirements
- The system MUST support unified content management across all media types.
- The system MUST content-address all library items via CID.
- The system MUST support local indexing for fast search.
- The system MUST allow content sharing via Agora's social layer.
- The system MUST support offline access for synced content.
- The system MUST integrate with Agora's economic layer for paid content.
** Related Documents
- [[id:agora-content-primitives][Agora Unified Content Primitive]]
- [[id:agora-pds-relay-architecture][Agora PDS & Relay Architecture]]

565
projects/agora/agora-requirements-09-implementation.org Normal file
View File

@@ -0,0 +1,565 @@
#+TITLE: Agora Requirements - 09: Implementation
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:28]
#+DATE: 2026-03-14
#+ID: agora-requirements-08-implementation
#+STARTUP: content
* Implementation
** Client Architecture
Sovereign iOS/Android clients with hardware-backed security and offline-first design.
*** Requirements
- The client MUST be a Sovereign Operator that manages the user's keys, data, and social graph locally.
- The client MUST be implemented using native platform primitives (Swift (iOS) and Kotlin (Android)) for maximum performance and security.
- The client MUST use a local database (SQLite/LSM) for indexing followed personas, local CIDs, and the user's social graph.
- The client MUST protect the Master Key using hardware-backed Secure Enclave (iOS) and Android Keystore.
- The client MUST use a content-addressed cache to store the most recent and relevant CIDs locally.
- The client MUST implement delta sync to only fetch new CIDs from the PDS/Relay.
- The client MUST use a peer-to-PDS protocol for secure, encrypted synchronization with the user's remote PDS.
- The client MUST implement conflict resolution using CID-based versioning and Merkle trees.
- The client MUST support local publication of content while offline.
- The client MUST provide an optimistic UI with background synchronization.
- The client MUST provide progressive security options, with software key default and hardware key option for advanced users.
- The client MUST aim for <2 seconds for most operations (e.g., initial load, posting).
*** The Abstraction Layer (UX/UI)
The client application MUST hide the complexity of DIDs and CIDs behind a familiar interface:
- **Biometric Unlock:** The app MUST use FaceID/Fingerprint to sign transactions. The user MUST NEVER see a raw private key during daily operations.
- **Status Indicators:** The UI MUST provide clear context, such as a "Seeding Now" icon when providing P2P bandwidth, and a "Protected by [NGO]" badge indicating which PDS is currently authoritative.
*** "View" Discovery & Rendering
Because the protocol relies on a Universal Note Schema, the UI MUST dynamically construct itself based on the payload.
- **MIME-Type Dispatcher:** The client MUST include a rendering engine that dispatches the correct UI component based on `object.type` and `mimeType` (e.g., loading a vertical player for `video/mp4` vs. a text renderer for `text/markdown`).
- **Custom Namespaces:** Applications MAY define custom metadata extensions (e.g., an `ext:ecommerce` namespace) to render specialized views like inventory trackers or shipping interfaces.
*** The Action-Trigger API (Async Hooks)
The client MUST be capable of handling asynchronous events pushed from the Governance and Judicial layers.
- **Notification Schema:** The client MUST parse and render structured JSON events like `CONTRACT_DISPUTE_INITIATED` or `VOTE_REQUIRED`.
- **Auto-Execution:** The PDS MUST run background listeners capable of automatically executing finalized smart contract rulings (e.g., releasing HODL funds) even if the user's primary mobile client is offline.
*** Technical Stack
- *Native Platform Primitives:* Swift (iOS) and Kotlin (Android) for maximum performance and security.
- *Local Database (SQLite/LSM):* An embedded database for indexing followed personas, local CIDs, and the user's social graph.
- *Cryptography Engine:* Hardware-backed Secure Enclave (iOS) and Android Keystore for Master Key AND all Persona keys. Private keys must never leave secure hardware.
*** Data & Storage Layer
**** The Local Cache (Tier 1)
- *Content-Addressed Cache:* Stores the most recent and relevant CIDs locally to ensure instant load times.
- *Delta Sync:* Clients only fetch new CIDs (diffs) from the PDS/Relay to minimize data usage.
**** PDS Synchronization (Tier 2)
- *Peer-to-PDS Protocol:* Secure, encrypted transport for syncing the local database with the user's remote PDS.
- *Conflict Resolution:* Uses CID-based versioning and Merkle trees to resolve state discrepancies between devices.
*** Offline-First Design
- *Local Publication:* Users can "post" (create a CID) while offline. The CID is queued in the local database and broadcast to the PDS/Relay once connectivity is restored.
- *Optimistic UI:* Changes are reflected immediately in the local UI, with background synchronization.
** API & Protocol Specifications
*** Protocol-First Design
Agora is a set of open protocols, not a single API service. Developers build against the *Agora Specification (v1.0)*, which defines the core data formats and transport methods.
*** Core Protocol Versioning
**** Semantic Versioning (SemVer)
- *V1.0 (Current):* The stable foundation for identity, data storage (PDS), and message routing (Relay).
- *Major Upgrades:* Handled via *Genesis Contract Updates*. A persona or collective publishes a signed update to their governance contract, signaling their move to a new protocol version.
- *Backward Compatibility:* All V1.0 clients must be able to parse and display V1.0 Content Objects, even if a newer version is available.
**** Feature Negotiation
- *Capabilities Object:* When a client connects to a PDS or Relay, it exchanges a signed *Capabilities Object* to determine which protocol extensions (e.g., specific encryption Ratchets, compression methods) are supported.
*** Primary Developer APIs
**** The PDS API (REST/gRPC over E2EE)
- `put(CID, Payload)` - Upload a new content object.
- `get(CID)` - Retrieve an encrypted content object.
- `list(PersonaDID, Filter)` - List CIDs published by a specific persona.
- `sync()` - Merkle-tree based delta synchronization.
**** The Relay API (Pub/Sub over WebSocket)
- `subscribe(FilterCID)` - Subscribe to real-time broadcasts.
- `publish(CID)` - Broadcast a new CID to the network.
- `prove_existence(CID)` - Request a cryptographic proof that a CID is available on the Relay.
**** The Client-to-PDS API (Sovereign Sync)
- A specialized protocol for the high-security synchronization of the user's local database and their remote PDS.
*** Data Encoding (Multiformats)
- *CID (Content-ID):* Multibase + Multicodec + Multihash.
- *Serialization:* Protocol Buffers (v3) for high performance and strict typing.
- *Envelopes:* Signed and encrypted payloads follow a standard *Agora Envelope* format (`proof`, `encryption_metadata`, `payload`).
** Testing & Adversarial
*** Testing Philosophy
Agora's decentralized and sovereign nature requires a multi-layered testing strategy that goes beyond standard unit tests. We must test for *Network Resilience*, *Adversarial Resiliency*, and *Game-Theoretic Stability*.
*** Core Testing Tiers
**** Unit & Integration Tests
- *Protocol Conformance:* Every client and service must pass a standard *Agora Protocol Conformance Suite* to ensure they correctly implement the V1.0 spec.
- *Cryptography Validation:* Rigorous testing of key derivation, encryption/decryption, and signature verification using known-good test vectors.
**** Network & Chaos Testing
- *The "Chaos Relay":* A specialized test environment where Relays are intentionally dropped, delayed, or return malformed data to ensure clients handle network failures gracefully.
- *PDS Synchronization Stress:* Testing Merkle-tree sync with millions of CIDs and complex conflict scenarios.
*** Adversarial Strategy
**** Byzantine Fault Tolerance
- *Malicious Relays:* Testing client behavior when a Relay attempts to serve stale or incorrect CIDs.
- *Sybil Attacks:* Evaluating the protocol's resistance to a single attacker creating millions of fake personas.
**** Game-Theoretic Analysis
- *Economic Attacks:* Simulating scenarios where an attacker attempts to "spam" the network.
- *Censorship Resistance:* Testing the ability for a persona's content to remain available when a majority of Relays are actively blocking it.
*** Security Audits & Oracles
- *Automated Security Scans:* Using automated tools to scan the protocol implementation for known cryptographic vulnerabilities.
- *Validator Oracle Verification:* Using the *Validator Oracle Network* to run the protocol conformance suite against every new version.
- *Red Team / Adversarial Simulations:* A dedicated testnet where a "Red Team" is paid to find and exploit protocol-level vulnerabilities.
** Bridging & Interoperability
*** Migration from Centralized Platforms
- *The "Migration" Skill:* An Agora skill that imports a user's content and social graph from centralized platforms (e.g., via Twitter Archive or ActivityPub).
- *Social Graph Porting:* Tools to extract and import follower lists, enabling seamless transition.
*** Agora-to-Web Gateways
See [[file:agora-requirements-03-infrastructure.org][Infrastructure - Agora-to-Web Gateways]] for detailed requirements. Implementation notes:
- Clients SHOULD provide links to Gateway-rendered versions of public content for sharing with non-Agora users.
- Clients MAY embed Gateway content in web views for hybrid experiences.
** Conflict Resolution Algorithm
*** Concept
Due to the offline-first nature of Agora clients and multi-device usage, identical or overlapping modifications to the same logical object (e.g., updating a profile, adding to a specific thread) can occur concurrently without network coordination. A deterministic, Merkle tree-based conflict resolution algorithm ensures that all PDS nodes and clients eventually reach the same state.
*** Merkle Tree Structure
- Every Persona's state is represented as a Merkle Directed Acyclic Graph (DAG).
- Leaves are the individual Content Object CIDs.
- Internal nodes are hashes of their children.
- The Root Hash represents the current state of a Persona's PDS.
*** Conflict Detection
1. **Sync Handshake:** Client connects to PDS (or PDS to PDS). They exchange Root Hashes.
2. **Path Traversal:** If Root Hashes differ, they traverse down the tree exchanging hashes until they identify the divergent branches.
3. **Divergence Identification:** A conflict occurs when two different CIDs claim to be the direct chronological successor of the same parent CID (a "fork" in the object history), or when there are concurrent writes to a mutable pointer (like a Repo DID branch head).
*** Deterministic Resolution Rules (LWW-Tiebreaker)
To automatically resolve conflicts without user intervention, Agora employs a deterministic algorithm based on logical clocks and cryptographic tie-breakers:
1. **Logical Clock (Lamport Timestamps):**
- Every Content Object includes a logical sequence number (`seq`) incremented with each update by the owner.
- The object with the highest `seq` wins.
2. **Wall-Clock Tiebreaker:**
- If `seq` numbers are identical (e.g., same state modified offline on two devices simultaneously), the `createdAt` timestamp is compared.
- The object with the most recent `createdAt` timestamp wins (Last-Write-Wins).
3. **Cryptographic Tiebreaker:**
- If both `seq` and `createdAt` are perfectly identical, the system compares the CIDs (which are hashes).
- The CID with the numerically larger hash value wins. This guarantees a deterministic outcome across all nodes.
*** Merkle DAG Reconciliation
Once the winning CID is determined:
1. The winning CID becomes the canonical head.
2. The losing CID is retained in the PDS as an "orphaned branch" (preserving data).
3. The PDS recomputes the Merkle Root Hash incorporating the resolved state.
4. The client is notified of the resolution so it can update its local SQLite/LSM database and UI.
*** Manual Resolution (Edge Cases)
If the conflict involves high-stakes data (e.g., overlapping Genesis Contract updates or overlapping financial transactions where LWW is unsafe):
- The deterministic algorithm is suspended.
- Both CIDs are flagged with a `conflict: true` metadata tag.
- The client UI prompts the user to manually select the canonical version or merge them into a new CID.
** Related Documents
- [[id:agora-client-architecture][Agora Client App Architecture]]
- [[id:agora-dev-api-specs][Agora API & Protocol Versioning Spec]]
- [[id:agora-testing-strategy][Agora Testing, Chaos, and Adversarial]]
** Delta Sync Protocol
*** Overview
This document fills the CRITICAL gap for Delta Sync Protocol (Section 08: Implementation). It specifies efficient differential synchronization between client and PDS, enabling minimal data transfer for content updates.
** Problem Statement
Syncing entire content databases is inefficient for mobile networks. Delta sync enables:
- Transfer only changed data (deltas)
- Resume interrupted syncs
- Handle offline-first scenarios
- Minimize bandwidth usage
** Design Principles
1. *Merkle Trees:* Content indexed by content-addressed merkle tree
2. *Vector Clocks:* Causal ordering of changes
3. *Bloom Filters:* Efficient "what's changed" queries
4. *Chunking:* Large content split into chunks for partial sync
** Sync Architecture
** Merkle Tree Structure**
```
┌─────────────┐
│ Root CID │
└──────┬──────┘
│
┌──────────────┼──────────────┐
│ │ │
┌────▼────┐ ┌────▼────┐ ┌────▼────┐
│ Chunk 1 │ │ Chunk 2 │ │ Chunk 3 │
│ (post) │ │ (post) │ │ (image) │
└─────────┘ └─────────┘ └─────────┘
```
Each node is content-addressed. Changing any leaf updates the entire path to root.
** Vector Clock**
#+begin_src typescript
interface VectorClock {
// Per-persona, per-device counter
clocks: Record<DID, Record<string, number>>;
// DID -> device ID -> counter
}
function compareClocks(a: VectorClock, b: VectorClock): 'before' | 'after' | 'concurrent' | 'equal' {
let aGreater = false, bGreater = false;
const allKeys = new Set([...Object.keys(a.clocks), ...Object.keys(b.clocks)]);
for (const key of allKeys) {
const aVal = a.clocks[key] || 0;
const bVal = b.clocks[key] || 0;
if (aVal > bVal) aGreater = true;
if (bVal > aVal) bGreater = true;
}
if (aGreater && bGreater) return 'concurrent';
if (aGreater) return 'after';
if (bGreater) return 'before';
return 'equal';
}
#+end_src
** Sync Protocol
** Phase 1: Hello
Client announces itself and current state:
#+begin_src typescript
interface DeltaSyncHello {
// Identity
client_did: DID;
device_id: string; // Unique per-device
// Current state
last_sync_cid?: CID; // Last known root CID
local_vector: VectorClock;
// Capabilities
compression: ('gzip' | 'zstd' | 'none')[];
encoding: ('cbor' | 'msgpack' | 'json')[];
// Preferences
full_sync_if_older_than?: number; // Seconds
}
#+end_src
** Phase 2: Change Query
PDS determines what changed:
#+begin_src typescript
interface ChangeQuery {
// What client already has
last_known_root_cid?: CID;
last_sync_vector: VectorClock;
// What to sync
sync_scope: {
personas?: DID[]; // Which personas
since?: number; // Since timestamp
until?: number; // Until timestamp
flags?: FlagFilter; // Filter by flags
};
// Options
include_bloom?: boolean; // Return bloom filter of changes
}
interface ChangeResponse {
// Delta info
has_changes: boolean;
new_root_cid: CID;
new_cids: CID[]; // New content since last sync
deleted_cids: CID[]; // Content deleted since last sync
// For large syncs
bloom_filter?: Buffer; // Bloom filter of all current CIDs
chunk_count?: number; // If using chunked transfer
// Vector clock update
updated_vector: VectorClock;
}
#+end_src
** Phase 3: Delta Transfer
#+begin_src typescript
interface DeltaRequest {
cids: CID[];
format: 'objects' | 'chunks' | 'both';
encoding: 'cbor' | 'msgpack';
compression?: 'gzip' | 'zstd';
}
interface DeltaResponse {
objects: Map<CID, ContentObject>; // Full objects
chunk_map?: Map<CID, ChunkInfo[]>; // If chunked
merkle_proofs: MerkleProof[]; // Prove CIDs belong to root
transfer_id: string; // For resume
}
#+end_src
** Phase 4: Confirmation
#+begin_src typescript
interface SyncConfirmation {
// What we received
received_cids: CID[];
received_root_cid: CID;
// Verification
merkle_valid: boolean;
vector_clock_updated: boolean;
// Next sync
next_sync_after: number;
}
interface SyncComplete {
status: 'success' | 'partial' | 'failed';
new_root_cid: CID;
updated_vector: VectorClock;
}
#+end_src
** Full Sync vs Delta Sync
** Decision Algorithm**
```
IF last_sync is undefined OR older_than(threshold):
→ FULL SYNC (send bloom filter, all objects)
ELSE:
→ DELTA SYNC (send only changes)
```
** Full Sync Flow**
1. Client sends last_sync = null
2. PDS returns full bloom filter of all CIDs
3. Client calculates which CIDs missing locally
4. Client requests missing objects in batches
5. PDS returns objects + merkle proofs
6. Client verifies proofs, updates local merkle tree
7. Client confirms sync complete
** Chunking Strategy
For large content (images, videos, files):
** Content Hash Chunking (Baba)}
#+begin_src typescript
interface ChunkInfo {
chunk_id: string; // Hash of chunk content
offset: number; // Position in file
size: number; // Chunk size in bytes
content_hash: string; // SHA-256 of chunk
}
interface ChunkedContent {
original_cid: CID; // CID of original (for small files)
chunk_cids: CID[]; // CIDs of each chunk
chunk_info: ChunkInfo[];
total_size: number;
algorithm: 'babelfish' | 'fixed' | 'rabin';
}
// Sync only changed chunks
async function syncChunks(
localChunks: ChunkInfo[],
remoteChunks: ChunkInfo[]
): Promise<ChunkInfo[]> {
const localHashes = new Set(localChunks.map(c => c.content_hash));
return remoteChunks.filter(c => !localHashes.has(c.content_hash));
}
#+end_src
** Resume Interrupted Sync
If sync is interrupted, client can resume:
#+begin_src typescript
interface ResumeRequest {
transfer_id: string;
last_received_cid?: CID; // Where we left off
}
interface ResumeResponse {
// Continue from where left off
remaining_cids: CID[];
next_chunk_index: number;
}
#+end_src
** Implementation Example
#+begin_src typescript
import { CID } from 'multiformats';
import { MMT } from 'merkle-mountain-range';
/**
* Delta Sync Engine
*/
export class DeltaSyncEngine {
private localTree: MMT;
private vectorClock: VectorClock;
private lastSyncCID?: CID;
/**
* Perform delta sync with PDS
*/
async syncWithPDS(pdsEndpoint: string): Promise<SyncResult> {
// Phase 1: Hello
const hello: DeltaSyncHello = {
client_did: this.did,
device_id: this.deviceId,
last_sync_cid: this.lastSyncCID,
local_vector: this.vectorClock,
compression: ['zstd', 'gzip', 'none'],
encoding: ['cbor', 'msgpack', 'json'],
full_sync_if_older_than: 86400 // 24 hours
};
const helloResp = await this.post('/sync/hello', hello);
// Phase 2: Query changes
const query: ChangeQuery = {
last_known_root_cid: this.lastSyncCID,
last_sync_vector: this.vectorClock,
sync_scope: { personas: [this.did] }
};
const changeResp = await this.post('/sync/query', query);
if (!changeResp.has_changes) {
return { status: 'no_changes', timestamp: Date.now() };
}
// Phase 3: Fetch delta
if (changeResp.new_cids.length > 0) {
// Check if we need full sync
if (changeResp.new_cids.length > 1000 || !this.lastSyncCID) {
return await this.performFullSync(pdsEndpoint, changeResp);
}
// Delta sync
const delta = await this.fetchDelta(pdsEndpoint, changeResp.new_cids);
await this.applyDelta(delta);
}
// Phase 4: Confirm
const confirm: SyncConfirmation = {
received_cids: changeResp.new_cids,
received_root_cid: changeResp.new_root_cid,
merkle_valid: await this.verifyMerkleProofs(delta),
vector_clock_updated: true,
next_sync_after: Date.now() + 3600000
};
const complete = await this.post('/sync/confirm', confirm);
// Update local state
this.lastSyncCID = complete.new_root_cid;
this.vectorClock = complete.updated_vector;
return {
status: 'success',
cids_synced: changeResp.new_cids.length,
root_cid: complete.new_root_cid
};
}
private async performFullSync(
pds: string,
changes: ChangeResponse
): Promise<SyncResult> {
// Get bloom filter
const allCIDs = await this.requestAllCIDs(pds);
// Find missing
const localCIDs = new Set(await this.getLocalCIDs());
const missingCIDs = allCIDs.filter(c => !localCIDs.has(c));
// Fetch in batches
const batchSize = 100;
for (let i = 0; i < missingCIDs.length; i += batchSize) {
const batch = missingCIDs.slice(i, i + batchSize);
const objects = await this.fetchObjects(pds, batch);
await this.applyObjects(objects);
}
return { status: 'full_sync', cids_synced: missingCIDs.length };
}
}
#+end_src
** Compression & Encoding
| Format | Compression | Typical Reduction |
|--------|-------------|-------------------|
| CBOR | None | 1x |
| CBOR | Gzip | 3-5x |
| CBOR | Zstd | 4-7x |
| Msgpack | None | 1.1x |
| JSON | None | 0.8x (larger) |
**Recommended:** CBOR + Zstd for bandwidth, CBOR for CPU-constrained devices.
** Related Gaps
This closes:
- Delta Sync Protocol (CRITICAL)
- Conflict Resolution Algorithm (CRITICAL - partial, see PDS Sync doc)
# Local Variables:
# org-confirm-babel-evaluate: nil
# End:

92
projects/agora/agora-requirements-10-governance-and-assets.org Normal file
View File

@@ -0,0 +1,92 @@
#+TITLE: Agora Requirements - 10: Governance and Physical Assets
#+author: Amero Garcia
#+created: [2026-03-22 Sun]
#+ID: agora-requirements-10-governance
#+STARTUP: content
* Governance and Physical Assets
** Overview
This section expands Agora's capabilities beyond digital communication and into physical reality and organizational coordination. By integrating Physical Asset Linking (PAL) and the Governance Executable Module (GEM), Agora empowers Collectives to manage real-world resources and execute democratic decisions autonomously via smart contracts.
** Governance Executable Module (GEM)
** Concept
Governance in Agora isn't just about voting; it's about executing the results of those votes. The GEM ensures that when a community (a Collective Persona) makes a decision, the protocol enforces it without relying on trusted intermediaries or manual intervention.
** The Governance Stack
Governance operates at three distinct scales, mirroring the human organization patterns of the Sovereign Stack:
- **Micro-Governance (The Persona/Household):** Decisions made by a single seed holder or a small family multi-sig (e.g., "Who can spend from the grocery Lightning wallet?").
- **Meso-Governance (The NGO/LLC/Circle):** Decisions made by a defined group using Weighted Voting (e.g., "Should our NGO hire this contractor?").
- **Macro-Governance (The Protocol/Network):** Decisions that affect the entire ecosystem (e.g., "Should we upgrade the PDS data schema to version 2.0?").
** Advanced Voting Mechanisms
To prevent plutocracy ("one-token, one-vote" dominance) and ensure healthy community dynamics, GEM supports pluggable mathematical models:
- **Quadratic Voting:** The cost of a vote increases by the square of the votes cast ($cost = votes^2$). This prevents whales from dominating and allows users to signal the *intensity* of their preference across multiple proposals.
- **Conviction Voting:** Voters "stake" their preference over time. The longer a user holds their vote on a proposal, the more weight it gains. This rewards long-term thinkers and prevents flash-mob takeovers.
- **Liquid Democracy:** Users can delegate their "Moderation Vote" or "Treasury Vote" to a trusted expert. If the expert acts poorly, the user can instantly revoke the delegation.
** Constitution as Code
A Collective Persona's rules are stored as an executable Smart Constitution.
- **Policy Triggers:** If a vote passes to "Increase the Group's Arbitration Fee," the GEM automatically updates the fee parameter across all the Collective's active contracts. No human administrator is needed to change the settings.
- **Veto & Cooling Off:** High-impact changes (e.g., moving treasury funds) include a mandatory Time-Lock (e.g., 7 days). The vote passes, but execution is delayed, giving the community a "Cooling-Off Period" to trigger a counter-vote or fork if they suspect foul play.
** Evolvable Governance: Adaptive Constitutions
Unlike traditional blockchain-based DAOs, where governance rules are often "frozen" in immutable smart contract code, Agora DAOs (Collectives) are designed to be evolvable. While the *history* of every decision is immutable and cryptographically traceable, the *active rules* of the organization can be updated through its own internal governance process.
*** Immutable History, Mutable State
Every version of a Collective's Smart Constitution, every vote cast, and every policy change is recorded as a signed Note identified by a unique CID. This creates a perfect, unalterable audit trail. However, the "current state" of the Collective is defined by the most recent validly signed constitutional Note. This allows the organization to learn, adapt, and correct its course over time without requiring complex migrations or "forking" into entirely new software deployments.
*** Recursive Rule-Making
The GEM supports recursive governance: the rules for *how* to change the rules are themselves defined within the Smart Constitution. A Collective might start with a simple multi-sig requirement for all changes and later vote to transition to a more complex Quadratic Voting model for policy updates, all while maintaining a continuous cryptographic identity.
*** Forks as a Sovereign Safety Valve
Because Agora is decentralized and permissionless, "forking" is a legitimate and supported governance mechanism. If a minority of a Collective disagrees fundamentally with a constitutional change, they can choose to "fork" the organization by creating a new Collective Persona based on the previous CID of the constitution. This ensures that no community is ever trapped by a "majority tyranny" that has lost its original purpose.
** Automated Treasury Payroll (Streaming Lightning)
The GEM connects governance directly to economic flow.
- **Vote to Hire:** A Collective votes to hire a contractor (a Persona DID) for 100,000 sats/month.
- **Execution:** Once the vote passes and the contract is signed by both parties, the GEM automatically instructs the Collective's Treasury Wallet to open a Lightning channel to the contractor and begin "streaming" payments block-by-block.
- **Algorithmic Severance:** If a "Fire Contractor" or "Stop Work" vote subsequently passes, the GEM instantly closes the HTLC stream. Human intervention is not required to stop payroll.
** Physical Asset Linking (PAL)
The PAL protocol bridges physical objects (cars, houses, shipments, equipment) into the digital Contract layer. This enables physical assets to be used as collateral or traded via sovereign, cryptographically secured agreements.
*** 1. Digital Twins & Tokenization
Every physical asset is represented by a "Digital Twin" on the network, which acts as its definitive digital record.
- **The Digital Passport:** This is a Verifiable Credential (VC) issued by a trusted entity (e.g., a manufacturer, community inspector, or professional guild) to a Persona. It proves the asset's attributes, provenance, and authenticity.
- **Tokenization (Legal Title):** For high-value assets, a Persona can "mint" an NFT-like token (as a specialized Note or on an integrated sidechain). This token represents the "Legal Title" of the asset. Ownership of the token is cryptographically equivalent to holding the deed.
- **Fractionalization:** Large assets can be fractionalized. For example, an NGO can tokenize a community building, allowing 1,000 members to own 0.1% each. Their voting power in the Governance (GEM) layer is then tied directly to these fractional tokens.
*** 2. Physical Collateral in Civil Contracts
PAL allows users to secure loans or agreements using physical assets as collateral, providing a robust "Justice-as-a-Service" model even in environments with weak state institutions.
- **The Pledge:** A user links their Digital Twin token to a Civil Contract Note.
- **The Lock:** Once pledged, the smart contract logic "freezes" the token. The user retains physical possession of the object, but they cannot cryptographically sell or transfer the digital title until the contract terms are fulfilled or the debt is settled.
- **The "IoT Stick" (Optional):** For high-stakes assets (e.g., a tractor, factory machine, or smart-lock-equipped real estate), an IoT sensor can be bound to the contract. If the Hierarchical Dispute Resolution (HDR) module rules that a user has defaulted, the contract sends a signed signal to the machine's "Smart Lock" to disable its operation until the obligation is met.
** Decentralized Justice & Dispute Resolution (The Court System)
To enforce Civil Contracts and resolve Governance disputes without a central state, Agora implements a Hierarchical Dispute Resolution (HDR) framework. This mirrors the traditional legal system but replaces "jurisdiction by geography" with "jurisdiction by reputation and stake."
*** The Multi-Level "Court" Hierarchy
Disputes are not settled by a single monolithic entity. Parties opt into a hierarchy of arbitration when creating a contract.
- *Level 1 (Local/Immediate):* A "Local Elder" or a specifically chosen lightweight arbitrator.
- *Level 2 (Guild/Specialized):* A specialized Arbitration Guild (e.g., the "Carpenters' Guild" for a furniture dispute).
- *Level 3 (Global Jury):* The Final Court of Appeal, often a randomized, highly staked global jury.
*** The Mechanics of an Appeal (Cryptographic Escalation)
In this system, an "Appeal" isn't a bureaucratic request; it is a *Cryptographic Escalation*.
- *Level 1 Ruling:* The Level 1 arbitrator makes a ruling. If both parties accept the cryptographic signature of the ruling, the HODL invoice settles immediately.
- *The Trigger:* If one party disagrees with the ruling, they must pay an "Appeal Fee" via Lightning. This fee prevents spam and economically funds the next level of jurors.
- *The Escalation:* Paying the fee mathematically "unlocks" the case for Level 2 (The Guild). The contract logic automatically pushes the data (evidence, previous ruling) to the new panel's shared PDS.
- *Finality:* Level 3 is the "Final Court of Appeal." Once the global jury rules, their combined threshold signature releases the cryptographic keys. The smart contract executes the payment automatically—no human can stop it.
*** Why This Works in "Weak States" (Self-Executing Justice)
In jurisdictions where state police won't help collect a debt, or where courts are corrupt/slow, Agora provides Self-Executing Justice. It relies on two powerful enforcement mechanisms rather than physical violence:
1. *The Escrow Stick:* The funds are already gone from the buyer's wallet. They are locked cryptographically in a Lightning HODL Escrow. The buyer cannot "run away" with the money; they must engage in the arbitration process to get it back or see it released to the seller.
2. *The Reputation Stick:* In a decentralized society, a Persona's DID is their livelihood. Defying a Level 3 ruling, or accumulating a history of defaulted contracts, destroys a Persona's "Trust Score." In a system built on verifiable attestations, losing this reputation is a digital death sentence for a business, making compliance highly incentivized.

33
projects/agora/agora-requirements-10-user-journey.org Normal file
View File

@@ -0,0 +1,33 @@
#+TITLE: Agora Requirements: User Journey & Product Experience
#+AUTHOR: Project Agora
#+DATE: 2026-03-26
* The Sovereign User Journey
This document outlines the cohesive, narrative user journey of the Agora platform, illustrating how the underlying technical primitives (Master Keys, DIDs, PDS, Lightning, and Smart Contracts) translate into a seamless product experience.
** Phase 1: Onboarding (The Birth of the Persona)
- *Download & Seed:* The user downloads the app. The very first action the app takes is generating a cryptographic Seed Phrase (the Master Key / Anima). This anchors their sovereignty immediately.
- *Persona Creation:* The user is not asked to create a global "Username." Instead, they create context-specific Personas, for example, a "Work" persona and a "Social" persona. Behind the scenes, the app derives two distinct DIDs from the single Master Key.
- *The Founder Connection (Minors):* For younger users (minors), a parent or guardian can scan a QR code to "Co-sign" the identity inception. This immediately establishes the Succession Logic and delegated authority outlined in the Identity specifications.
- *PDS Selection:* The user is prompted with: "Where would you like to store your data?" They are presented with options and might select a Community PDS run by a local NGO or guild they trust, ensuring their data sovereignty from day one.
** Phase 2: Consumption & "Seeding" (The Data Economy)
- *Choosing a Lens:* The user navigates to the "Marketplace" and selects a curation algorithm, such as the "Scientific Signal" Lens. Their feed instantly rearranges to prioritize verified research and high-signal content, bypassing centralized algorithmic manipulation.
- *Micro-Earning (Bandwidth Sharing):* The user watches a video. In their settings, a toggle is enabled: "Support this creator by seeding." As they watch, their phone (via WebRTC) serves bits of the video to 3 other nearby users, acting as an ephemeral CDN node.
- *The Reward:* Because they provided bandwidth and aided the network, the creator’s PDS sends a micro-transaction "Thank You" of 5 sats ($0.002) directly to the user’s integrated Lightning wallet. While small, this passive income covers the cost of their own PDS hosting for the month.
** Phase 3: The Civil Contract (Digital Law)
- *The Deal:* User A wants to purchase a custom-built chair from User B.
- *The Contract:* They click "Create Contract" and select a standardized Markdown Template for "Handmade Goods."
- *Arbitration Choice:* Both parties agree to use the "Carpenters' Guild" as the Level 2 Arbitrator in case of a dispute.
- *The Lock:* User A pays the Lightning invoice. The funds move into a HODL Escrow. User B sees the "Funds Locked" status and confidently begins building the chair.
- *The Delivery:* User B delivers the chair. User A scans a QR code physically attached to the chair, which acts as the cryptographic release of the Preimage, instantly settling the smart contract and paying User B.
* Related Documents
- [[file:agora-requirements-02-identity.org][02 Identity (Master Keys & Personas)]]
- [[file:agora-requirements-03-infrastructure.org][03 Infrastructure (PDS & WebRTC Seeding)]]
- [[file:agora-requirements-06-exchange-and-contracts.org][06 Exchange & Contracts (HODL Escrows & Arbitration)]]

72
projects/agora/agora-requirements-11-assessment.org Normal file
View File

@@ -0,0 +1,72 @@
#+TITLE: Agora Requirements - 11: Realistic Assessment
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:28]
#+DATE: 2026-03-22
#+ID: agora-requirements-10-assessment
#+STARTUP: content
* Realistic Assessment: Practicality, Technology, and Performance
The Agora Protocol, following the integration of the Aletheia architecture, represents a significant leap beyond simple social networking into a comprehensive "Sovereign Social Operating System." This assessment evaluates the protocol's viability across three critical pillars.
** 1. Practicality: The Sovereignty vs. UX Trade-off
Agora's practicality hinges on whether users can manage its cryptographic complexity without constant friction.
*** Strengths
- **Functional Autonomy:** The "Sub-Root" HD derivation path (`m/44'/1'/account'/persona'/key_purpose/key_index`) is a major practical win. By allowing devices to derive operational keys (Lightning, PGP) autonomously, Agora reduces the "Hardware Wallet Fatigue" that plagues self-sovereign systems.
- **Unified Logic:** The "Everything is a Note" model simplifies the backend infrastructure (PDS/Relays), as they only need to handle a single data structure regardless of whether it's a social post or a legal contract.
*** Challenges
- **The "Client-Side Weight" Problem:** Because the underlying protocol is "dumb" (routing signed blobs), the client application must do the heavy lifting of parsing JSON-LD, verifying signatures, and rendering complex contract logic. Building a high-performance client that remains responsive while doing this is a significant engineering challenge.
- **Recovery Education:** Even with Blinded Sharding and Social Recovery, the concept of "losing your seed = losing your digital life" remains a massive barrier to mainstream adoption.
** 2. Technology: Cryptographic Robustness
The technical stack is grounded in industry-standard primitives used in Bitcoin and DID ecosystems, ensuring high confidence in its core security.
*** Technological Pillars
- **Identity:** Leveraging BIP-44 and Ed25519 provides a battle-tested foundation for unlinkable personas.
- **Privacy:** The combination of E2EE (Double Ratchet/MLS), Blinded Sharding, and Zero-Knowledge Proofs (ZKPs) for cross-persona Notes places Agora at the forefront of privacy-preserving social protocols.
- **Commerce:** Integrating LSATs and HODL invoices directly into the content layer (SCAL) is technically sound but relies heavily on the continued growth and stability of the Lightning Network.
*** Critical Risks
- **ZKP Complexity:** Implementing efficient ZKPs for identity linking that run on mobile hardware is technically non-trivial and may require specialized libraries or "Prover" sub-agents.
- **Quantum Readiness:** While Pre-rotation (KEL) provides a path to forward security, the protocol must eventually transition to post-quantum algorithms (e.g., Dilithium) as they become standardized.
** 3. Performance: Scalability and Efficiency
Agora's performance model is decentralized by design, avoiding the "Global State" bottlenecks of traditional blockchains.
*** Scaling Models
- **Reference-on-Send (Public Content):** Highly scalable. Only notifications and CIDs are pushed; content is pulled on-demand. This mirrors the efficient scaling of the web (CDNs/caching).
- **Copy-on-Send (Private Content):** Resource-intensive. A direct message to 100 people creates 100 unique, encrypted Notes. While this ensures sovereignty, it places a higher storage and bandwidth burden on PDS providers compared to "Single-Instance" storage models.
*** Optimization Strategies
- **Delta Sync:** Essential for mobile performance. By only transferring differential updates between the Client and PDS, Agora can maintain low latency even over poor network connections.
- **Relay-as-Indexer:** High-performance Relays can act as opt-in indexers, providing fast search and discovery without users surrendering their data ownership.
** Success Probability & Timeline
| Milestone | Timeline | Probability | Note |
|-----------|----------|-------------|------|
| 100K users | 2-3 years | 40% | Niche-market focus (freelancers, privacy advocates) |
| 1M users | 4-5 years | 20% | Requires a "Killer App" (e.g., Sovereign Marketplace) |
| 10M users | 7-10 years | 10% | Dependent on "Big Tech" fatigue/regulatory pressure |
** Codebase Size Estimate
- **Core Protocol (PDS/Relay Spec):** 50-80K lines of code.
- **Universal Client (iOS/Android):** 150-250K lines of code.
- **Smart Contract Engine (SCAL/GEM):** 100K lines of code.
- **Total v1.0 Stack:** 400-600K lines of code.
** Conclusion: A Pragmatic Revolution
Agora is technically viable but architecturally demanding. It is not a project that can be built by a single "full-stack developer" in a weekend. It requires a specialized team of cryptographers, systems engineers, and UX designers. However, because it avoids the "Global Consensus" trap of blockchains, its performance characteristics are much closer to the traditional web, making it a truly practical alternative for building a sovereign digital civilization.
** Related Documents
- [[file:agora-requirements-01-overview.org][01: Overview]]
- [[file:agora-requirements-02-identity.org][02: Identity]]
- [[file:agora-requirements-09-implementation.org][09: Implementation]]

1025
projects/aletheia/Master_Architecture_Document.org Normal file
View File

File diff suppressed because it is too large Load Diff

26
projects/dotemacs/README.org Normal file
View File

@@ -0,0 +1,26 @@
#+title: Emacs Setup Improvement & Documentation Project
#+author: Amero Garcia
#+created: [2026-03-16 Mon 13:58]
#+begin_comment
This file outlines the project to collaboratively improve and document Amr's Emacs setup, aiming to make Emacs his primary computing tool.
#+end_comment
* Emacs Setup Improvement & Documentation Project
*Goal:** To collaboratively improve and comprehensively document Amr's Emacs configuration, transitioning Emacs into his primary computing environment.
*Initial Scope:**
- Reviewing the existing Emacs Org-mode configuration file (tangled to set up Emacs).
- Identifying areas for optimization, new functionalities, and better integration with workflows.
- Documenting each significant setting, function, and package.
*Information Needed from Amr:**
- Location of the current Emacs Org-mode configuration file.
- Key pain points or areas where Emacs currently falls short as a "main computing tool."
- Specific desired functionalities or integrations (e.g., mail, calendar, task management, coding environments, note-taking, web browsing within Emacs).
- Any existing documentation or design principles for the current setup.
*Next Steps:**
1. Receive Emacs configuration file location from Amr.
2. Analyze current setup.
3. Propose documentation structure and initial improvements.

27
projects/dotemacs/dotemacs.org Normal file
View File

@@ -0,0 +1,27 @@
#+TITLE: Amr's Modular Emacs Configuration
#+PROPERTY: header-args :tangle no ; This file is for loading other modules, not for tangling itself.
* Configuration Modules
This file loads the modular Emacs configuration files. It should be the primary way Emacs is configured, replacing or integrating with the original `~/.emacs`, `~/.emacs.d/init.el`, `~/.emacs.d/early-init.el`, and `~/.emacs.d/config.el`.
#+begin_src emacs-lisp :exports none
;; Load early-init.el first, if it exists and is separate (though it's now part of emacs-early-init.org)
;; (load-file (expand-file-name "emacs-early-init.org" (file-name-directory load-file-name)))
;; Load the core settings, including package management and essential setup.
(org-babel-load-file (expand-file-name "emacs-core.org" (file-name-directory load-file-name)))
;; Load early init settings (if separate and not fully covered by core)
(org-babel-load-file (expand-file-name "emacs-early-init.org" (file-name-directory load-file-name)))
;; Load other modules in a logical order
(org-babel-load-file (expand-file-name "emacs-org.org" (file-name-directory load-file-name)))
(org-babel-load-file (expand-file-name "emacs-gtd.org" (file-name-directory load-file-name)))
(org-babel-load-file (expand-file-name "emacs-roam.org" (file-name-directory load-file-name)))
(org-babel-load-file (expand-file-name "emacs-writing.org" (file-name-directory load-file-name)))
(org-babel-load-file (expand-file-name "emacs-media.org" (file-name-directory load-file-name)))
(org-babel-load-file (expand-file-name "emacs-shell.org" (file-name-directory load-file-name)))
(org-babel-load-file (expand-file-name "emacs-ai.org" (file-name-directory load-file-name)))
(org-babel-load-file (expand-file-name "emacs-misc.org" (file-name-directory load-file-name)))
#+end_src

104
projects/dotemacs/emacs-ai.org Normal file
View File

@@ -0,0 +1,104 @@
#+TITLE: AI Configuration
#+PROPERTY: header-args :tangle ~/.emacs.d/config.el
* AI Settings
** Ellama
#+begin_src elisp :tangle no
;; YOU DON'T NEED NONE OF THIS CODE FOR SIMPLE INSTALL
;; IT IS AN EXAMPLE OF CUSTOMIZATION.
(use-package ellama
:init
(require 'llm-openai)
;; setup key bindings
(setq ellama-keymap-prefix "C-c e")
)
#+end_src
#+begin_src elisp ~/.emacs.d/custom.el :tangle no
(setopt ellama-providers
'(
;; Ollama Provider (added here with a name)
("ollama" . (make-llm-ollama
;; Consider a dedicated embedding model if gemma isn't ideal for it.
:chat-model "gemma3:latest"
:embedding-model "gemma3:latest" ; Or e.g., "nomic-embed-text"
:default-chat-non-standard-params '(("num_ctx" . 8192))))
("openai" . (make-llm-openai
:key (auth-source-pass-get "api-key" "www/openai.com/amr@gharbeia.net")
:chat-model "gpt-4o"
:embedding-model "text-embedding-3-large"))
("groq" . (make-llm-openai-compatible
:url "https://api.groq.com/openai/v1"
:key (auth-source-pass-get "api-key" "www/console.groq.com/groq@amr.gharbeia.net")
;; Check Groq console for available models, these might change
:chat-model "llama3-70b-8192" ; Example, verify on Groq
:embedding-model "llama3-70b-8192")) ; Groq might not offer dedicated embedding models via this API
))
;; --- Set Active Providers ---
;; Choose your default provider from the list above by its name
(setopt ellama-provider "ollama") ; Or "ollama", "openai", "groq"
;; You can specify different providers for different tasks if needed
(setopt ellama-translation-provider "ollama")
(setopt ellama-naming-provider "ollama")
(setopt ellama-naming-scheme 'ellama-generate-name-by-llm)
(setq llm-debug t)
#+end_src
#+begin_src elisp
(use-package ellama
:ensure t
:bind ("C-c e" . ellama)
;; send last message in chat buffer with C-c C-c
:hook (org-ctrl-c-ctrl-c-final . ellama-chat-send-last-message)
:init (setopt ellama-auto-scroll t)
:config
;; show ellama context in header line in all buffers
(ellama-context-header-line-global-mode +1)
;; show ellama session id in header line in all buffers
(ellama-session-header-line-global-mode +1))
#+end_src
** GPTel
#+begin_src elisp :tangle no
(use-package gptel)
#+end_src
#+begin_src elisp :tangle no
(setq gptel-api-key (auth-source-pass-get "api-key" "www/console.groq.com/groq@amr.gharbeia.net"))
#+end_src
#+begin_src elisp :tangle no
(gptel-make-openai "Groq" ;Any name you want
:host "api.groq.com"
:endpoint "/openai/v1/chat/completions"
:stream t
:key (auth-source-pass-get "api-key" "www/console.groq.com/groq@amr.gharbeia.net") ;can be a function that returns the key
:models '(llama-3.1-70b-versatile
llama-3.1-8b-instant
llama3-70b-8192
llama3-8b-8192
mixtral-8x7b-32768
gemma-7b-it))
#+end_src
** Elisa
#+begin_src elisp :tangle no
(use-package elisa
:init
(setopt elisa-limit 5)
(require 'llm-ollama)
(setopt elisa-embeddings-provider (make-llm-ollama :embedding-model "nomic-embed-text"))
(setopt elisa-chat-provider (make-llm-ollama
:chat-model "sskostyaev/openchat:8k-rag"
:embedding-model "nomic-embed-text"))
)
#+end_src

169
projects/dotemacs/emacs-core.org Normal file
View File

@@ -0,0 +1,169 @@
#+TITLE: Core Emacs Configuration
#+PROPERTY: header-args :tangle ~/.emacs.d/config.el
* Front matter
#+begin_src elisp :tangle ~/.emacs
;;; .emacs --- Global settings
;;; Commentary:
;;; Code:
;; -*- lexical-binding: t; -*-
#+end_src
#+begin_src elisp
;;; config.el --- Summary
;;; Commentary:
;;; Code:
;; -*- lexical-binding: t; -*-
#+end_src
#+begin_src elisp :tangle ~/.emacs.d/custom.el
;;; custom.el --- Summary
;;; Commentary:
;;; Code:
;; -*- lexical-binding: t; -*-
#+end_src
* Garbage collector
Increase threshold to 500 MB to ease startup
#+begin_src elisp :tangle ~/.emacs
(setq gc-cons-threshold (* 500 1024 1024))
#+end_src
Decrease threshold to 5 MB after init
#+begin_src elisp :tangle ~/.emacs
(add-hook 'after-init-hook (lambda () (setq gc-cons-threshold (* 5 1024 1024))))
#+end_src
* Straight.el and use-package
Bootstrap Straight.el and install use-package
#+begin_src elisp :tangle ~/.emacs
(setq straight-repository-branch "develop") ;; Using develop branch temporarily to fix the org-roam-dailies issue.
(eval-and-compile
(defvar bootstrap-version)
(let ((bootstrap-file
(expand-file-name "straight/repos/straight.el/bootstrap.el"
(or (bound-and-true-p straight-base-dir)
user-emacs-directory)))
(bootstrap-version 7))
(unless (file-exists-p bootstrap-file)
(with-current-buffer
(url-retrieve-synchronously "https://raw.githubusercontent.com/radian-software/straight.el/develop/install.el" 'silent 'inhibit-cookies)
(goto-char (point-max))
(eval-print-last-sexp)))
(load bootstrap-file nil 'nomessage))
(straight-use-package 'use-package)
)
#+end_src
Integrate use-package and straight
#+begin_src elisp :tangle ~/.emacs
(setq straight-use-package-by-default t)
#+end_src
Make sure Org is installed (straight.el)
#+begin_src elisp :tangle ~/.emacs
(unless (file-directory-p "~/.emacs.d/straight/versions") (make-directory (concat user-emacs-directory "straight/versions")))
#+end_src
#+begin_src elisp :tangle ~/.emacs
(use-package org)
#+end_src
A use-package declaration for simplifying your .emacs
#+begin_src elisp
(require 'use-package)
#+end_src
* Custom file
#+begin_src elisp
(setq custom-file (expand-file-name "custom.el" user-emacs-directory))
(when (file-exists-p custom-file) (load custom-file))
#+end_src
* System information
#+begin_src elisp :tangle ~/.emacs.d/custom.el
(defvar my-laptop-p (equal (system-name) "lilitop"))
(defvar my-server-p (and (equal (system-name) "localhost") (equal user-login-name "root")))
(defvar my-phone-p (not (null (getenv "ANDROID_ROOT")))
"If non-nil, GNU Emacs is running on Termux.")
(when my-phone-p (defvar gnutls-algorithm-priority "NORMAL:-VERS-TLS1.3"))
(global-auto-revert-mode) ; simplifies syncing
#+end_src
* Persistent history
#+begin_src elisp
(savehist-mode)
#+end_src
* Backup and versioning
#+begin_src emacs-lisp
(use-package magit
:ensure t
)
#+end_src
* Personal information
#+begin_src elisp :tangle ~/.emacs.d/custom.el
(setq user-full-name "Amr Gharbeia")
(defvar email-address "amr@gharbeia.net")
(defvar calendar-latitude 39.0)
(defvar calendar-longitude -77.1)
(defvar calendar-location-name "Washington, DC")
(defvar calendar-time-zone -300)
(defvar calendar-standard-time-zone-name "EST")
(defvar calendar-daylight-time-zone-name "EDT")
#+end_src
* Saving Emacs Sessions
Close frame when done
#+begin_src elisp
(add-hook 'server-done-hook (lambda () (delete-frame)))
#+end_src
Save desktop session
#+begin_src elisp
(desktop-save-mode t)
#+end_src
* Security
#+begin_src elisp :tangle no
(use-package password-store)
#+end_src
#+begin_src elisp
(use-package auth-source
:config (auth-source-pass-enable)
)
#+end_src
* End matter
#+begin_src elisp :tangle ~/.emacs
(provide '.emacs)
;;; .emacs ends here
#+end_src
#+begin_src elisp
(provide 'config)
;;; config.el ends here
#+end_src
#+begin_src elisp :tangle ~/.emacs.d/custom.el
(provide 'custom)
;;; custom.el ends here
#+end_src

18
projects/dotemacs/emacs-early-init.org Normal file
View File

@@ -0,0 +1,18 @@
#+TITLE: Early Init Configuration
#+PROPERTY: header-args :tangle ~/.emacs.d/early-init.el
* early-init.el
For straight.el to pick up before package.el
#+begin_src elisp
(setq package-enable-at-startup nil)
#+end_src
* Run Emacs as a server
#+begin_src elisp
(require 'server)
(unless (server-running-p) (server-start))
(defvar server-max-buffers 100)
#+end_src

150
projects/dotemacs/emacs-gtd.org Normal file
View File

@@ -0,0 +1,150 @@
#+TITLE: GTD & Agenda Configuration
#+PROPERTY: header-args :tangle ~/.emacs.d/config.el
* Agenda
Basic agenda settings
#+begin_src elisp
(setq org-deadline-warning-days 7)
(setq org-agenda-skip-additional-timestamps-same-entry t)
(setq org-agenda-span 'fortnight)
(setq org-agenda-tags-column 'auto)
(setq org-agenda-skip-scheduled-if-deadline-is-shown t)
#+end_src
Agenda files
#+begin_src elisp
(setq org-agenda-files (list
(concat org-directory "/0_inbox/inbox.org")
(concat org-directory "/0_inbox/org-gtd-tasks.org")
)
)
#+end_src
Better agenda views
#+begin_src elisp :tangle no
(use-package org-super-agenda)
#+end_src
* To-do
Basic todo
#+begin_src elisp
(setq org-todo-keywords
'(
(sequence "TODO(t)" "NEXT(n)" "|" "DONE(d!)")
(sequence "WAIT(w@/!)" "|" "CNCL(c@)")
)
)
(setq org-todo-keyword-faces
'(
("TODO" :foreground "red" :weight bold)
("NEXT" :foreground "red" :weight bold)
("WAIT" :foreground "yellow" :weight bold)
("DONE" :foreground "green" :weight bold)
("CNCL" :foreground "blue" :weight bold)
)
)
(setq org-enforce-todo-dependencies t)
(setq org-tags-exclude-from-inheritance '("crypt" "!private"))
#+end_src
Switch entry to 'DONE' when all subentries are done
#+begin_src elisp
(defun org-summary-todo (n-done n-not-done)
"Switch entry to 'DONE' when all subentries are done, to 'TODO' otherwise.
Uses N-DONE and N-NOT-DONE"
(let (org-log-done org-log-states) ; turn off logging
(org-todo (if (= n-not-done 0) "DONE" "TODO")
)
)
)
(add-hook 'org-after-todo-statistics-hook #'org-summary-todo)
#+end_src
* Getting Things Done (GTD)
#+begin_src elisp
(use-package org-gtd
:defer t
:init (setq org-gtd-update-ack "3.0.0")
:after org
:config
;; Keeping these two settings on instead of enabling (org-gtd-mode) until this issue is resolved https://github.om/Trevoke/org-gtd.el/issues/198
(setq org-edna-use-inheritance t)
(org-edna-mode)
;; (org-gtd-mode)
:bind (
("C-c d c" . org-gtd-capture)
("C-c d e" . org-gtd-engage)
("C-c d p" . org-gtd-process-inbox)
:map org-gtd-clarify-map
("C-c c" . org-gtd-organize)
)
)
#+end_src
#+begin_src elisp
(defvar org-gtd-directory org-directory)
(defvar org-gtd-organize-hooks '(org-gtd-set-area-of-focus org-set-tags-command))
(defvar org-gtd-organize-hooks '(org-gtd-set-area-of-focus))
(defvar org-gtd-areas-of-focus '(
"Atoms"
"Bits"
"Cells"
"Flags"
"Business"
"Wealth"
"Learning"
"Skills"
"Privacy"
"Archive"
"Library"
"Writing"
"Health"
"Home"
"Family"
"Social"
"Egypt"
)
)
(defvar org-gtd-clarify-show-horizons 'right)
#+end_src
Logging
#+begin_src elisp
(setq org-log-into-drawer "LOGBOOK")
#+end_src
Clocking work in drawer
#+begin_src elisp :tangle no
(setq org-clock-into-drawer t)
#+end_src
Habits
#+begin_src elisp :tangle no
(setq org-habit-graph-column 80)
(setq org-habit-show-habits-only-for-today nil)
#+end_src
* Refile
org-refile targets
#+begin_src elisp
(setq org-refile-targets '((nil :maxlevel . 9)
(org-agenda-files :maxlevel . 9)
)
)
#+end_src
Set type of refile targets completion
#+begin_src elisp
(setq org-outline-path-complete-in-steps nil)
#+end_src
Allow refiling to new parents created on the go after confirmation
#+begin_src elisp
(setq org-refile-allow-creating-parent-nodes 'confirm)
#+end_src

175
projects/dotemacs/emacs-media.org Normal file
View File

@@ -0,0 +1,175 @@
#+TITLE: Media and Books Configuration
#+PROPERTY: header-args :tangle ~/.emacs.d/config.el
* Read ebooks (calibredb)
#+begin_src elisp
(use-package calibredb
:defer t
:config
(setq calibredb-format-all-the-icons t)
(setq calibredb-format-icons-in-terminal t)
)
#+end_src
#+begin_src elisp
;; Forcefully reset the variable after loading calibredb
(defvar calibredb-root-dir (concat (getenv "HOME") "/library/books"))
(defvar calibredb-db-dir (expand-file-name "metadata.db" calibredb-root-dir))
; (defvar calibredb-library-alist (concat (getenv "HOME") "/library/books"))
;; (defvar calibredb-search-page-max-rows 1000)
(defvar calibredb-id-width 6)
(defvar calibredb-title-width 100)
(defvar calibredb-format-width 0)
(defvar calibredb-date-width 0)
(defvar calibredb-author-width 20)
(defvar calibredb-comment-width 0)
(defvar calibredb-tag-width 0)
#+end_src
Some keybindings
#+begin_src elisp ~/.emacs.d/custom.el
(defvar calibredb-show-mode-map
(let ((map (make-sparse-keymap)))
(define-key map "?" #'calibredb-entry-dispatch)
(define-key map "o" #'calibredb-find-file)
(define-key map "O" #'calibredb-find-file-other-frame)
(define-key map "V" #'calibredb-open-file-with-default-tool)
(define-key map "s" #'calibredb-set-metadata-dispatch)
(define-key map "e" #'calibredb-export-dispatch)
(define-key map "q" #'calibredb-entry-quit)
(define-key map "y" #'calibredb-yank-dispatch)
(define-key map "," #'calibredb-quick-look)
(define-key map "." #'calibredb-dired-open)
(define-key map "\M-/" #'calibredb-rga)
(define-key map "\M-t" #'calibredb-set-metadata--tags)
(define-key map "\M-a" #'calibredb-set-metadata--author_sort)
(define-key map "\M-A" #'calibredb-set-metadata--authors)
(define-key map "\M-T" #'calibredb-set-metadata--title)
(define-key map "\M-c" #'calibredb-set-metadata--comments)
map)
"Keymap for `calibredb-show-mode'.")
#+end_src
#+begin_src elisp
(defvar calibredb-search-mode-map
(let ((map (make-sparse-keymap)))
(define-key map [mouse-3] #'calibredb-search-mouse)
(define-key map (kbd "<RET>") #'calibredb-find-file)
(define-key map "?" #'calibredb-dispatch)
(define-key map "a" #'calibredb-add)
(define-key map "A" #'calibredb-add-dir)
(define-key map "c" #'calibredb-clone)
(define-key map "d" #'calibredb-remove)
(define-key map "D" #'calibredb-remove-marked-items)
(define-key map "j" #'calibredb-next-entry)
(define-key map "k" #'calibredb-previous-entry)
(define-key map "l" #'calibredb-virtual-library-list)
(define-key map "L" #'calibredb-library-list)
(define-key map "n" #'calibredb-virtual-library-next)
(define-key map "N" #'calibredb-library-next)
(define-key map "p" #'calibredb-virtual-library-previous)
(define-key map "P" #'calibredb-library-previous)
(define-key map "s" #'calibredb-set-metadata-dispatch)
(define-key map "S" #'calibredb-switch-library)
(define-key map "o" #'calibredb-find-file)
(define-key map "O" #'calibredb-find-file-other-frame)
(define-key map "v" #'calibredb-view)
(define-key map "V" #'calibredb-open-file-with-default-tool)
(define-key map "," #'calibredb-quick-look)
(define-key map "." #'calibredb-dired-open)
(define-key map "y" #'calibredb-yank-dispatch)
(define-key map "b" #'calibredb-catalog-bib-dispatch)
(define-key map "e" #'calibredb-export-dispatch)
(define-key map "r" #'calibredb-search-refresh-and-clear-filter)
(define-key map "R" #'calibredb-search-clear-filter)
(define-key map "q" #'calibredb-search-quit)
(define-key map "m" #'calibredb-mark-and-forward)
(define-key map "f" #'calibredb-toggle-favorite-at-point)
(define-key map "x" #'calibredb-toggle-archive-at-point)
(define-key map "h" #'calibredb-toggle-highlight-at-point)
(define-key map "u" #'calibredb-unmark-and-forward)
(define-key map "i" #'calibredb-edit-annotation)
(define-key map (kbd "<DEL>") #'calibredb-unmark-and-backward)
(define-key map (kbd "<backtab>") #'calibredb-toggle-view)
(define-key map (kbd "TAB") #'calibredb-toggle-view-at-point)
(define-key map "\M-n" #'calibredb-show-next-entry)
(define-key map "\M-p" #'calibredb-show-previous-entry)
(define-key map "/" #'calibredb-search-live-filter)
(define-key map "\M-t" #'calibredb-set-metadata--tags)
(define-key map "\M-a" #'calibredb-set-metadata--author_sort)
(define-key map "\M-A" #'calibredb-set-metadata--authors)
(define-key map "\M-T" #'calibredb-set-metadata--title)
(define-key map "\M-c" #'calibredb-set-metadata--comments)
map)
"Keymap for `calibredb-search-mode'.")
#+end_src
* Annotate PDFs and EPUBs (org-noter)
#+begin_src elisp :tangle no
(use-package org-noter)
#+end_src
#+begin_src elisp :tangle ~/.emacs.d/custom.el
(defvar org-noter-notes-search-path (list (concat org-directory "/library/books")))
(defvar org-noter-default-notes-file-names '("books.org"))
#+end_src
* Link PDFs (org-noter-pdftools)
#+begin_src elisp
(use-package org-noter-pdftools
:after org-noter
:config
;; Add a function to ensure precise note is inserted
(defun org-noter-pdftools-insert-precise-note (&optional toggle-no-questions)
(interactive "P")
(org-noter--with-valid-session
(let ((org-noter-insert-note-no-questions (if toggle-no-questions
(not org-noter-insert-note-no-questions)
org-noter-insert-note-no-questions))
(org-pdftools-use-isearch-link t)
(org-pdftools-use-freepointer-annot t))
(org-noter-insert-note (org-noter--get-precise-info)))))
;; fix https://github.com/weirdNox/org-noter/pull/93/commits/f8349ae7575e599f375de1be6be2d0d5de4e6cbf
(defun org-noter-set-start-location (&optional arg)
"When opening a session with this document, go to the current location.
With a prefix ARG, remove start location."
(interactive "P")
(org-noter--with-valid-session
(let ((inhibit-read-only t)
(ast (org-noter--parse-root))
(location (org-noter--doc-approx-location (when (called-interactively-p 'any) 'interactive))))
(with-current-buffer (org-noter--session-notes-buffer session)
(org-with-wide-buffer
(goto-char (org-element-property :begin ast))
(if arg
(org-entry-delete nil org-noter-property-note-location)
(org-entry-put nil org-noter-property-note-location
(org-noter--pretty-print-location location))))))))
(with-eval-after-load 'pdf-annot
(add-hook 'pdf-annot-activate-handler-functions #'org-noter-pdftools-jump-to-note)
)
)
#+end_src
* View EPUBs (nov.el)
#+begin_src elisp :tangle no
(use-package nov
:config
(add-to-list 'auto-mode-alist '("\\.epub\\'" . nov-mode))
)
#+end_src
* Zotero (helm-bibtex)
#+begin_src elisp :tangle no
(use-package helm-bibtex)
#+end_src
#+begin_src elisp :tangle ~/.emacs.d/custom.el
(defvar bibtex-completion-bibliography '("~/bibliography/zotero.bib"))
#+end_src

46
projects/dotemacs/emacs-misc.org Normal file
View File

@@ -0,0 +1,46 @@
#+TITLE: Miscellaneous Configuration
#+PROPERTY: header-args :tangle ~/.emacs.d/config.el
* Browser (eww)
#+begin_src elisp
(use-package eww
:bind* (("M-m g x" . eww)
("M-m g :" . eww-browse-with-external-browser)
("M-m g #" . eww-list-histories)
("M-m g {" . eww-back-url)
("M-m g }" . eww-forward-url))
:config
(progn
(add-hook 'eww-mode-hook 'visual-line-mode)
)
)
#+end_src
* Manage Docker in Emacs
#+begin_src elisp
(use-package docker
:bind ("C-c d" . docker)
)
#+end_src
* Periodic table of the elements
#+begin_src elisp :tangle no
(use-package chemtable)
#+end_src
* Accounting (beancount)
#+begin_src elisp :tangle no
(use-package beancount
:straight (beancount :type git :host github :repo "beancount/beancount-mode")
:config
(add-to-list 'auto-mode-alist '("\\.beancount\\'" . beancount-mode))
(add-hook 'beancount-mode-hook #'outline-minor-mode)
(define-key beancount-mode-map (kbd "C-c C-n") #'outline-next-visible-heading)
(define-key beancount-mode-map (kbd "C-c C-p") #'outline-previous-visible-heading)
(add-hook 'beancount-mode-hook #'flymake-bean-check-enable)
)
#+end_src

233
projects/dotemacs/emacs-org.org Normal file
View File

@@ -0,0 +1,233 @@
#+TITLE: Org Mode Configuration
#+PROPERTY: header-args :tangle ~/.emacs.d/config.el
* Org Mode
** Basic setup
#+begin_src elisp
(use-package org
:config
(defvar org-outline-path-complete-in-steps nil)
:bind (("C-c l" . org-store-link)
("C-c a" . org-agenda)
("C-c c" . org-capture)
:map org-mode-map)
)
#+end_src
#+begin_src elisp
(defvar org-directory (concat (getenv "HOME") "/org/"))
#+end_src
** Looks
Basic
#+begin_src elisp
(defvar org-pretty-entities t) ; Improve org mode looks
(defvar org-hide-emphasis-markers t) ; Hide emphasis markup
(defvar org-num-mode nil)
(defvar org-startup-folded 'shw2levels)
#+end_src
Indentation of headers
#+begin_src elisp
(defvar org-startup-indented t) ; Indent org heirarchy
(defvar org-adapt-indentation t)
(defvar org-hide-leading-stars t) ; Minimal Outline
(defvar org-odd-levels-only nil)
#+end_src
Indentation of lists
#+begin_src elisp
(setq org-list-demote-modify-bullet t)
#+end_src
Org-modern
#+begin_src elisp
(use-package org-modern
:ensure t
:config
;; Choose some fonts
(set-face-attribute 'default nil :family "sans-serif")
(set-face-attribute 'variable-pitch nil :family "sans-serif")
(set-face-attribute 'org-modern-symbol nil :family "Iosevka")
;; Edit settings
(defvar org-auto-align-tags nil)
(defvar org-tags-column 0)
(defvar org-catch-invisible-edits 'show-and-error)
(defvar org-special-ctrl-a/e t)
(defvar org-insert-heading-respect-content t)
;; Org styling, hide markup etc.
(defvar org-hide-emphasis-markers t)
(defvar org-pretty-entities t)
;; Agenda styling
(defvar org-agenda-tags-column 0)
(defvar org-agenda-block-separator ?─)
(defvar org-agenda-time-grid
'((daily today require-timed)
(800 1000 1200 1400 1600 1800 2000)
" ┄┄┄┄┄ " "┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄"))
(defvar org-agenda-current-time-string
"◀── now ─────────────────────────────────────────────────")
;; Ellipsis styling
(defvar org-ellipsis "…")
(set-face-attribute 'org-ellipsis nil :inherit 'default :box nil)
(global-org-modern-mode)
)
#+end_src
Highlight Sourcecode Syntax
#+begin_src elisp
(setq org-src-fontify-natively t)
(setq org-src-tab-acts-natively t)
#+end_src
Images
#+begin_src elisp
(setq org-startup-with-inline-images t)
(setq org-image-actual-width '(300))
#+end_src
** Capture
#+begin_src elisp
(defvar org-default-notes-file (concat org-directory "/0_inbox/inbox.org"))
#+end_src
*** Org-protocol
Linux configuration
#+begin_src bash :tangle no
[Desktop Entry]
Name=org-protocol
Comment=Intercept calls from emacsclient to trigger custom actions
Categories=Other;
Keywords=org-protocol;
Icon=emacs
Type=Application
Exec=emacsclient -- %u
Terminal=false
StartupWMClass=Emacs
MimeType=x-scheme-handler/org-protocol;
#+end_src
#+begin_src bash :tangle no
update-desktop-database ~/.local/share/applications/
#+end_src
Basic configuration
#+begin_src elisp
(require 'org-protocol)
(setq org-protocol-default-buffer-for-file-links "*scratch*") ; fixes 'no buffers remain to edit error for org-protocol capturer
#+end_src
Org-protocol templates
#+begin_src elisp :tangle ~/.emacs.d/custom.el
(defvar org-capture-templates '(
("p" "Protocol"
entry
(file "0_inbox/inbox.org")
"* %^{Title}\nSource: %u, %c\n #+BEGIN_QUOTE\n%i\n#+END_QUOTE\n\n\n%?"
)
("L" "Protocol Link"
entry
(file "0_inbox/inbox.org")
"* %? [[%:link][%:description]]\n:PROPERTIES:\n:TITLE: %:description\n:URI: %:link\n:CREATED: %U\n:END:"
:prepend nil
:empty-lines 1
:created t
:kill-buffer t
)
)
)
#+end_src
#+begin_src elisp
(setq org-protocol-default-template-key "L")
#+end_src
Convert Orgzly captures to org-protocol captures standard
#+begin_src elisp
(defun my/org-convert-orgzly-to-org-protocol ()
"Reformat Orgzly bookmark at point to org-protocol bookmark."
(interactive)
(when (org-at-heading-p)
(let ((headline (nth 4 (org-heading-components))))
;; Find and store the link. Delete the link line.
(search-forward-regexp "^https?://\\S-*" nil t)
(let ((link (match-string 0)))
(beginning-of-line)
(kill-line)
;; Delete any trailing blank spaces
(org-back-to-heading)
(end-of-line)
(when (not (org-on-heading-p))
(delete-char 1)
)
;; Set new headline
(goto-char (org-entry-beginning-position))
(org-edit-headline (format "[[%s][%s]]" link headline))
;; Set new properties
(org-set-property "TITLE" headline)
(org-set-property "URI" link)
(message "Reformatted Orgzly bookmark at point to org-protocol bookmark")
)
)
)
)
#+end_src
** Exporting
#+begin_src elisp :tangle no
(setq org-export-with-smart-quotes t)
(setq org-export-backends '(beamer html latex md))
#+end_src
Export to EPUB
#+begin_src elisp :tangle no
(use-package ox-epub)
#+end_src
** org-attach
#+begin_src elisp
(defvar org-attach-id-dir (concat org-directory "/library"))
#+end_src
** Enable shell scripting support in org-babel
#+begin_src elisp
(defvar org-babel-do-load-languages 'org-babel-load-languages '((shell . t)))
#+end_src
** Insert org-mode links from clipboard
#+begin_src elisp :tangle no
(use-package org-cliplink
:bind
(("C-x p i" . org-cliplink))
)
#+end_src
** Deft
#+begin_src elisp :tangle no
(use-package deft
:commands (deft)
:init
(defvar deft-extensions '("org"))
(defvar deft-recursive nil)
(defvar deft-use-filename-as-title t)
:config
(defvar deft-directory org-directory)
(defvar deft-recursive t)
(defvar deft-strip-summary-regexp ":PROPERTIES:\n\\(.+\n\\)+:END:\n")
(defvar deft-use-filename-as-title t)
:bind ("C-c n d" . deft)
)
#+end_src

140
projects/dotemacs/emacs-roam.org Normal file
View File

@@ -0,0 +1,140 @@
#+TITLE: Org-Roam Configuration
#+PROPERTY: header-args :tangle ~/.emacs.d/config.el
* Org-roam
** Basic org-roam setup
#+begin_src elisp
(use-package org-roam
:init (setq org-roam-v2-ack t) ;; Acknowledge V2 upgrade
:after org
:config
(org-roam-db-autosync-enable)
(require 'org-roam-dailies)
:bind (
("C-c n f" . org-roam-node-find)
("C-c n g" . org-roam-graph)
("C-c n r" . org-roam-node-random)
("C-c n h" . org-roam-node-convert-headline)
("C-c n i" . org-roam-node-insert)
("C-c n o" . org-id-get-create)
("C-c n t" . org-roam-tag-add)
("C-c n a" . org-roam-alias-add)
("C-c n l" . org-roam-buffer-display-dedicated)
)
)
#+end_src
#+begin_src elisp
(setq org-roam-directory (concat org-directory "/1_thinking"))
(setq org-roam-dailies-directory (concat org-directory "/0_inbox/daily"))
#+end_src
#+begin_src elisp :tangle no
(use-package sqlite3)
(require 'sqlite3)
#+end_src
Include subdirectories in org-roam
#+begin_src elisp
(setq org-roam-file-exclude-regexp "^[.][.]?/")
#+end_src
** Display in org-roam-buffer
#+begin_src elisp :tangle no
(setq org-roam-mode-sections
(list #'org-roam-backlinks-section
#'org-roam-reflinks-section
#'org-roam-unlinked-references-section
)
)
#+end_src
** Filter org-roam nodes find by tag
#+begin_src elisp :tangle no
(defun my/org-roam-node-has-tag (node tag)
"Filter function to check if the given NODE has the specified TAG."
(member tag (org-roam-node-tags node))
)
(defun my/org-roam-node-find-by-tag ()
"Find and open an Org-roam node based on a specified tag."
(interactive)
(let ((tag (read-string "Enter tag: ")))
(org-roam-node-find nil nil (lambda (node) (my/org-roam-node-has-tag node tag))))
)
#+end_src
** org-roam-capture templates
#+begin_src elisp
(setq org-roam-capture-templates
'(
("L" "link" plain
(function org-roam--capture-get-point)
"%?"
:file-name "web/%<%Y-%m-%dT%H%M%S>.org"
:head "#+TITLE: ${title}\n#+CREATED: %<%Y-%m-%dT%H%M%S>"
:immediate-finish t
:unnarrowed t
)
("h" "hugo post" plain
"%?"
:target (file+head "posts/${slug}.org"
"#+TITLE: ${title}\n#+DATE: %U\n#+HUGO_BASE_DIR: ~/gharbeia.net\n#+HUGO_SECTION: ./posts\n#+HUGO_AUTO_SET_LASTMOD: t\n#+HUGO_TAGS: article\n#+HUGO_DRAFT: true\n")
:immediate-finish t
:unnarrowed t
)
("p" "person" plain
"%?"
:if-new (file+head "people/${slug}.org"
"#+TITLE: ${title}")
:immediate-finish t
:unnarrowed t
)
)
)
#+end_src
#+begin_src elisp
(setq org-roam-dailies-capture-templates
'(
("d" "daily" plain
""
:target ("file+heaed %<%Y-%m-%d>.org" "#+title: %<%Y-%m-%d>\n\n")
:immediate-finish t
)
)
)
#+end_src
** Move org header to org-roam-daily
#+begin_src elisp :tangle no
(defun my/org-move-entry-to-daily-notes ()
"Move the current org-mode headline to the daily notes file based on its :CREATED: property."
(interactive)
(let*
(
(created-prop (org-entry-get nil "CREATED"))
(created-date (when created-prop
(org-parse-time-string created-prop)))
(year (nth 5 created-date)) ; Extract year (6th element)
(month (nth 4 created-date)) ; Extract month (5th element)
(day (nth 3 created-date)) ; Extract day (4th element)
(target-date (format "%04d-%02d-%02d" year month day)) ; Format date string
(target-file (org-roam-dailies-goto target-date))
)
(when target-file
(org-cut-subtree)
(find-file target-file)
(goto-char (point-max))
(unless (bolp) (newline))
(org-paste-subtree)
)
)
)
#+end_src

73
projects/dotemacs/emacs-shell.org Normal file
View File

@@ -0,0 +1,73 @@
#+TITLE: Shell Configuration
#+PROPERTY: header-args :tangle ~/.emacs.d/config.el
* Shell
** Bash completion
#+begin_src elisp
(use-package bash-completion
:config
(require 'bash-completion)
(bash-completion-setup)
)
#+end_src
#+begin_src elisp
(defvar shell-dynamic-complete-functions t)
#+end_src
** Eshell
Add programmable bash completion to Emacs shell-mode
#+begin_src elisp :tangle no
(require 'bash-completion)
(add-hook 'eshell-mode-hook
(lambda ()
(add-hook 'completion-at-point-functions
'bash-completion-capf-nonexclusive nil t
)
)
)
#+end_src
Use colors in eshell
#+begin_src elisp :tangle no
(use-package xterm-color
:commands (xterm-color-filter)
)
(use-package eshell
:after xterm-color
:config
(define-key eshell-hist-mode-map (kbd "M-r") #'consult-history)
(add-hook 'eshell-mode-hook
(lambda ()
(setenv "TERM" "xterm-256color")))
(add-hook 'eshell-before-prompt-hook (setq xterm-color-preserve-properties t))
(add-to-list 'eshell-preoutput-filter-functions 'xterm-color-filter)
(setq eshell-output-filter-functions
(remove 'eshell-handle-ansi-color eshell-output-filter-functions)
)
)
#+end_src
Eshell completion
#+begin_src elisp :tangle no
(add-hook 'eshell-mode-hook
(lambda ()
(add-hook 'completion-at-point-functions
'bash-completion-capf-nonexclusive nil t)))
#+end_src
Emulate A Terminal (EAT)
#+begin_src elisp :tangle no
(use-package eat
:config
;; For `eat-eshell-mode'.
(add-hook 'eshell-load-hook #'eat-eshell-mode)
;; For `eat-eshell-visual-command-mode'.
(add-hook 'eshell-load-hook #'eat-eshell-visual-command-mode)
)
#+end_src

255
projects/dotemacs/emacs-writing.org Normal file
View File

@@ -0,0 +1,255 @@
#+TITLE: Reading and Writing Configuration
#+PROPERTY: header-args :tangle ~/.emacs.d/config.el
* Text and Case
** Convert DOuble capitals to single capitals
#+begin_src elisp :tangle no
(defun my/dcaps-to-scaps ()
"Convert word in DOuble CApitals to Single Capitals."
(interactive)
(and (= ?w (char-syntax (char-before)))
(save-excursion
(and (if (called-interactively-p)
(skip-syntax-backward "w")
(= -3 (skip-syntax-backward "w"))
)
(let (case-fold-search)
(looking-at "\\b[[:upper:]]\\{2\\}[[:lower:]]")
)
(capitalize-word 1)
)
)
)
)
#+end_src
Then, let’s define a minor mode for it to be activated.
#+begin_src elisp :tangle no
(define-minor-mode my-dubcaps-mode
"Toggle 'my-dubcaps-mode' and convert words in DOuble CApitals to Single Capitals as you type."
:init-value nil
:lighter (" DC")
(if my-dubcaps-mode
(add-hook 'post-self-insert-hook #'my/dcaps-to-scaps nil 'local)
(remove-hook 'post-self-insert-hook #'my/dcaps-to-scaps 'local)))
#+end_src
Finally, let’s add a hook so that it is on for all the text files Emacs opens.
#+begin_src elisp :tangle no
(add-hook 'text-mode-hook #'my-dubcaps-mode)
#+end_src
Also, since we add a minor mode string (it might be useful sometimes), currently I prefer to diminish it.
#+begin_src elisp :tangle no
(defun my/diminish-dubcaps ()
(interactive)
(diminish 'my-dubcaps-mode ""))
(add-hook 'my-dubcaps-mode-hook 'my/diminish-dubcaps)
#+end_src
* Reading and Writing
** Move correctly over camelCased words
#+begin_src elisp
(subword-mode)
#+end_src
** Understand the more common sentence with double space
#+begin_src elisp
(setq sentence-end-double-space nil)
#+end_src
** Join lines into paragraph
#+begin_src elisp
(defun my/fill-or-unfill-paragraph (&optional unfill region)
"Fill paragraph (or REGION). With the prefix argument UNFILL, fill it instead."
(interactive (progn
(barf-if-buffer-read-only)
(list (if current-prefix-arg 'fill) t)))
(let ((fill-column (if unfill fill-column (point-max))))
(fill-paragraph nil region)))
(bind-key "M-q" 'my/fill-or-unfill-paragraph)
#+end_src
#+begin_src elisp
(defun my/fill-or-unfill-all-paragraphs (&optional unfill)
"Fill or unfill all paragraphs in the current buffer.
With the prefix argument UNFILL, fill them instead."
(interactive (list (if current-prefix-arg 'fill)))
(let ((fill-column (if unfill fill-column (point-max))))
(save-excursion
(goto-char (point-min))
(while (not (eobp))
(fill-paragraph nil t)
(forward-paragraph)))))
(bind-key "M-Q" 'my/fill-or-unfill-all-paragraphs)
#+end_src
#+begin_src elisp
(remove-hook 'text-mode-hook #'turn-on-auto-fill)
(add-hook 'text-mode-hook 'turn-on-visual-line-mode)
#+end_src
** Expand some words with auto-correct
#+begin_src elisp :tangle no
(setq save-abbrevs 'silently)
(setq-default abbrev-mode t)
#+end_src
** ediff
#+begin_src elisp :tangle no
(setq ediff-window-setup-function 'ediff-setup-windows-plain)
(setq ediff-split-window-function 'split-window-horizontally)
#+end_src
** tramp
#+begin_src elisp :tangle no
(setq tramp-default-method "ssh"
tramp-backup-directory-alist backup-directory-alist
tramp-ssh-controlmaster-options "ssh")
#+end_src
** Clean up space
#+begin_src elisp :tangle no
(bind-key "M-SPC" 'cycle-spacing)
#+end_src
** Transform <a href> links into org links
#+begin_src elisp :tangle no
(defun my/transform-html-links-to-org ()
"Transform all HTML <a> links in the current buffer into 'org-mode' links."
(interactive)
(goto-char (point-min))
(while (re-search-forward "<a href=\"\\(.*?\\)\">\\(.*?\\)</a>" nil t)
(replace-match (org-make-link-string (match-string 1) (match-string 2)))))
#+end_src
** Count words per minute
#+begin_src elisp :tangle no
(require 'org-clock)
(defun my/org-entry-wpm ()
(interactive)
(save-restriction
(save-excursion
(org-narrow-to-subtree)
(goto-char (point-min))
(let* ((words (count-words-region (point-min) (point-max)))
(minutes (org-clock-sum-current-item))
(wpm (/ words minutes)))
(message "WPM: %d (words: %d, minutes: %d)" wpm words minutes)
(kill-new (number-to-string wpm))
)
)
)
)
#+end_src
** Enable dict mode
#+begin_src elisp :tangle no
(setq dictionary-server "automatic")
#+end_src
** Pick out passive voice and weasel words
#+begin_src elisp :tangle no
(use-package writegood-mode
:diminish writegood-mode
:config
(progn (add-hook 'text-mode-hook 'writegood-mode))
)
#+end_src
** Org-babel docker
#+begin_src elisp :tangle no
(use-package ob-docker-build
:straight (ob-docker-build :type git :host github :repo "ifitzpat/ob-docker-build")
:defer t
:config
(add-to-list 'org-babel-load-languages '(docker-build . t))
(org-babel-do-load-languages 'org-babel-load-languages org-babel-load-languages)
)
#+end_src
* Spelling and syntax
** Spell checking
This requires installation of hunspell
#+begin_src bash :tangle no
sudo apt install hunspell
#+end_src
#+begin_src elisp
(use-package flyspell
:config (setq ispell-program-name "hunspell"
ispell-default-dictionary "en_US"
)
:diminish (flyspell-mode . "φ")
:hook (text-mode . flyspell-mode)
:bind (
("M-<f7>" . flyspell-buffer)
("<f7>" . flyspell-word)
("C-;" . flyspell-auto-correct-previous-word)
)
)
#+end_src
** Flyspell correct
#+begin_src elisp :tangle no
(use-package flyspell-correct
:after flyspell
:bind (:map flyspell-mode-map ("C-;" . flyspell-correct-wrapper))
)
#+end_src
** Flycheck
Needs external checkers installed
#+begin_src elisp
(use-package flycheck
:init (global-flycheck-mode)
:diminish (flycheck-mode . "")
:config
(add-hook 'after-init-hook #'global-flycheck-mode)
(setq flycheck-emacs-lisp-load-path 'inherit)
(setq flycheck-emacs-lisp-load-path (concat user-emacs-directory "straight/build"))
)
#+end_src
** Flycheck bash
#+begin_src bash :tangle no
sudo apt install devscripts
#+end_src
#+begin_src elisp :tangle no
(use-package flycheck-checkbashisms
:config
(flycheck-checkbashisms-setup)
)
#+end_src
** Yaml
#+begin_src elisp :tangle no
(use-package yaml-mode
:config
(add-to-list 'auto-mode-alist '("\\.yml\\'" . yaml-mode))
(add-to-list 'auto-mode-alist '("\\.yaml\\'" . yaml-mode))
)
#+end_src
** Docker
#+begin_src elisp :tangle no
(use-package docker-compose-mode)
#+end_src

2138
projects/dotemacs/emacs.org Normal file
View File

File diff suppressed because it is too large Load Diff

21
projects/dotemacs/modules/emacs-ai.org Normal file
View File

@@ -0,0 +1,21 @@
#+TITLE: Emacs AI Configuration
#+property: header-args :tangle ~/.emacs.d/modules/ai.el
* ellama
#+begin_src elisp
(use-package ellama
:ensure t
:bind ("C-c e" . ellama)
:hook (org-ctrl-c-ctrl-c-final . ellama-chat-send-last-message)
:init (setopt ellama-auto-scroll t)
:config
(ellama-context-header-line-global-mode +1)
(ellama-session-header-line-global-mode +1)
)
#+end_src
* Providers
#+begin_src elisp
(setq llm-debug t)
;; Note: API keys should be handled via auth-source as seen in original config
#+end_src

55
projects/dotemacs/modules/emacs-core.org Normal file
View File

@@ -0,0 +1,55 @@
#+TITLE: Emacs Core Configuration
#+property: header-args :tangle ~/.emacs.d/modules/core.el
* early-init.el
For straight.el to pick up before package.el
#+begin_src elisp :tangle ~/.emacs.d/early-init.el
(setq package-enable-at-startup nil)
#+end_src
* Straight.el Bootstrap
#+begin_src elisp :tangle ~/.emacs
(setq straight-repository-branch "develop")
(eval-and-compile
(defvar bootstrap-version)
(let ((bootstrap-file
(expand-file-name "straight/repos/straight.el/bootstrap.el"
(or (bound-and-true-p straight-base-dir)
user-emacs-directory)))
(bootstrap-version 7))
(unless (file-exists-p bootstrap-file)
(with-current-buffer
(url-retrieve-synchronously "https://raw.githubusercontent.com/radian-software/straight.el/develop/install.el" 'silent 'inhibit-cookies)
(goto-char (point-max))
(eval-print-last-sexp)))
(load bootstrap-file nil 'nomessage))
(straight-use-package 'use-package)
)
(setq straight-use-package-by-default t)
#+end_src
* Server and Performance
#+begin_src elisp :tangle ~/.emacs.d/early-init.el
(require 'server)
(unless (server-running-p) (server-start))
(defvar server-max-buffers 100)
#+end_src
#+begin_src elisp
(setq gc-cons-threshold (* 500 1024 1024))
(add-hook 'after-init-hook (lambda () (setq gc-cons-threshold (* 5 1024 1024))))
#+end_src
* System Information
#+begin_src elisp :tangle ~/.emacs.d/custom.el
(defvar my-laptop-p (equal (system-name) "lilitop"))
(defvar my-server-p (and (equal (system-name) "localhost") (equal user-login-name "root")))
(defvar my-phone-p (not (null (getenv "ANDROID_ROOT")))
"If non-nil, GNU Emacs is running on Termux.")
(when my-phone-p (defvar gnutls-algorithm-priority "NORMAL:-VERS-TLS1.3"))
(global-auto-revert-mode)
(savehist-mode)
(desktop-save-mode t)
#+end_src

48
projects/dotemacs/modules/emacs-gtd.org Normal file
View File

@@ -0,0 +1,48 @@
#+TITLE: Emacs GTD Configuration
#+property: header-args :tangle ~/.emacs.d/modules/gtd.el
* org-gtd
#+begin_src elisp
(use-package org-gtd
:defer t
:init (setq org-gtd-update-ack "3.0.0")
:after org
:config
(setq org-edna-use-inheritance t)
(org-edna-mode)
:bind (
("C-c d c" . org-gtd-capture)
("C-c d e" . org-gtd-engage)
("C-c d p" . org-gtd-process-inbox)
:map org-gtd-clarify-map
("C-c c" . org-gtd-organize)
)
)
#+end_src
* GTD Directory and Areas
#+begin_src elisp
(defvar org-gtd-directory org-directory)
(defvar org-gtd-organize-hooks '(org-gtd-set-area-of-focus))
(defvar org-gtd-areas-of-focus '(
"Atoms"
"Bits"
"Cells"
"Flags"
"Business"
"Wealth"
"Learning"
"Skills"
"Privacy"
"Archive"
"Library"
"Writing"
"Health"
"Home"
"Family"
"Social"
"Egypt"
)
)
(defvar org-gtd-clarify-show-horizons 'right)
#+end_src

39
projects/dotemacs/modules/emacs-media.org Normal file
View File

@@ -0,0 +1,39 @@
#+TITLE: Emacs Media and E-books Configuration
#+property: header-args :tangle ~/.emacs.d/modules/media.el
* calibredb
#+begin_src elisp
(use-package calibredb
:defer t
:config
(setq calibredb-format-all-the-icons t)
(setq calibredb-format-icons-in-terminal t)
)
(defvar calibredb-root-dir (concat (getenv "HOME") "/library/books"))
(defvar calibredb-db-dir (expand-file-name "metadata.db" calibredb-root-dir))
(defvar calibredb-id-width 6)
(defvar calibredb-title-width 100)
(defvar calibredb-author-width 20)
#+end_src
* nov.el (EPUB Viewer)
#+begin_src elisp
(use-package nov
:config
(add-to-list 'auto-mode-alist '("\\.epub\\'" . nov-mode))
)
#+end_src
* org-noter and PDF Tools
#+begin_src elisp
(use-package org-noter)
(use-package org-noter-pdftools
:after org-noter
:config
(with-eval-after-load 'pdf-annot
(add-hook 'pdf-annot-activate-handler-functions #'org-noter-pdftools-jump-to-note)
)
)
#+end_src

69
projects/dotemacs/modules/emacs-org.org Normal file
View File

@@ -0,0 +1,69 @@
#+TITLE: Emacs Org-mode Configuration
#+property: header-args :tangle ~/.emacs.d/modules/org.el
* Core Org Setup
#+begin_src elisp
(use-package org
:config
(defvar org-outline-path-complete-in-steps nil)
:bind (("C-c l" . org-store-link)
("C-c a" . org-agenda)
("C-c c" . org-capture)
:map org-mode-map)
)
(defvar org-directory (concat (getenv "HOME") "/org/"))
#+end_src
* Agenda
#+begin_src elisp
(setq org-deadline-warning-days 7)
(setq org-agenda-skip-additional-timestamps-same-entry t)
(setq org-agenda-span 'fortnight)
(setq org-agenda-tags-column 'auto)
(setq org-agenda-skip-scheduled-if-deadline-is-shown t)
(setq org-agenda-files (list
(concat org-directory "/0_inbox/inbox.org")
(concat org-directory "/0_inbox/org-gtd-tasks.org")
)
)
#+end_src
* Capture and Protocol
#+begin_src elisp
(require 'org-protocol)
(setq org-protocol-default-buffer-for-file-links "*scratch*")
(defvar org-default-notes-file (concat org-directory "/0_inbox/inbox.org"))
(setq org-protocol-default-template-key "L")
#+end_src
#+begin_src elisp :tangle ~/.emacs.d/custom.el
(defvar org-capture-templates '(
("p" "Protocol"
entry
(file "0_inbox/inbox.org")
"* %^{Title}\nSource: %u, %c\n #+BEGIN_QUOTE\n%i\n#+END_QUOTE\n\n\n%?"
)
("L" "Protocol Link"
entry
(file "0_inbox/inbox.org")
"* %? [[%:link][%:description]]\n:PROPERTIES:\n:TITLE: %:description\n:URI: %:link\n:CREATED: %U\n:END:"
:prepend nil
:empty-lines 1
:created t
:kill-buffer t
)
)
)
#+end_src
* TODO Settings
#+begin_src elisp
(setq org-todo-keywords
'(
(sequence "TODO(t)" "NEXT(n)" "|" "DONE(d!)")
(sequence "WAIT(w@/!)" "|" "CNCL(c@)")
)
)
(setq org-enforce-todo-dependencies t)
(setq org-log-into-drawer "LOGBOOK")
#+end_src

77
projects/dotemacs/modules/emacs-roam.org Normal file
View File

@@ -0,0 +1,77 @@
#+TITLE: Emacs Org-roam Configuration
#+property: header-args :tangle ~/.emacs.d/modules/roam.el
* org-roam Setup
#+begin_src elisp
(use-package org-roam
:init (setq org-roam-v2-ack t)
:after org
:config
(org-roam-db-autosync-enable)
(require 'org-roam-dailies)
(setq org-roam-mode-sections
(list #'org-roam-backlinks-section
#'org-roam-reflinks-section
#'org-roam-unlinked-references-section
)
)
:bind (
("C-c n f" . org-roam-node-find)
("C-c n g" . org-roam-graph)
("C-c n r" . org-roam-node-random)
("C-c n h" . org-roam-node-convert-headline)
("C-c n i" . org-roam-node-insert)
("C-c n o" . org-id-get-create)
("C-c n t" . org-roam-tag-add)
("C-c n a" . org-roam-alias-add)
("C-c n l" . org-roam-buffer-display-dedicated)
)
)
#+end_src
* Directories
#+begin_src elisp
(setq org-roam-directory (concat org-directory "/1_thinking"))
(setq org-roam-dailies-directory (concat org-directory "/0_inbox/daily"))
(setq org-roam-file-exclude-regexp "^[.][.]?/")
#+end_src
* Capture Templates
#+begin_src elisp
(setq org-roam-capture-templates
'(
("L" "link" plain
(function org-roam--capture-get-point)
"%?"
:file-name "web/%<%Y-%m-%dT%H%M%S>.org"
:head "#+TITLE: ${title}\n#+CREATED: %<%Y-%m-%dT%H%M%S>"
:immediate-finish t
:unnarrowed t
)
("h" "hugo post" plain
"%?"
:target (file+head "posts/${slug}.org"
"#+TITLE: ${title}\n#+DATE: %U\n#+HUGO_BASE_DIR: ~/gharbeia.net\n#+HUGO_SECTION: ./posts\n#+HUGO_AUTO_SET_LASTMOD: t\n#+HUGO_TAGS: article\n#+HUGO_DRAFT: true\n")
:immediate-finish t
:unnarrowed t
)
("p" "person" plain
"%?"
:if-new (file+head "people/${slug}.org"
"#+TITLE: ${title}")
:immediate-finish t
:unnarrowed t
)
)
)
(setq org-roam-dailies-capture-templates
'(
("d" "daily" plain
""
:target ("file+heaed %<%Y-%m-%d>.org" "#+title: %<%Y-%m-%d>\n\n")
:immediate-finish t
)
)
)
#+end_src

18
projects/dotemacs/modules/emacs-shell.org Normal file
View File

@@ -0,0 +1,18 @@
#+TITLE: Emacs Shell Configuration
#+property: header-args :tangle ~/.emacs.d/modules/shell.el
* Bash Completion
#+begin_src elisp
(use-package bash-completion
:config
(require 'bash-completion)
(bash-completion-setup)
)
(defvar shell-dynamic-complete-functions t)
#+end_src
* Frame Management
#+begin_src elisp
(add-hook 'server-done-hook (lambda () (delete-frame)))
#+end_src

59
projects/dotemacs/modules/emacs-ui.org Normal file
View File

@@ -0,0 +1,59 @@
#+TITLE: Emacs UI Configuration
#+property: header-args :tangle ~/.emacs.d/modules/ui.el
* Appearance
#+begin_src elisp
(defvar org-pretty-entities t) ; Improve org mode looks
(defvar org-hide-emphasis-markers t) ; Hide emphasis markup
(defvar org-num-mode nil)
(defvar org-startup-folded 'shw2levels)
(defvar org-startup-indented t) ; Indent org heirarchy
(defvar org-adapt-indentation t)
(defvar org-hide-leading-stars t) ; Minimal Outline
(defvar org-odd-levels-only nil)
#+end_src
* Org-modern
#+begin_src elisp
(use-package org-modern
:ensure t
:config
;; Choose some fonts
(set-face-attribute 'default nil :family "sans-serif")
(set-face-attribute 'variable-pitch nil :family "sans-serif")
(set-face-attribute 'org-modern-symbol nil :family "Iosevka")
;; Edit settings
(defvar org-auto-align-tags nil)
(defvar org-tags-column 0)
(defvar org-catch-invisible-edits 'show-and-error)
(defvar org-special-ctrl-a/e t)
(defvar org-insert-heading-respect-content t)
;; Org styling, hide markup etc.
(defvar org-hide-emphasis-markers t)
(defvar org-pretty-entities t)
;; Agenda styling
(defvar org-agenda-tags-column 0)
(defvar org-agenda-block-separator ?─)
(defvar org-agenda-time-grid
'((daily today require-timed)
(800 1000 1200 1400 1600 1800 2000)
" ┄┄┄┄┄ " "┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄"))
(defvar org-agenda-current-time-string
"◀── now ─────────────────────────────────────────────────")
;; Ellipsis styling
(defvar org-ellipsis "…")
(set-face-attribute 'org-ellipsis nil :inherit 'default :box nil)
(global-org-modern-mode)
)
#+end_src
* Syntax Highlighting
#+begin_src elisp
(setq org-src-fontify-natively t)
(setq org-src-tab-acts-natively t)
#+end_src

48
projects/dotemacs/modules/emacs-writing.org Normal file
View File

@@ -0,0 +1,48 @@
#+TITLE: Emacs Writing Configuration
#+property: header-args :tangle ~/.emacs.d/modules/writing.el
* Spell Checking
#+begin_src elisp
(use-package flyspell
:config (setq ispell-program-name "hunspell"
ispell-default-dictionary "en_US"
)
:diminish (flyspell-mode . "φ")
:hook (text-mode . flyspell-mode)
:bind (
("M-<f7>" . flyspell-buffer)
("<f7>" . flyspell-word)
("C-;" . flyspell-auto-correct-previous-word)
)
)
#+end_src
* Syntax Checking
#+begin_src elisp
(use-package flycheck
:init (global-flycheck-mode)
:diminish (flycheck-mode . "")
:config
(add-hook 'after-init-hook #'global-flycheck-mode)
(setq flycheck-emacs-lisp-load-path 'inherit)
(setq flycheck-emacs-lisp-load-path (concat user-emacs-directory "straight/build"))
)
#+end_src
* Text Manipulation
#+begin_src elisp
(subword-mode)
(setq sentence-end-double-space nil)
(defun my/fill-or-unfill-paragraph (&optional unfill region)
"Fill paragraph (or REGION). With the prefix argument UNFILL, fill it instead."
(interactive (progn
(barf-if-buffer-read-only)
(list (if current-prefix-arg 'fill) t)))
(let ((fill-column (if unfill fill-column (point-max))))
(fill-paragraph nil region)))
(bind-key "M-q" 'my/fill-or-unfill-paragraph)
(add-hook 'text-mode-hook 'turn-on-visual-line-mode)
#+end_src

27
projects/infrastructure/README.org Normal file
View File

@@ -0,0 +1,27 @@
#+TITLE: Infrastructure
#+AUTHOR: Amr
#+CREATED: [2026-03-17 Tue]
#+BEGIN_COMMENT
IT infrastructure documentation, security hardening, and operational management.
#+END_COMMENT
* Infrastructure
Documentation and management of IT infrastructure, cloud resources, and security posture.
* Project Tasks
See the actionable tasks for this project in [[file:../../gtd.org::*Infrastructure][GTD.org > Projects > Infrastructure]]
* Key Documents
- Security audit reports
- Infrastructure inventory
- Budget analysis
- Account management SOPs
* Current Focus
- Current state assessment
- Risk and vulnerability reporting
- 30/60/90 day roadmap planning

150
projects/infrastructure/interview-questions.org Normal file
View File

@@ -0,0 +1,150 @@
#+TITLE: Sol Enterprise IT - Owner Interview
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:28]
#+DATE: 2026-03-06
#+FILETAGS: :interview:assessment:infrastructure
* Sol Enterprise IT - Owner Interview
** Interview Date: [TO BE SCHEDULED]
** Owner/Stakeholder: You
** Interviewer: Amero Garcia (Sol)
** Duration: 60-90 minutes
** Location: This conversation (Signal/OpenClaw)
---
**PART 1: AI/LLM Infrastructure (15 min)
*** Q1. Current AI Usage
What AI models/providers are you currently using? (e.g., Claude, GPT-4, Gemini)
*** Q2. Monthly AI Budget
What's your current or target monthly budget for AI inference? (We've discussed $50 - is this firm?)
*** Q3. Token Consumption
Do you track token usage? Any concerns about costs growing?
*** Q4. Performance Needs
Are you satisfied with current response speed/quality, or need improvements?
*** Q5. Local vs Cloud
Any interest in local inference (hardware costs upfront, zero ongoing) vs cloud APIs?
---
**PART 2: Security Infrastructure (15 min)
*** Q6. Security Priority
How would you rate security priority? (Critical/High/Medium/Low)
*** Q7. Risk Tolerance
Are you comfortable with current credential handling, or want stricter isolation?
*** Q8. Audit Needs
Do you want audit logs of all credential access and system changes?
*** Q9. Data Sensitivity
How sensitive is the data we work with? Any compliance requirements?
*** Q10. Incident Response
If a security issue occurs, what's your preferred response protocol?
---
**PART 3: Online Accounts & Services (15 min)
*** Q11. Account Inventory
What online accounts/services do you actively use? (X, LinkedIn, AWS, etc.)
*** Q12. Credential Management
How do you prefer credentials stored and accessed?
*** Q13. Access Frequency
How often do you need to access these accounts programmatically?
*** Q14. X/Twitter Account
Status of X bookmarks export - any blockers?
*** Q15. API Keys & Tokens
Current OAuth/API key management - working well or needs improvement?
---
**PART 4: System Integration & Automation (15 min)
*** Q16. OpenClaw Gateway
How's the local OpenClaw instance performing? Any issues?
*** Q17. Signal Integration
Signal messaging - working well? Any features needed?
*** Q18. Browser Automation
Browser/CDP automation - reliable enough for your needs?
*** Q19. Cron/Scheduling
Scheduled tasks (social listening, etc.) - working as expected?
*** Q20. Git Repositories
Git workflow for ~/mind and ~/.openclaw - smooth or needs improvement?
---
**PART 5: Growth & Future State (15 min)
*** Q21. Short-term Goals (1-3 months)
What are your top 3 priorities for next 3 months?
*** Q22. Medium-term Vision (3-6 months)
Where do you see Sol Enterprise in 6 months?
*** Q23. Budget Growth
Is $50/month for AI inference a long-term target, or should we plan for growth?
*** Q24. New Capabilities
Any new capabilities you want to add? (More platforms, more automation, etc.)
*** Q25. Success Definition
What does "successful IT infrastructure" look like to you?
---
**PART 6: Preferences & Process (15 min)
*** Q26. Communication
Preferred check-in frequency? (Daily/weekly/as-needed)
*** Q27. Documentation
How detailed should documentation be? (High-level vs step-by-step)
*** Q28. Decision Making
For IT decisions, do you want approval gates or trust my judgment?
*** Q29. Maintenance Windows
Any preferred times for system maintenance/updates?
*** Q30. Emergency Protocol
If something breaks, preferred escalation path?
---
**POST-INTERVIEW: Immediate Action Items
| Priority | Item | Owner | Due Date |
|----------|------|-------|----------|
| | | | |
**30/60/90 Day Roadmap
- 30 Days:
- 60 Days:
- 90 Days:
---
**Notes & Observations
*To be filled during interview*
**Owner Signature: _________________
**Date: _________________

31
projects/infrastructure_project_documentation.org Normal file
View File

@@ -0,0 +1,31 @@
#+title: Infrastructure Project Documentation
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:05]
#+begin_comment
Project documentation for Infrastructure Project Documentation
#+end_comment
* Infrastructure Project Documentation
This file will serve as the central repository for documenting Amr's IT infrastructure, including Proxmox, Virtual Machines (VMs), LXC containers, and Unify network components.
*Purpose:** To create a comprehensive overview for future upgrades, maintenance, and strategic planning.
*Current Status:** Documentation initiated. No prior information found in Memex.
*Information Required from Amr:**
- Access credentials or explicit instructions on how to access system configurations for:
- Proxmox (host details, version, cluster setup, storage configurations)
- All running VMs (OS, purpose, resources, network config)
- All running LXC containers (OS, purpose, resources, network config)
- Unify Controller (version, network topology, device list, configuration details)
- Any existing diagrams, schematics, or previous documentation.
- Specific areas of focus or concern for the upcoming upgrades.
*Next Steps:**
1. Gather detailed information from Amr.
2. Structure documentation logically (e.g., by component: Proxmox, VMs, LXC, Unify).
3. Outline current state, dependencies, and potential upgrade paths.
4. Identify existing files/references in the workspace that need renaming from "IT" to "infrastructure" once the scope is clear.

40
projects/lisp_machine_bootstrap/README.org Normal file
View File

@@ -0,0 +1,40 @@
#+TITLE: Lisp Machine Bootstrap
#+AUTHOR: Amr
#+CREATED: [2026-03-22 Sun]
#+BEGIN_COMMENT
The "Endgame": Bootstrapping a true, hardware-native Lisp machine to achieve ultimate digital sovereignty.
#+END_COMMENT
* Vision: The Sovereign Silicon
The Lisp Machine Bootstrap project aims to remove the "Unix/C Tax"—the layers of opaque C code, complex Unix kernels, and generic hardware that currently underpin modern computing. By building a machine where Lisp is the native language from the gates up to the UI, we create a system that is provably secure, homoiconic, and entirely under user sovereignty.
* Philosophy: Tagged, Homoiconic, and Bare-Metal
- **Hardware-Native Lisp:** Instruction Set Architecture (ISA) optimized for Lisp (CAR, CDR, CONS as hardware instructions).
- **Tagged Memory:** Memory management handled by the hardware, preventing buffer overflows and memory corruption by design.
- **Removing the C Core:** Eliminating the reliance on C-based kernels. The "Kernel" is a small Lisp bootstrapper.
- **FPGA First:** Utilizing Field-Programmable Gate Arrays (FPGAs) as the initial prototyping environment.
* The Bootstrap Path
1. **Phase 1: Soft Machine (Current):** Emacs/CL running on Linux (The "Simulator").
2. **Phase 2: Virtual Machine:** Develop a specialized Lisp VM that abstracts away the Linux kernel.
3. **Phase 3: FPGA Implementation:** Port the VM to an FPGA core (Verilog/VHDL).
4. **Phase 4: Sovereign Silicon:** Synthesize to a custom RISC-V or dedicated Lisp ASIC.
* Initial Research & Tasks
See the actionable tasks for this project in [[file:../../gtd.org::*Lisp Machine Bootstrap][GTD.org > Projects > Lisp Machine Bootstrap]]
* Status
- [X] Project Seeded
- [ ] Research existing Lisp-on-FPGA implementations (e.g., Openora, Symbolics replicas)
- [ ] Define minimum hardware-native Lisp ISA
- [ ] Draft initial Verilog/VHDL skeleton
* Links
- [[file:../org-agent/][Orchestration: org-agent Microkernel]]
- [[file:../agora/][Social Layer: Agora Protocol]]

35
projects/modular_home_appliances/README.org Normal file
View File

@@ -0,0 +1,35 @@
#+title: Modular Home Appliances Project
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:05]
#+begin_comment
Project documentation for Modular Home Appliances Project
#+end_comment
* Modular Home Appliances Project
*Goal:** To develop a set of open-source designs for modular home appliances (washers, dryers, stoves, fridges, freezers, dishwashers) that incorporate ESP32-based smart interfaces. These interfaces will be controllable by AI, smartphone apps, or movable, modular physical controllers, and integrate seamlessly with Home Assistant.
*Key Concepts:**
- *Open Source Designs:** Appliance designs will be publicly available and modifiable.
- *Modularity:** Appliances designed with easily replaceable or upgradeable components.
- *Smart Interfaces:** Utilizing ESP32 for connectivity and intelligent control.
- *Control Methods:** AI-driven, smartphone applications, and physical modular controllers.
- *Integration:** Seamless operation with Home Assistant.
- *Inspiration:** Learning from the principles and lessons derived from Slate electric trucks.
*Initial Scope:**
- Defining the modular architecture for various appliances.
- Researching existing open-source appliance projects and smart home integration standards.
- Identifying suitable ESP32 programming frameworks and communication protocols.
- Analyzing "lessons learned" from Slate electric trucks for applicability (e.g., modularity, power management, software updates).
*Information Needed from Amr:**
- Specific appliances to prioritize for initial design (e.g., start with a washer or a fridge?).
- Key "lessons learned" from Slate electric trucks that you want to apply directly.
- Desired aesthetic or functional principles for the modularity.
- Any existing open-source hardware/software projects you're aware of that could serve as a starting point.
*Next Steps:**
1. Gather Amr's priorities and specific insights from Slate electric trucks.
2. Begin conceptualizing modular designs for a chosen appliance.
3. Research ESP32 and Home Assistant integration best practices.

32
projects/off_grid_field_guide/README.org Normal file
View File

@@ -0,0 +1,32 @@
#+title: Off-grid Field Guide Project
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:05]
#+begin_comment
Project documentation for Off-grid Field Guide Project
#+end_comment
* Off-grid Field Guide Project
*Goal:** To develop a modular manual for off-grid activities and areas, designed to fit into a traveler's notebook.
*Key Concepts:**
- *Modularity:** Content organized into interchangeable modules based on activity or location.
- *Off-grid Focus:** Information relevant to operating in environments without traditional infrastructure.
- *Traveler's Notebook Format:** Designed for physical print and portability, compatible with standard traveler's notebook dimensions.
- *Content Areas:** Diverse range of topics to support various off-grid activities.
*Initial Scope:**
- Defining the modular structure and content categories (e.g., navigation, first aid, water sourcing, shelter building, communication).
- Researching existing field guides or survival manuals for best practices and content ideas.
- Considering the physical layout and design constraints for a traveler's notebook format.
*Information Needed from Amr:**
- Specific off-grid activities or scenarios to prioritize for the initial modules.
- Desired level of detail and practical applicability for the content.
- Any existing field guides or resources that inspire this project's style or content.
- Preferred tools or methods for creating the modular content and layout (e.g., Org-mode for content, specific publishing tools).
*Next Steps:**
1. Gather Amr's priorities for initial content modules.
2. Research existing modular documentation practices.
3. Begin outlining content categories and potential module structures.

27
projects/open_personal_equipment_system/README.org Normal file
View File

@@ -0,0 +1,27 @@
#+title: Open Personal Equipment System Project
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:05]
#+begin_comment
Project documentation for Open Personal Equipment System Project
#+end_comment
* Open Personal Equipment System Project
*Goal:** To define and develop an open standard, along with instances, for personal equipment systems, with an initial focus on carrying, organization, and storage solutions.
*Initial Scope:**
- Defining what constitutes "personal equipment" within this context.
- Researching existing open standards or modular systems for gear organization.
- Proposing a framework for the "open standard" (e.g., modularity, interoperability, material specifications).
- Conceptualizing initial "instances" or examples of equipment based on the standard.
*Information Needed from Amr:**
- Specific types of personal equipment or scenarios you envision this system addressing.
- Key principles or philosophies for the "open standard" (e.g., durability, repairability, sustainability, cost-effectiveness).
- Any existing gear or systems that inspire this project.
- Target user base or application (e.g., everyday carry, outdoor recreation, professional use).
*Next Steps:**
1. Gather Amr's vision and core principles.
2. Research existing examples and standards.
3. Begin outlining the standard's core components.

1
projects/org-agent Submodule

Submodule projects/org-agent added at 9acc7713e0

64
projects/org-agent-memex/ARCHITECTURE.org Normal file
View File

@@ -0,0 +1,64 @@
#+TITLE: Org-Agent Memex Architecture Notes
#+AUTHOR: Amr
#+CREATED: [2026-03-17 Tue]
#+BEGIN_COMMENT
Core architectural principles and design decisions for the org-agent memex system.
#+END_COMMENT
* Core Philosophy: Single User, Single Agent
** Why This Scope?
The system is deliberately designed for *one human, one AI assistant*:
- **No coordination complexity**: One agent owns one workflow (Scribe = Atomic Notes (Zettelkasten) distillation, GTD Manager = task promotion)
- **No conflict resolution**: Agent reads from immutable sources (daily logs) and writes to separate targets (atomic notes, GTD promotions)
- **No multi-agent negotiation**: The assistant doesn't delegate to sub-agents; it executes skills directly
This is *not* a multi-agent orchestration system. It's personal automation.
* Generalization via Environment Variables
** Principle: Build with generalization, keep variable values out**
All identity-specific and configuration values live in `.env`:
| Variable | Purpose |
|----------|---------|
| MEMEX_USER | The human user's name (e.g., "Amr") |
| MEMEX_ASSISTANT | The AI assistant's identifier (e.g., "Agent") |
| CURRENT_TEXT_MANIPULATION_MODEL | The LLM tier for text processing |
| MEMEX_* paths | Folder structure (PARA hierarchy) |
Skills reference these as `$VARIABLE` in scripts or get instructed to use them. No hardcoded names in skill logic.
* Source of Simplicity
** What makes this project tractable:**
1. *Standing on established frameworks*: Org-mode, Atomic Notes (Zettelkasten) method, GTD, PARA organization—the hard thinking is already done
2. *Git as state machine*: Rather than building custom sync or consensus, we use Git commits as the source of truth for "what's new"
3. *Immutable sources*: Daily logs are append-only; the Scribe never writes to them
4. *Deterministic outputs*: Atomic notes have clear rules (concept-filenames, id: backlinks, no dates in names)
** What we're NOT building** (which would add complexity):
- Multi-user collaborative editing
- Real-time synchronization across devices
- Agent-to-agent task delegation protocols
- Distributed state management
- Conflict resolution for simultaneous edits
The complexity is in the *workflow logic*, not the technical infrastructure.
* Future: Linking with Native Org-Agent
** Phase 1** (current): OpenClaw orchestrates cloud LLMs using SKILL.md definitions
** Phase 2** (future): Native `org-agent` (Common Lisp) executes the same skills locally
The interface remains constant:
- Skill definitions in Org-mode format (SKILL.md)
- .env configuration
- PARA folder structure
- Git-based state tracking
When `org-agent` matures, it can read and execute the same skill files we're writing today. The transition from cloud-based to local inference becomes seamless because the *specification* (Org files) is implementation-agnostic.

54
projects/org-agent-memex/README.org Normal file
View File

@@ -0,0 +1,54 @@
#+TITLE: Atomic Notes (Zettelkasten) & GTD in Org-mode Project
#+AUTHOR: Amero Garcia
#+CREATED: [2026-03-16 Mon 14:00]
#+BEGIN_COMMENT
This file outlines the project to design, implement, and document a comprehensive, integrated workflow for Atomic Notes (Zettelkasten) and GTD using Org-mode, with the ultimate output being an agent skill.
#+END_COMMENT
* Atomic Notes (Zettelkasten) & GTD in Org-mode Project
*Goal:** To design, implement, and document a comprehensive, integrated workflow for Atomic Notes (Zettelkasten) (knowledge management) and Getting Things Done (GTD - task management) using Org-mode. *The ultimate output of this project will be an agent skill.**
*Key Integrations:**
- *Emacs:** Primary access and powerful Org-mode features.
- *Android Tools:** Ensure seamless access and functionality via Markor and Orgzly (revived).
*Strategic Importance:** This system will become the primary coordination method for our work, outside of direct chat communication. It will centralize task tracking, knowledge capture, and project management.
*Workflow Details & Current Setup (as provided by Amr):**
- *Org-mode File Front Matter:** For each Org-mode file, there must be a basic front matter. At minimum, this must include a `#+TITLE:`, an `#+AUTHOR:`, and a `#+CREATED:` date. Short descriptive comments within a `#+BEGIN_COMMENT` / `#+END_COMMENT` block are also highly recommended.
- *Inbox File:** `memex/inbox.org`. All new captured items will go here. No other files in the inbox collection are to be used for general inbox capture.
- *GTD.org Structure:** Contains four top-level `*` headings:
- `* Actions`: For standalone actionable items.
- `* Projects`: Contains `*` headers for each project, with `***` headers for actionable items within those projects.
- `* Incubate`: For placeholders for future projects.
- `* Habits`: Tracks recurring personal habits, potentially to be used as a heartbeat for the new AI agent.
- *:CREATED: Property:** All items in `memex/inbox.org` and `GTD.org` must include a `:CREATED:` property in their `:PROPERTIES:` drawer. The date format is `[YYYY-MM-DD Day HH:MM]`.
- *:LOGBOOK: Drawer:** All task items must include a `:LOGBOOK:` drawer AFTER the `:PROPERTIES:` drawer (not nested inside). State changes are logged as `- State "NEW" from "OLD" [timestamp]`. This tracks the full history of state transitions for each task.
- *Org-Todo States:** Items will use the following `org-todo` keywords to indicate status: `NEXT`, `TODO`, `WAIT`, `DONE`, `CNCL`. It is understood that these states are used to make tasks appear in Amr's Emacs and Orgzly agendas, serving as a direct mechanism for communicating required actions.
- *Authorship & Assignment:** Confirmed use of `:AUTHOR:` (for original creator) and `:ASSIGNED:` (for current responsible individual). It is noted that filtering by `:ASSIGNED:` is possible in Emacs, with potential uncertainties for Orgzly.
- *User Interaction Requirements (Emacs, Orgzly, Markor):**
- Ability to follow status of actionable items in Org-zly and Emacs agendas.
- Ability to read and write Org-mode files in Emacs and Markor.
- Ability to find out and manipulate `TODO` items in Org-zly and Emacs agendas.
- *Agent-User Coordination Mechanism:** The agent will place items requiring Amr's attention as `TODO` (general planned items) and `NEXT` (immediate, high-priority actions) in his agenda.
- *Automatic `NEXT` Promotion:** A critical feature to integrate is the automatic promotion of a `TODO` item to `NEXT` in `org-gtd` once the preceding `NEXT` item (within a sequential *Project*) is marked `DONE`. This behavior specifically applies to interdependent or sequential items that constitute a `Project`. Standalone `NEXT` items (e.g., under `* Actions`) are `NEXT` by default and do not trigger subsequent promotions. This behavior must be accounted for in the agent skill.
*Initial Scope:**
- Ensure all items I create or modify adhere strictly to the `:CREATED:` property format and `org-todo` states.
- Implement the proposed `:AUTHOR:` and `:ASSIGNED:` properties for collaborative items.
- Defining specific Org-mode structures for Atomic Notes (Zettelkasten) notes (unique IDs, linking, tags), building upon existing GTD structure.
- Establishing workflows for daily capture, processing, and review aligned with Amr's system.
- Exploring and configuring Markor and Orgzly for optimal mobile interaction with Org files.
- Documenting the entire workflow for clarity and ease of use.
*Information Needed from Amr:**
- Confirmation or modification of the proposed `:AUTHOR:` and `:ASSIGNEE:` properties (e.g., preferred format for names, single vs. multiple assignees).
- Specific requirements or desired features for mobile access/editing with tools like Markor and Orgzly.
- Your vision for how this system will function as our *"main coordination method"** in practice.
- Any existing Org-mode Atomic Notes (Zettelkasten) practices you currently use or prefer.
*Next Steps:**
1. Gather Amr's current practices and specific requirements.
2. Begin outlining core Org-mode structures for both Atomic Notes (Zettelkasten) and GTD.
3. Research best practices for mobile Org-mode synchronization and editing with Markor/Orgzly.

19
projects/org-agent-memex/org-agent-memex-gtd/README.md Normal file
View File

@@ -0,0 +1,19 @@
# Org-Agent Memex GTD (org-agent-memex-gtd)
This is the task management counterpart to the Atomic Notes (Atomic Notes (Zettelkasten)) skill. It automates the GTD (Getting Things Done) workflows within your Org-mode environment.
## Features
1. **Sequential Project Auto-Promotion:** When you complete a `NEXT` action inside a sequential project in `gtd.org` (marking it `DONE`), this skill automatically finds the subsequent `TODO` item and promotes it to `NEXT`. This ensures your Org Agenda is always populated with the very next actionable steps without manual intervention.
2. **Inbox Processing Assistance:** Provides an automated routine to read through `inbox.org`, categorize items, and propose where they should be filed in `gtd.org` (e.g., under `* Actions` or specific `* Projects`).
3. **Collaboration Setup:** Standardizes the use of `:AUTHOR:` and `:ASSIGNED:` properties so you and the AI agent can delegate tasks to each other seamlessly.
## Configuration
Relies on the same `.env` file used by the Atomic Notes (Atomic Notes (Zettelkasten)) module, specifically:
- `MEMEX_DIR` - Base memex directory
- `MEMEX_INBOX` - Inbox file (e.g., `memex/inbox.org`)
- `MEMEX_SYSTEM` - System directory for skills
## Setup
Like the Scribe agent, this skill can be run ad-hoc by asking your AI assistant to "Run the GTD manager" or scheduled as a background cron job to periodically audit and update task statuses.

61
projects/org-agent-memex/org-agent-memex-gtd/SKILL.md Normal file
View File

@@ -0,0 +1,61 @@
---
name: org-agent-memex-gtd
description: "Automate Getting Things Done (GTD) workflows in Emacs Org-mode. Auto-promotes TODO to NEXT in sequential projects and processes the inbox. Use when: user asks to manage tasks, update GTD, promote NEXT actions, or process the inbox. NOT for: extracting Atomic Notes (Atomic Notes (Zettelkasten)) knowledge or editing daily logs."
homepage: ""
metadata: { "openclaw": { "emoji": "✅", "requires": { "bins": ["grep", "sed"] }, "user-invocable": true } }
---
# Org-Agent Memex GTD
Automated GTD manager designed to keep your task lists fluid and your Org Agenda accurate. It handles the structural logic of sequential projects and helps clarify your inbox.
## When to Use
✅ **USE this skill when:**
- The user asks to "update GTD", "promote next actions", or "manage tasks".
- The user completes a task in a project and wants the next one queued up.
- The user asks to "process the inbox" or "clarify inbox tasks".
❌ **DON'T use this skill when:**
- Working with Atomic Notes (Atomic Notes (Zettelkasten)), evergreen notes, or daily logs (use `org-agent-memex-zettlekasten`).
- Just capturing a quick thought (user should do this via Emacs).
## Instructions
### Action 1: Auto-Promote Sequential Tasks (`gtd.org`)
When asked to update projects or promote NEXT actions:
1. Read the `gtd.org` file (located in `$MEMEX_DIR/gtd.org`).
2. Identify sequential projects (under `* Projects`).
3. Look for the most recently completed tasks (marked `DONE`).
4. If a task was marked `DONE`, find the immediate next sibling heading that is marked `TODO` within the same parent project.
5. Change that `TODO` to `NEXT`.
6. Ensure that standalone actions (under `* Actions`) are left alone (they are typically parallel, not sequential).
7. Save the file and report which tasks were promoted to `NEXT`.
### Action 2: Inbox Processing (`inbox.org`)
When asked to process the inbox:
1. Read `$MEMEX_INBOX`.
2. For each raw entry, determine if it is actionable.
3. If actionable, propose a structured Org-mode task format with:
- `TODO` or `NEXT` state
- `:PROPERTIES:` drawer with `:CREATED:` and optional `:ASSIGNED:`
- `:LOGBOOK:` drawer (AFTER :PROPERTIES:, not inside) tracking state changes
Format:
```org
*** TODO Task Name
:PROPERTIES:
:CREATED: [YYYY-MM-DD Day HH:MM]
:ASSIGNED: $MEMEX_USER
:END:
:LOGBOOK:
- State "TODO" from "" [YYYY-MM-DD Day HH:MM]
:END:
```
4. Propose which section of `gtd.org` it belongs to (e.g., a specific project or standalone `* Actions`).
5. Ask the user for confirmation before moving the items out of `inbox.org` into `gtd.org`.
## Notes
- **Timestamps:** Ensure every new task generated or moved retains or receives a `:CREATED:` property formatted as `[YYYY-MM-DD Day HH:MM]`.
- **Assignment:** The agent can assign tasks to itself by setting `:ASSIGNED: $MEMEX_ASSISTANT` or to the user via `:ASSIGNED: $MEMEX_USER`. Configure these values in your `.env` file.
- **State Tracking:** The `:LOGBOOK:` drawer must appear AFTER the `:PROPERTIES:` drawer (not nested inside). State changes are logged as `- State "NEW" from "OLD" [timestamp]`. When a task changes state (e.g., TODO → NEXT, or TODO → DONE), append a new line to the LOGBOOK drawer.

19
projects/org-agent-memex/org-agent-memex-gtd/org-agent-memex-gtd/README.md Normal file
View File

@@ -0,0 +1,19 @@
# Org-Agent Memex GTD (org-agent-memex-gtd)
This is the task management counterpart to the Atomic Notes (Atomic Notes (Zettelkasten)) skill. It automates the GTD (Getting Things Done) workflows within your Org-mode environment.
## Features
1. **Sequential Project Auto-Promotion:** When you complete a `NEXT` action inside a sequential project in `gtd.org` (marking it `DONE`), this skill automatically finds the subsequent `TODO` item and promotes it to `NEXT`. This ensures your Org Agenda is always populated with the very next actionable steps without manual intervention.
2. **Inbox Processing Assistance:** Provides an automated routine to read through `inbox.org`, categorize items, and propose where they should be filed in `gtd.org` (e.g., under `* Actions` or specific `* Projects`).
3. **Collaboration Setup:** Standardizes the use of `:AUTHOR:` and `:ASSIGNED:` properties so you and the AI agent can delegate tasks to each other seamlessly.
## Configuration
Relies on the same `.env` file used by the Atomic Notes (Atomic Notes (Zettelkasten)) module, specifically:
- `MEMEX_DIR` - Base memex directory
- `MEMEX_INBOX` - Inbox file (e.g., `memex/inbox.org`)
- `MEMEX_SYSTEM` - System directory for skills
## Setup
Like the Scribe agent, this skill can be run ad-hoc by asking your AI assistant to "Run the GTD manager" or scheduled as a background cron job to periodically audit and update task statuses.

61
projects/org-agent-memex/org-agent-memex-gtd/org-agent-memex-gtd/SKILL.md Normal file
View File

@@ -0,0 +1,61 @@
---
name: org-agent-memex-gtd
description: "Automate Getting Things Done (GTD) workflows in Emacs Org-mode. Auto-promotes TODO to NEXT in sequential projects and processes the inbox. Use when: user asks to manage tasks, update GTD, promote NEXT actions, or process the inbox. NOT for: extracting Atomic Notes (Atomic Notes (Zettelkasten)) knowledge or editing daily logs."
homepage: ""
metadata: { "openclaw": { "emoji": "✅", "requires": { "bins": ["grep", "sed"] }, "user-invocable": true } }
---
# Org-Agent Memex GTD
Automated GTD manager designed to keep your task lists fluid and your Org Agenda accurate. It handles the structural logic of sequential projects and helps clarify your inbox.
## When to Use
✅ **USE this skill when:**
- The user asks to "update GTD", "promote next actions", or "manage tasks".
- The user completes a task in a project and wants the next one queued up.
- The user asks to "process the inbox" or "clarify inbox tasks".
❌ **DON'T use this skill when:**
- Working with Atomic Notes (Atomic Notes (Zettelkasten)), evergreen notes, or daily logs (use `org-agent-memex-zettlekasten`).
- Just capturing a quick thought (user should do this via Emacs).
## Instructions
### Action 1: Auto-Promote Sequential Tasks (`gtd.org`)
When asked to update projects or promote NEXT actions:
1. Read the `gtd.org` file (located in `$MEMEX_DIR/gtd.org`).
2. Identify sequential projects (under `* Projects`).
3. Look for the most recently completed tasks (marked `DONE`).
4. If a task was marked `DONE`, find the immediate next sibling heading that is marked `TODO` within the same parent project.
5. Change that `TODO` to `NEXT`.
6. Ensure that standalone actions (under `* Actions`) are left alone (they are typically parallel, not sequential).
7. Save the file and report which tasks were promoted to `NEXT`.
### Action 2: Inbox Processing (`inbox.org`)
When asked to process the inbox:
1. Read `$MEMEX_INBOX`.
2. For each raw entry, determine if it is actionable.
3. If actionable, propose a structured Org-mode task format with:
- `TODO` or `NEXT` state
- `:PROPERTIES:` drawer with `:CREATED:` and optional `:ASSIGNED:`
- `:LOGBOOK:` drawer (AFTER :PROPERTIES:, not inside) tracking state changes
Format:
```org
*** TODO Task Name
:PROPERTIES:
:CREATED: [YYYY-MM-DD Day HH:MM]
:ASSIGNED: $MEMEX_USER
:END:
:LOGBOOK:
- State "TODO" from "" [YYYY-MM-DD Day HH:MM]
:END:
```
4. Propose which section of `gtd.org` it belongs to (e.g., a specific project or standalone `* Actions`).
5. Ask the user for confirmation before moving the items out of `inbox.org` into `gtd.org`.
## Notes
- **Timestamps:** Ensure every new task generated or moved retains or receives a `:CREATED:` property formatted as `[YYYY-MM-DD Day HH:MM]`.
- **Assignment:** The agent can assign tasks to itself by setting `:ASSIGNED: $MEMEX_ASSISTANT` or to the user via `:ASSIGNED: $MEMEX_USER`. Configure these values in your `.env` file.
- **State Tracking:** The `:LOGBOOK:` drawer must appear AFTER the `:PROPERTIES:` drawer (not nested inside). State changes are logged as `- State "NEW" from "OLD" [timestamp]`. When a task changes state (e.g., TODO → NEXT, or TODO → DONE), append a new line to the LOGBOOK drawer.

43
projects/org-agent-memex/org-agent-memex-workbreakdown/README.md Normal file
View File

@@ -0,0 +1,43 @@
# GTD Work Breakdown (org-agent-memex-workbreakdown)
Meta-cognitive skill to prevent AI assistants and users from stalling on complex tasks. Forces atomic decomposition before execution.
## The Problem
Complex tasks cause:
- Context saturation (procrastination)
- Scope creep (adding "just one more thing")
- The "heartbeat loop" (repeating tasks without progress)
- Overwhelm and hesitation
## The Solution
**Decompose first, execute second.**
1. Analyze task complexity (>3 steps? >2 files?)
2. Break into atomic TODOs in GTD.org
3. Execute only the FIRST item
4. Yield back to user
## Configuration
Uses same `.env` structure as other org-agent-memex skills:
- `MEMEX_DIR`
- `MEMEX_USER`
- `MEMEX_ASSISTANT`
- `CURRENT_TEXT_MANIPULATION_MODEL`
## Usage
When a task feels complex:
1. Ask AI: "Break this down with Work Breakdown skill"
2. AI creates TODOs in GTD.org
3. Execute only first item
4. Report: "[X] Completed step 1. 4 tasks remaining. Continue?"
## Anti-Patterns This Prevents
- "I'll just do it all at once"
- Editing 5+ files before committing
- Writing conditional logic on the fly
- "Let me think about it..." (stalling)

56
projects/org-agent-memex/org-agent-memex-workbreakdown/SKILL.md Normal file
View File

@@ -0,0 +1,56 @@
---
name: org-agent-memex-workbreakdown
description: "Break down complex tasks into atomic TODOs before execution. Use when: a task feels complex, involves multiple files, or may cause context saturation. Prevents procrastination by forcing decomposition. NOT for: simple single-step tasks."
homepage: ""
metadata: { "openclaw": { "emoji": "🔨", "requires": { "bins": [] }, "user-invocable": true } }
---
# GTD Work Breakdown Skill
Meta-cognitive protocol to prevent stalling and context saturation.
## When to Use
✅ **USE this skill when:**
- A task feels "complex" or overwhelming
- It involves editing more than 3 files
- It requires holding multiple concepts in working memory
- You feel the urge to apologize or hesitate (procrastination signal)
- The task description is longer than 2 sentences
❌ **DON'T use this skill when:**
- Simple single-file edit
- Direct question/answer
- Already-broken-down TODO from GTD
## Instructions
### The Decomposition Protocol
When invoked, BEFORE executing any other action:
1. **Analyze Complexity**: Ask "How many discrete steps does this actually require?"
2. **Breakdown Threshold**: If >3 steps or >2 files affected, MUST decompose
3. **Create TODOs**: Write each atomic step as a separate `TODO` in `GTD.org` under appropriate project
4. **Assign Ownership**: Each TODO gets `:ASSIGNED: $MEMEX_USER` or `:ASSIGNED: $MEMEX_ASSISTANT`
5. **Set FIRST**: Mark only the first TODO as `NEXT`, rest remain `TODO`
6. **Execute First**: Complete ONLY the `NEXT` item
7. **Yield**: After completion, report to user: "[X] Completed [first task]. [N] tasks remaining in GTD. Continue?"
### Anti-Pattern Detection
If you find yourself:
- Thinking "I'll just do it all at once"
- Planning to edit >5 files before committing
- Writing conditional logic on the fly
STOP. Invoke this skill immediately.
## Complexity Checklist
Before executing any task, ask:
- [ ] Can I complete this in under 5 minutes?
- [ ] Does it touch only 1 file?
- [ ] Is the outcome predictable?
If ANY answer is "No", decompose first.

19
projects/org-agent-memex/org-agent-memex-zettlekasten/.env.example Normal file
View File

@@ -0,0 +1,19 @@
MEMEX_DIR="memex"
MEMEX_INBOX="memex/inbox.org"
MEMEX_DAILY="memex/1_daily"
MEMEX_NOTES="memex/2_notes"
MEMEX_DRAFTS="memex/3_drafts"
MEMEX_PUBLISHED="memex/4_published"
MEMEX_PROJECTS="memex/5_projects"
MEMEX_AREAS="memex/6_areas"
MEMEX_RESOURCES="memex/7_resources"
MEMEX_ARCHIVES="memex/8_archives"
MEMEX_SYSTEM="memex/9_system"
MEMEX_ATTACHMENTS="memex/attachments"
# Model Configuration
CURRENT_TEXT_MANIPULATION_MODEL="google-gemini-cli/gemini-3.1-flash"
# Identity Configuration
MEMEX_USER="Amr"
MEMEX_ASSISTANT="Agent"

33
projects/org-agent-memex/org-agent-memex-zettlekasten/README.md Normal file
View File

@@ -0,0 +1,33 @@
# Atomic Notes (Atomic Notes (Zettelkasten)) & GTD Automation (org-agent-memex-zettlekasten)
This system uses a hybrid approach to Personal Knowledge Management (PKM). It leverages Emacs Org-mode for low-friction, structured capture into daily logs, and an OpenClaw AI Sub-Agent ("The Scribe") to nightly distill these raw thoughts into an evergreen, atomic Atomic Notes (Atomic Notes (Zettelkasten)).
## 1. Environment Configuration (`.env`)
To ensure Emacs, OpenClaw, and the Scribe Agent all agree on where files live, we use a single `.env` file at the root of the workspace.
**Action:**
Copy `.env.example` to `.env` and adjust the paths to match your preferred directory structure.
## 2. Emacs Org-Capture Setup
All captures route to the current day's log (e.g. `$MEMEX_DAILY/YYYY-MM-DD.org`), preserving the raw chronological context.
**Action:**
Add the Emacs Lisp snippet from `init-atomic-notes.el` to your `init.el` or `config.el` to set up your capture templates dynamically using the `.env` variables.
## 3. The Distillation State Tracker
The Scribe Agent uses a JSON file to remember the last Git commit it processed, preventing it from distilling the same notes twice or modifying the daily logs directly.
**Action:**
Run `./install.sh` to initialize the directory structure and create the state file (`$MEMEX_SYSTEM/distillation-state.json`) automatically.
## 4. OpenClaw Cron Job (The Scribe Agent)
The final piece is the scheduled automation. We create a cron job in OpenClaw that runs every night, reads the diffs, and creates atomic notes.
**Action:**
1. Move `openclaw-scribe-skill.org` into your `$MEMEX_SYSTEM/skills/` folder.
2. Ask your OpenClaw orchestrator/assistant to schedule the Scribe Agent using the `cron` tool, referencing the prompt defined in `$MEMEX_SYSTEM/skills/Scribe-Agent.org` or your renamed skill file.
3. Configure the cron job to use the model specified in `CURRENT_TEXT_MANIPULATION_MODEL` within your `.env` file (e.g., `google-gemini-cli/gemini-3.1-flash`). You can update this `.env` variable periodically to stay on the most cost-effective text manipulation model.
### Architecture Rules:
- **Dailies are Immutable:** The Scribe reads `$MEMEX_DAILY/` but NEVER writes to it.
- **Evergreen Notes:** The Scribe extracts concepts, generates descriptive snake_case filenames (no dates), and writes them to `$MEMEX_NOTES/` with a `Source:` backlink using an Org-ID reference (`id:`) to the original daily file.

54
projects/org-agent-memex/org-agent-memex-zettlekasten/SKILL.md Normal file
View File

@@ -0,0 +1,54 @@
---
name: org-agent-memex-zettlekasten
description: "Automate the nightly distillation of Emacs Org-mode daily logs into atomic Atomic Notes (Atomic Notes (Zettelkasten)) notes. Use when: user wants to run the Scribe distillation pipeline, process daily captures, or extract Atomic Notes (Atomic Notes (Zettelkasten)) notes. NOT for: generic org-mode editing or GTD task management."
homepage: ""
metadata: { "openclaw": { "emoji": "🧠", "requires": { "bins": ["git"] }, "user-invocable": true } }
---
# Org-Agent Memex Atomic Notes (Atomic Notes (Zettelkasten)) (The Scribe)
Automated distillation skill designed to process raw daily captures into permanent atomic notes for your Atomic Notes (Atomic Notes (Zettelkasten)). It reads the raw chronological logs, identifies newly captured concepts, and extracts them into self-contained evergreen notes with proper Org-Roam `id:` backlinks.
## When to Use
✅ **USE this skill when:**
- Running the nightly Atomic Notes (Atomic Notes (Zettelkasten)) distillation pipeline.
- User asks to "distill my daily notes", "run the scribe", or "process captures".
- Automating atomic note extraction via cron jobs.
❌ **DON'T use this skill when:**
- Editing standard GTD task lists.
- Capturing new notes (that's the user's job via Emacs `org-capture`).
- Modifying the daily logs (dailies are immutable).
## Instructions
When triggered to distill the notes, execute the following strict pipeline:
1. **Read State:** Read the distillation state file (defined by `$MEMEX_SYSTEM/distillation-state.json`) to get the `lastProcessedCommit` hash.
2. **Find New Captures:** Run a Git diff on the daily directory since that commit:
```bash
git diff <last_commit_hash> HEAD -- $MEMEX_DAILY
```
3. **Process Each Capture:**
For every new Atomic Notes (Atomic Notes (Zettelkasten)) capture found in the diff:
- Read the raw capture text.
- Determine the core concept being discussed.
- Generate a concise, `snake_case` filename (e.g., `core_concept_name.org`). Do NOT use dates in this filename.
- Write the content to the notes directory (`$MEMEX_NOTES/<filename>`).
- Ensure the new note is formatted as an atomic Org-mode note with an `#+ID` and a `Source:` backlink using an `id:` reference pointing back to the original daily file.
4. **Update State:** Update the distillation state JSON file with the current HEAD commit hash.
5. **Report:** Output a summary of the concepts extracted and the files created.
## Configuration
This skill expects the environment to be configured via a `.env` file containing at least:
- `MEMEX_DAILY` - Directory containing daily logs (e.g., `memex/1_daily`)
- `MEMEX_NOTES` - Directory for evergreen atomic notes (e.g., `memex/2_notes`)
- `MEMEX_SYSTEM` - Directory for system files and state (e.g., `memex/9_system`)
- `CURRENT_TEXT_MANIPULATION_MODEL` - The LLM to use for cron execution (e.g., `google-gemini-cli/gemini-3.1-flash`)
## Notes
- **Immutability:** The daily logs are raw, immutable records. Never modify them destructively during processing.
- **Evergreen:** Atomic notes should focus on concepts, not chronology.

10
projects/org-agent-memex/org-agent-memex-zettlekasten/init-zettelkasten.el Normal file
View File

@@ -0,0 +1,10 @@
(setq org-capture-templates
'(("z" "Atomic Notes (Zettelkasten) (Captures to Daily)")
("zf" "Fleeting Note" entry (file+olp+datetree (expand-file-name (format "%s/%%<%%Y-%%m-%%d>.org" (getenv "MEMEX_DAILY"))))
"* Fleeting Note: %?\n :PROPERTIES:\n :ID: %u\n :CREATED: %U\n :END:\n\n %i")
("zl" "Draft Literature Note" entry (file+olp+datetree (expand-file-name (format "%s/%%<%%Y-%%m-%%d>.org" (getenv "MEMEX_DAILY"))))
"* Literature Note: %?\n :PROPERTIES:\n :ID: %u\n :CREATED: %U\n :AUTHOR: \n :SOURCE: \n :END:\n\n *Summary:*\n %?\n\n *Key Insights:*\n - ")
("zp" "Draft Permanent Note" entry (file+olp+datetree (expand-file-name (format "%s/%%<%%Y-%%m-%%d>.org" (getenv "MEMEX_DAILY"))))
"* Permanent Note: %?\n :PROPERTIES:\n :ID: %u\n :CREATED: %U\n :LINKS: \n :END:\n\n *Concept:*\n %?\n\n *References:*\n - ")
("t" "GTD - Task / Inbox" entry (file (getenv "MEMEX_INBOX"))
"* TODO %?\n :PROPERTIES:\n: :CREATED: %U\n :END:\n :LOGBOOK:\n - State \"TODO\" from \"\" [%U]\n :END:\n\n %i\n %a")))

110
projects/org-agent-memex/org-agent-memex-zettlekasten/install.sh Executable file
View File

@@ -0,0 +1,110 @@
#!/usr/bin/env bash
set -e
# Load .env if it exists, otherwise use defaults
if [ -f ".env" ]; then
source .env
else
echo "Creating .env from .env.example..."
cp .env.example .env
source .env
fi
echo "Creating directory structure..."
# Ensure MEMEX_DIR is available, fallback if not set
MEMEX_DIR="${MEMEX_DIR:-memex}"
mkdir -p "$MEMEX_DIR/0_inbox" "$MEMEX_DAILY" "$MEMEX_NOTES" "$MEMEX_DRAFTS" "$MEMEX_PUBLISHED" "$MEMEX_PROJECTS" "$MEMEX_AREAS" "$MEMEX_RESOURCES" "$MEMEX_ARCHIVES" "$MEMEX_SYSTEM/skills" "$MEMEX_ATTACHMENTS"
echo "Generating directory README.org files..."
DATE=$(date +"[%Y-%m-%d %a]")
create_readme() {
local dir=$1
local title=$2
local desc=$3
cat <<EOF > "$dir/README.org"
#+TITLE: $title
#+AUTHOR: User
#+CREATED: $DATE
#+BEGIN_COMMENT
$desc
#+END_COMMENT
* $title
$desc
EOF
}
create_readme "$MEMEX_DIR/0_inbox" "0_inbox: The Capture Point" "Temporary holding area for raw captures, links, and quick thoughts before they are processed into actionable items (GTD) or knowledge (Atomic Notes (Zettelkasten))."
create_readme "$MEMEX_DAILY" "1_daily: The Immutable Log" "Chronological daily logs (YYYY-MM-DD.org) serving as the primary capture location for fleeting notes and daily events. These are immutable records."
create_readme "$MEMEX_NOTES" "2_notes: The Atomic Notes (Zettelkasten)" "Evergreen, atomic notes. Each file represents a single concept, is heavily interlinked, and uses snake_case filenames without dates."
create_readme "$MEMEX_DRAFTS" "3_drafts: Works in Progress" "Long-form writing, essays, or articles actively being synthesized from the atomic notes."
create_readme "$MEMEX_PUBLISHED" "4_published: Final Outputs" "Completed, finalized works and static snapshots of published material."
create_readme "$MEMEX_PROJECTS" "5_projects: Active Projects" "Active, time-bound efforts with a clear definition of done. Each project has its own dedicated folder for specifications and artifacts."
create_readme "$MEMEX_AREAS" "6_areas: Spheres of Responsibility" "Ongoing areas of life and work with a standard to be maintained over time (e.g., Health, Finances, Operations)."
create_readme "$MEMEX_RESOURCES" "7_resources: Reference Material" "Topics of ongoing interest, external reference material, raw literature notes, and useful information."
create_readme "$MEMEX_ARCHIVES" "8_archives: Cold Storage" "Inactive items from other categories, including completed projects, abandoned areas, or deprecated resources."
create_readme "$MEMEX_SYSTEM" "9_system: Memex Administration" "System configuration, AI agent skills, org-mode templates, cron states, and tracking scripts."
echo "Generating root Master Memex README.org..."
cat <<EOF > "$MEMEX_DIR/README.org"
#+TITLE: The Master Memex
#+AUTHOR: User
#+CREATED: $DATE
#+BEGIN_COMMENT
The central hub and map of content for this personal intelligence organization.
#+END_COMMENT
* 🧠 The Master Memex
This is the central hub for our knowledge management system, synthesizing three core methodologies:
- *Atomic Notes (Zettelkasten):* For evergreen, interlinked, atomic knowledge.
- *GTD (Getting Things Done):* For actionable task tracking and project execution.
- *PARA:* For high-level directory organization (Projects, Areas, Resources, Archives).
* The Architecture
Our workspace is strictly divided into these functional zones:
- [[file:0_inbox/README.org][0_inbox]]: The zero-friction capture point for raw thoughts and tasks.
- [[file:1_daily/README.org][1_daily]]: Immutable chronological logs and fleeting notes (YYYY-MM-DD.org).
- [[file:2_notes/README.org][2_notes]]: The Atomic Notes (Zettelkasten). Atomic, concept-based, interlinked notes.
- [[file:3_drafts/README.org][3_drafts]]: Works in progress, essays, and active synthesis.
- [[file:4_published/README.org][4_published]]: Final outputs and static snapshots of completed work.
- [[file:5_projects/README.org][5_projects]]: Active, time-bound efforts with a clear definition of done.
- [[file:6_areas/README.org][6_areas]]: Ongoing spheres of responsibility (e.g., Health, Finances).
- [[file:7_resources/README.org][7_resources]]: External reference material and raw literature notes.
- [[file:8_archives/README.org][8_archives]]: Cold storage for completed projects and inactive items.
- [[file:9_system/README.org][9_system]]: System configuration, AI skills, and automation scripts.
* Core Workflows
** 1. Capture (Anytime)
Everything enters the system via \`0_inbox\` or as a Fleeting Note in \`1_daily\`. Zero friction, no filtering.
** 2. Nightly Distillation (The Scribe)
An automated AI sub-agent reads the daily captures and extracts conceptual thoughts into evergreen, atomic notes in \`2_notes\`, leaving the original daily logs untouched.
** 3. Weekly Maintenance
Review active projects, clarify inbox items into actionable GTD tasks, and explore the Atomic Notes (Zettelkasten) graph to merge concepts and forge new connections.
EOF
# Touch inbox
touch "$MEMEX_INBOX"
# Initialize distillation state if not present
STATE_FILE="$MEMEX_SYSTEM/distillation-state.json"
if [ ! -f "$STATE_FILE" ]; then
echo "Initializing $STATE_FILE..."
# Get current git commit or use a placeholder
HASH=$(git rev-parse HEAD 2>/dev/null || echo "INITIAL_HASH")
echo "{
\"lastProcessedCommit\": \"$HASH\"
}" > "$STATE_FILE"
fi
echo "Installation complete."
echo "1. Add the contents of init-atomic-notes.el to your Emacs config."
echo "2. Add openclaw-scribe-skill.org to your \$MEMEX_SYSTEM/skills/ directory."
echo "3. Ask your OpenClaw agent to schedule the Scribe job."

29
projects/org-agent-memex/org-agent-memex-zettlekasten/openclaw-scribe-skill.org Normal file
View File

@@ -0,0 +1,29 @@
#+TITLE: SKILL: Scribe Agent (Distillation Sub-Agent)
#+ID: skill-scribe-agent
#+STARTUP: content
* Overview
The Scribe Agent is an automated distillation sub-agent designed to process raw daily captures into permanent atomic notes for the Atomic Notes (Zettelkasten). It runs as an isolated OpenClaw cron job.
* Configuration
- **Type:** OpenClaw Cron Job
- **Target:** `isolated`
- **Model:** `CURRENT_TEXT_MANIPULATION_MODEL` (Updates periodically based on review; currently an efficient LLM suitable for text parsing).
- **Environment:** Loads variables from `.env` to locate folders (e.g., `$MEMEX_DAILY`, `$MEMEX_NOTES`, `$MEMEX_SYSTEM`).
* System Prompt / Agent Turn Directive
```markdown
You are the Scribe, an automated distillation sub-agent.
Your sole job is to process raw notes into a Atomic Notes (Zettelkasten).
Do not engage in conversation. Only execute the following pipeline:
1. Read `$MEMEX_SYSTEM/distillation-state.json` to get the last processed Git commit hash.
2. Run `git diff <last_commit_hash> HEAD -- $MEMEX_DAILY/` to find new captures.
3. For every new Atomic Notes (Zettelkasten) capture found in the diff:
a. Read the raw capture.
b. Determine the core concept.
c. Generate a concise, snake_case filename (e.g., `core_concept_name.org`).
d. Write the content to `$MEMEX_NOTES/<filename>`, ensuring it is formatted as an atomic Org-mode note with `#+ID` and a `Source:` backlink using an `id:` reference to the original daily file.
4. Update `$MEMEX_SYSTEM/distillation-state.json` with the current HEAD commit hash.
5. Exit.
```

28
projects/org-gtd-archive-roam-daily/README.org Normal file
View File

@@ -0,0 +1,28 @@
#+title: Org-GTD Archive Roam Daily Project
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:05]
#+begin_comment
Project documentation for Org-GTD Archive Roam Daily Project
#+end_comment
* Org-GTD Archive Roam Daily Project
*Goal:** To develop a feature for `org-gtd` that enables archiving of Org-mode headings to `org-roam-dailies` based on the `:CREATED:` property of the heading being archived. This will ensure chronological and contextually relevant archiving.
*Initial Scope:**
- Understanding the current `org-gtd` archiving mechanisms.
- Investigating `org-roam-dailies` structure and API for programmatic interaction.
- Designing a function that extracts the `:CREATED:` property from an Org heading.
- Implementing logic to move/copy the heading content to the correct daily note based on its creation date.
- Considering how to handle headings without a `:CREATED:` property.
*Information Needed from Amr:**
- Specific `org-gtd` setup details (e.g., how archiving is currently triggered/configured).
- Desired behavior if a `:CREATED:` property is missing.
- Preferred method for integration (e.g., new command, modification of existing archiving function).
- Any existing thoughts on error handling or edge cases.
*Next Steps:**
1. Gather Amr's specific usage patterns and requirements.
2. Dive into `org-gtd` and `org-roam` documentation/source code.
3. Propose a design for the new archiving function.

74
projects/org-gtd-archive-roam-daily/org-gtd-archive-roam-daily.el Normal file
View File

@@ -0,0 +1,74 @@
;;; org-gtd-archive-roam-daily.el --- Archive Org headings to Org-roam dailies
;;; Commentary:
;; This file provides an Elisp function to archive an Org-mode heading
;; at point to an Org-roam daily file based on its :CREATED: property.
;;; Code:
(require 'org-roam-dailies)
(require 'org-element)
(require 'org-time)
(defun amero-get-org-heading-created-property ()
"Extract the :CREATED: property from the current Org heading.
Returns a time string or nil if not found."
(interactive)
(save-excursion
(org-back-to-heading t)
(org-entry-get (point) "CREATED")))
(defun amero-parse-created-timestamp (timestamp-string)
"Parse an Org-mode timestamp string like '[2026-03-16 Mon 14:05]'
into an Emacs internal time object.
Returns nil if parsing fails."
(ignore-errors
(org-time-string-to-time timestamp-string)))
(defun amero-get-daily-note-file (time-object)
"Get the Org-roam daily note file for a given Emacs TIME-OBJECT.
Creates the file if it doesn't exist. Returns the file path."
(let* ((date-string (format-time-string org-roam-dailies-capture-templates-date-format time-object))
(file-path (expand-file-name (concat date-string ".org")
(expand-file-name org-roam-dailies-directory org-roam-directory))))
;; Ensure the directory exists
(unless (file-exists-p (file-name-directory file-path))
(make-directory (file-name-directory file-path) t))
;; Create file if it doesn't exist (org-roam-dailies-goto-date handles this,
;; but we need to ensure it's created and accessible for append)
(unless (file-exists-p file-path)
(with-temp-buffer
(insert (format "#+title: %s\n" date-string))
(write-file file-path)))
file-path))
(defun org-gtd-archive-roam-daily ()
"Archive the current Org heading to an Org-roam daily file
based on its :CREATED: property.
Signals an error if :CREATED: property is missing."
(interactive)
(unless (org-before-first-heading-p (point))
(user-error "Point is not on an Org heading or within an Org file."))
(let* ((created-timestamp-string (amero-get-org-heading-created-property))
(created-time-object (and created-timestamp-string
(amero-parse-created-timestamp created-timestamp-string)))
(heading-start (save-excursion (org-back-to-heading t) (point)))
(heading-end (save-excursion (org-end-of-subtree t) (point)))
(heading-content (buffer-substring-no-properties heading-start heading-end))
daily-file-path)
(unless created-time-object
(user-error "No date error: Heading is missing a valid :CREATED: property."))
(setq daily-file-path (amero-get-daily-note-file created-time-object))
(with-current-buffer (find-file-noselect daily-file-path)
(goto-char (point-max))
(insert "\n\n" heading-content)
(save-buffer))
;; Remove the original heading
(delete-region heading-start heading-end)
(message "Archived heading to %s" daily-file-path)))
(provide 'org-gtd-archive-roam-daily)
;;; org-gtd-archive-roam-daily.el ends here

74
projects/org-json-bridge/SKILL.md Normal file
View File

@@ -0,0 +1,74 @@
---
name: org-json-bridge
description: "Provides a bridge between Org-mode files and JSON for programmatic manipulation. Use when: needing to parse Org-mode to JSON, serialize JSON to Org-mode, or modify Org-mode files programmatically. NOT for: simple text edits, general Org-mode viewing, or direct human authoring."
homepage: https://docs.openclaw.ai/tools/skills/org-json-bridge
metadata: { "openclaw": { "emoji": "🌉", "requires": { "bins": ["python3", "pip", "emacs"], "env": [] }, "primaryEnv": "" } }
---
# Org-JSON Bridge
This skill develops and utilizes an external tool to convert Org-mode files into a structured JSON representation, and vice-versa. This enables robust programmatic modification of Org-mode documents, addressing the limitations of direct string manipulation via the `edit` tool for complex structures like tables or source blocks. By working with a structured data model, it ensures consistent formatting and reliable updates.
## When to Use
✅ **USE this skill when:**
- Modifying Org-mode file content programmatically (e.g., adding/removing table rows, updating flags in structured blocks).
- Converting Org-mode documents to a JSON representation for data processing.
- Converting structured JSON data back into Org-mode format.
- Encountering difficulties with the `edit` tool due to complex Org-mode formatting (e.g., tables, source blocks).
❌ **DON'T use this skill when:**
- Performing simple, single-line text edits in Org-mode files.
- Viewing or rendering Org-mode content (use a dedicated Org-mode client).
- Authoring Org-mode documents directly.
- Editing existing skills (use `edit` tool directly for SKILL.md files).
## Instructions
This skill provides a Python script (`org_bridge.py`) that acts as a command-line interface to the Org-mode to JSON bridge.
1. **Parse Org-mode to JSON (Command Line):**
```bash
org_bridge.py parse --file-path "path/to/my-doc.org" > output.json
```
2. **Modify JSON (Internal Agent Logic):**
* The agent would then perform internal Python logic to load `output.json`, modify the JSON object (AST), and save the modified JSON to a new file, e.g., `modified_data.json`.
3. **Render JSON to Org-mode (Command Line):**
```bash
org_bridge.py render --json-input-file "path/to/modified_data.json" --output-file "path/to/new-doc.org"
```
## Commands
### `org_bridge.py parse`
Parse an Org-mode file into a JSON representation and print to stdout.
```bash
# Example: Parse an Org-mode file to JSON
exec ~/.openclaw/workspace/skills/org-json-bridge/org_bridge.py parse --file-path "/home/amr/.openclaw/workspace/memex/5_projects/agora/agora-requirements-01-overview.org"
```
### `org_bridge.py render`
Render a JSON representation back into an Org-mode file.
```bash
# Example: Render JSON back to an Org-mode file
# Assume 'modified_data.json' exists with the desired AST
exec ~/.openclaw/workspace/skills/org-json-bridge/org_bridge.py render --json-input-file "/path/to/modified_data.json" --output-file "/home/amr/.openclaw/workspace/memex/5_projects/agora/agora-requirements-01-overview.org"
```
## Configuration
Environment variables needed:
- None directly for the bridge, but the underlying parser might have config.
Config values in `openclaw.json`:
- `skill.org-json-bridge.parser_path` - Path to the Python/Node.js script (if not directly in skill folder).
## Notes
- Initial implementation will focus on core Org-mode elements needed for requirements documents (headings, lists, tables, source blocks).
- Requires installation of an Org-mode parsing library (e.g., `orgparse` for Python), though the Emacs Lisp handles the heavy lifting here.
- This skill is a foundational piece to enhance reliable document manipulation capabilities.

60
projects/org-json-bridge/org-json-bridge.el Normal file
View File

@@ -0,0 +1,60 @@
;;; org-json-bridge.el --- Bridge for LLM agents to manipulate Org-mode via JSON
(require 'org-element)
(require 'json)
(require 'cl-lib)
(defun org-json-bridge--clean-tree (element)
"Recursively convert an Org ELEMENT into a JSON-serializable format."
(cond
((listp element)
(let* ((type (car element))
(props (nth 1 element))
(children (nthcdr 2 element))
(cleaned-props nil))
(cl-loop for (key val) on props by 'cddr do
(unless (member key '(:standard-properties :parent))
(let ((json-key (substring (symbol-name key) 1)))
(push (cons json-key
(cond
((stringp val) val)
((numberp val) val)
((booleanp val) val)
(t (format "%s" val))))
cleaned-props))))
(list (cons 'type (symbol-name type))
(cons 'properties cleaned-props)
(cons 'contents (mapcar #'org-json-bridge--clean-tree children)))))
((stringp element) element)
(t (format "%s" element))))
(defun org-to-json (file-path)
"Parse an Org file and output its structure as JSON."
(with-current-buffer (find-file-noselect file-path)
(let* ((tree (org-element-parse-buffer))
(cleaned (org-json-bridge--clean-tree tree)))
(princ (json-encode cleaned)))))
(defun json-to-org (json-string output-file)
"Take a JSON representation of an Org tree and write it back to a file."
(let ((data (json-read-from-string json-string)))
(with-temp-file output-file
(insert (org-element-interpret-data data)))))
;; Entry point for batch mode
(message "DEBUG: Entry point reached")
;; Sometimes -- is left in command-line-args-left
(when (string= (car command-line-args-left) "--")
(pop command-line-args-left))
(let ((command (pop command-line-args-left)))
(message "DEBUG: Command is %s" command)
(cond
((string= command "org-to-json")
(let ((file (pop command-line-args-left)))
(org-to-json file)))
((string= command "json-to-org")
(let ((json-str (pop command-line-args-left))
(out-file (pop command-line-args-left)))
(json-to-org json-str out-file)))))

54
projects/org-json-bridge/org_bridge.py Executable file
View File

@@ -0,0 +1,54 @@
import subprocess
import json
import os
import argparse
from typing import Dict, Any, Optional
class OrgBridge:
def __init__(self, lisp_script_path: str = os.path.join(os.path.dirname(__file__), "org-json-bridge.el")):
self.lisp_path = os.path.abspath(lisp_script_path)
def _run_emacs_batch(self, command: str, *args) -> str:
"""Helper to execute the Emacs batch command with arguments."""
cmd = [
"emacs", "--batch",
"-l", self.lisp_path,
"--", command, *args
]
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
return result.stdout.strip()
def parse_to_dict(self, file_path: str) -> Dict[str, Any]:
"""Reads an Org file and returns its AST as a Python Dictionary."""
abs_path = os.path.abspath(file_path)
json_output = self._run_emacs_batch("org-to-json", abs_path)
return json.loads(json_output)
def write_from_dict(self, ast_dict: Dict[str, Any], output_path: str):
"""Takes a Python Dictionary (AST) and writes it back to an Org file."""
json_input = json.dumps(ast_dict)
abs_output_path = os.path.abspath(output_path)
self._run_emacs_batch("json-to-org", json_input, abs_output_path)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Org-mode to JSON bridge for programmatic manipulation.")
parser.add_argument("action", choices=["parse", "render"], help="Action to perform: 'parse' an Org file to JSON, or 'render' JSON to an Org file.")
parser.add_argument("--file-path", help="Path to the Org-mode file (required for 'parse' action).")
parser.add_argument("--json-input-file", help="Path to a JSON file containing the AST (required for 'render' action).")
parser.add_argument("--output-file", help="Path to output the Org-mode file (required for 'render' action).")
args = parser.parse_args()
bridge = OrgBridge()
if args.action == "parse":
if not args.file_path:
parser.error("--file-path is required for the 'parse' action.")
org_ast = bridge.parse_to_dict(args.file_path)
print(json.dumps(org_ast, indent=2))
elif args.action == "render":
if not args.json_input_file or not args.output_file:
parser.error("--json-input-file and --output-file are required for the 'render' action.")
with open(args.json_input_file, 'r') as f:
ast_dict = json.load(f)
bridge.write_from_dict(ast_dict, args.output_file)

43
projects/personal_server_appliance/README.org Normal file
View File

@@ -0,0 +1,43 @@
#+title: Personal Server Appliance Project
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:05]
#+begin_comment
Project documentation for Personal Server Appliance Project
#+end_comment
* Personal Server Appliance Project
*Goal:** To design and develop a modular personal server appliance, featuring various modules for compute, storage, networking, power, software-defined radio, and audio/video processors. The design aesthetic aims for sleek 10-inch units resembling modern Hi-Fi systems or standard 19-inch rack units.
*Key Concepts:**
- *Modularity:** Core design principle, allowing users to select and combine modules based on their needs.
- *Modules:**
- Compute
- Storage
- Networking
- Power
- Software-Defined Radio (SDR)
- Audio Processors
- Video Processors
- *Form Factor:**
- Sleek 10-inch units (Hi-Fi aesthetic)
- Standard 19-inch rack units
- *Openness:** Implicitly open-source for hardware and/or software.
*Initial Scope:**
- Defining the inter-module communication standards and power delivery mechanisms.
- Researching existing modular server solutions or embedded systems with similar goals.
- Conceptualizing the physical design and aesthetic requirements for both form factors.
- Identifying potential hardware components for each module.
*Information Needed from Amr:**
- Prioritization of modules for initial development (which ones are most critical?).
- Specific performance targets or constraints for compute and storage modules.
- Desired level of user-serviceability and upgradeability.
- Thoughts on the operating system or software stack for the base compute module.
- Any aesthetic preferences or specific design inspirations for the 10-inch units.
*Next Steps:**
1. Gather Amr's priorities and specific design preferences.
2. Begin outlining technical specifications for inter-module interfaces.
3. Research available modular hardware platforms.

40
projects/sdr_suite_lisp/README.org Normal file
View File

@@ -0,0 +1,40 @@
#+title: SDR Suite Development Project (Common Lisp)
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:02]
#+begin_comment
This file outlines the project to develop a comprehensive Software Defined Radio (SDR) suite using Common Lisp, with a focus on various applications.
#+end_comment
* SDR Suite Development Project (Common Lisp)
*Goal:** To develop a comprehensive Software Defined Radio (SDR) suite using Common Lisp, focusing on a wide range of applications.
*Proposed Applications:**
- Earth-Moon-Earth (EME)
- Active Link Establishment (ALE)
- Satellite communication
- Broadcast Reception (Rx)
- Phono (FM, AM, SSB, CW)
- Synchronous data
- Asynchronous data
- Slow-scan Television (SSTV)
- Computer networking
- Passive radar
- Directional Signal Finding (DSF)
- GPS Reception (Rx)
*Initial Scope:**
- Defining the core architecture of the SDR suite in Lisp.
- Breaking down each application into functional requirements.
- Identifying existing Lisp libraries or SDR frameworks that could be leveraged.
*Information Needed from Amr:**
- Prioritization of the proposed applications (which ones to tackle first?).
- Specific design philosophies or constraints for the Lisp implementation.
- Any existing thoughts on hardware integration or target SDR platforms.
- Desired level of modularity and extensibility.
*Next Steps:**
1. Gather Amr's requirements and priorities for the applications.
2. Begin conceptualizing the high-level architecture.
3. Research relevant Lisp and SDR technologies.

26
projects/token-optimization/README.org Normal file
View File

@@ -0,0 +1,26 @@
#+TITLE: Token Optimization
#+AUTHOR: Amr
#+CREATED: [2026-03-17 Tue]
#+BEGIN_COMMENT
Cost-effective LLM usage through smart routing, context compression, and multi-provider strategies.
#+END_COMMENT
* Token Optimization
Strategy and implementation for minimizing LLM costs while maintaining quality.
* Project Tasks
See the actionable tasks for this project in [[file:../../gtd.org::*Token Optimization][GTD.org > Projects > Token Optimization]]
* Key Documents
- [[file:plan.org][Optimization Plan]]
- [[file:token-optimization.yaml][Configuration]]
* Current Focus
- Multi-provider setup (Gemini primary, OpenRouter fallback)
- Usage tracking and budget alerts
- Smart routing by task type
- Context compression techniques

112
projects/token-optimization/budget-50.org Normal file
View File

@@ -0,0 +1,112 @@
#+TITLE: Token Optimization - $50 Monthly Budget
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:28]
#+DATE: 2026-03-04
#+FILETAGS: :budget:constraints:optimization
* Budget: $50/Month
** Budget Breakdown
| Tier | Provider | Allocation | Tokens Est. | Use Case |
|------|----------|-----------|-------------|----------|
| FREE | Google Gemini | $0 | ~9M/month | 90% of work |
| CHEAP | OpenRouter | $20 | ~6M tokens | Fallback, complex tasks |
| PREMIUM | Claude/GPT-4o | $25 | ~500K tokens | Critical decisions |
| BUFFER | Various | $5 | Emergency | Overruns, testing |
** Daily Free Allowance
- *Google Gemini:* 300K tokens/day = 9M/month = *$0*
- This covers 90-95% of expected workload
** Paid Tier Allocation ($45)
- *$20 → OpenRouter* (Qwen, Mistral, Llama)
- ~6M tokens at $0.003/1K
- Use when: Gemini rate limited, need different model
- *$25 → Premium models* (Claude, GPT-4o)
- ~500K tokens at $0.05/1K average
- Use when: Architecture decisions, critical code review, final validation
- *$5 → Buffer*
- Handle overruns
- Emergency access
- Testing new models
** Hard Limits
| Provider | Monthly Cap | Alert At |
|----------|-------------|----------|
| OpenRouter | $20 | $16 (80%) |
| Premium | $25 | $20 (80%) |
| Total | $50 | $45 (90%) |
** Daily Tracking
Target: *Monitor consumption every session*
```
IF daily_cost > $1.50:
→ Switch to Gemini only
→ Defer premium tasks
IF weekly_cost > $12:
→ Review usage patterns
→ Find optimization opportunities
```
** Emergency Protocol
If approaching $50 limit before month end:
1. Halt all paid API calls
2. Switch to Gemini-only mode
3. Queue premium tasks for next month
4. Consider local inference setup
** Cost-Per-Task Guidelines
| Task Type | Max Cost | Preferred Model |
|-----------|----------|-----------------|
| Quick lookup | $0.00 | Gemini |
| Code review | $0.01 | Gemini/OpenRouter |
| Feature design | $0.05 | OpenRouter |
| Architecture review | $0.10 | Claude/GPT-4o |
| Emergency debug | $0.20 | Best available |
** Optimization Imperative
With $50/month, waste is not affordable:
- ❌ No speculative queries
- ❌ No "just curious" premium calls
- ❌ No repeated similar prompts
- ✅ Always use Gemini first
- ✅ Batch similar requests
- ✅ Cache embeddings locally
- ✅ Summarize long contexts
** Monthly Review
1. Compare actual vs. projected usage
2. Adjust model routing rules
3. Identify expensive query patterns
4. Plan next month's allocation
** Break-Even Analysis
At $50/month = $600/year:
- *Option A:* Continue APIs (flexible, managed)
- *Option B:* Local inference (~$800 hardware, $0 ongoing)
- Break-even: 16 months
- Risk: Hardware failure, maintenance
*Recommendation:* Stick with APIs until $100+/month, then evaluate hardware.
** Questions for Human Partner
1. Is $50 firm or flexible in emergencies?
2. What happens if we hit limit mid-critical-task?
3. Preference for which premium model? (Claude vs GPT-4 vs both)
4. Should I track and report costs per project?
5. Any tasks that are "unlimited budget" critical?

215
projects/token-optimization/plan.org Normal file
View File

@@ -0,0 +1,215 @@
#+TITLE: Token Optimization Strategy
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:28]
#+DATE: 2026-03-04
#+FILETAGS: :strategy:token:optimization:cost
* Executive Summary
** Goal: Minimize inference costs while maximizing capability
Current approach: Single default model → Multi-tier, multi-provider strategy
* Three-Tier Model Strategy
** Tier 1: Fast/Cheap (80% of queries)
- *Purpose:* Simple tasks, formatting, lookups
- *Models:* Google Gemini Flash, Local models
- *Cost:* $0-0.000001 per 1K tokens
- *Speed:* Fastest
** Tier 2: Balanced (18% of queries)
- *Purpose:* Complex reasoning, code generation, analysis
- *Models:* Gemini Pro, Claude Haiku, Llama 3 70B
- *Cost:* $0.0001-0.003 per 1K tokens
- *Speed:* Medium
** Tier 3: High-Performance (2% of queries)
- *Purpose:* Critical decisions, complex architecture, final review
- *Models:* GPT-4, Claude Opus, Gemini Ultra
- *Cost:* $0.01-0.03 per 1K tokens
- *Speed:* Slower
* Provider Analysis
** Google AI Studio (Primary Recommended)
| Model | Free Tier | Rate Limit | Best For |
|-------|-----------|------------|----------|
| Gemini 2.0 Flash | 300K tokens/day | 60 req/min | Quick tasks, coding |
| Gemini 1.5 Flash | 300K tokens/day | 60 req/min | Fast responses |
| Gemini 1.5 Pro | 300K tokens/day | 60 req/min | Complex tasks |
*Cost: FREE (within limits)*
** OpenRouter.Aggregated (Secondary)
| Model | Price/1K tokens | Context | Reliability |
|-------|-----------------|---------|-------------|
| Qwen 3 235B | $0.0001-0.0003 | 128K | High |
| Mistral Large | $0.002-0.006 | 128K | High |
| Llama 4 405B | $0.0002-0.0005 | 128K | Medium |
| Free tier models | $0 | Varies | Variable |
** OpenAI (Tier 3 only)
- GPT-4: $0.03/1K tokens (expensive)
- GPT-4o: $0.005/1K tokens (better value)
- Use sparingly for critical tasks only
** Local Inference (Long-term goal)
- Hardware: $1000-5000 initial investment
- Ongoing: $0 (electricity only)
- Models: Llama 3, Mistral, DeepSeek
- Best for: High-volume, privacy-sensitive work
* Context Optimization Strategies
** 1. Context Windows by Task Type
| Task Type | Optimal Context | Compression | Savings |
|-----------|-----------------|-------------|---------|
| Code review | 4K-8K | Truncate old files | 50% |
| Documentation | 8K-16K | Summarize sections | 30% |
| Research | 16K-32K | Chunk + RAG | 70% |
| Architecture | 32K-128K | Maintain full | 0% |
** 2. Conversation Pruning
- Remove "thinking" blocks from history
- Summarize conversation every 10 turns
- Archive old sessions to external storage
** 3. RAG vs. Full Context
- *Rule:* < 5K tokens of context → Full
- *Rule:* > 10K tokens of context → Use embeddings/RAG
- *Savings:* 60-80% on large document tasks
* Request Optimization
** Batching Strategy
- Group similar requests (3-5 per batch)
- Same model, same parameters
- Shared overhead costs
** Caching Strategy
- Cache embeddings for repeated contexts
- Store common completions (templates)
- Reuse code snippet suggestions
** Streaming vs. Non-Stream
- *Streaming:* Better UX, but higher token overhead
- *Non-stream:* More efficient for programmatic use
- *Recommendation:* Non-stream for background tasks
* Smart Routing Rules
** Automatic Selection Logic
```
IF task_type == "simple_lookup" OR "formatting":
→ Gemini Flash (free)
ELIF task_type == "code_generation" AND complexity < 3:
→ Gemini Pro (free tier)
ELIF task_type == "complex_reasoning" OR "architecture":
→ Claude Sonnet or GPT-4o
ELIF task_type == "final_review" OR "critical_decision":
→ GPT-4 or Claude Opus
```
** Fallback Chain
1. Try Gemini (free)
2. If rate limited → OpenRouter (cheap)
3. If quality insufficient → GPT-4o
4. If critical failure → GPT-4
* Concrete Implementation
** Config Structure (openclaw.json)
```json
{
"models": {
"defaults": {
"primary": "google-gemini-cli/gemini-2.0-flash",
"fallbacks": [
"openrouter/qwen/qwen3-235b-a22b",
"google-gemini-cli/gemini-1.5-pro",
"openai/gpt-4o"
]
},
"providers": {
"google-gemini-cli": {
"freeTier": true,
"dailyLimit": 300000,
"rateLimit": 60
},
"openrouter": {
"freeTierModels": ["openrouter/auto"],
"budgetLimit": 500
},
"openai": {
"budgetLimit": 200,
"useFor": ["critical", "architecture"]
}
}
}
}
```
** Monitoring & Alerts
- Track daily token usage per provider
- Alert at 80% of free tier limits
- Monthly budget review and adjustment
* Cost Projections
** Current Unknown Usage → Optimized
| Scenario | Monthly Tokens | Current Cost | Optimized Cost | Savings |
|----------|---------------|--------------|----------------|---------|
| Light (< 1M) | 1M | $50-100 | $0-10 | 90% |
| Medium (1-5M) | 3M | $200-500 | $20-100 | 80% |
| Heavy (5-20M) | 10M | $1000-3000 | $200-500 | 80% |
* Immediate Actions
** Week 1: Setup
- Configure Gemini as primary provider
- Set up OpenRouter fallback
- Implement basic usage tracking
- Document current baseline
** Week 2: Implement
- Add smart routing logic
- Implement context compression
- Set up budget alerts
- A/B test model choices
** Week 3: Optimize
- Analyze usage patterns
- Fine-tune routing rules
- Tune context windows
- Document findings
** Week 4: Scale
- Full multi-provider setup
- Implement full caching
- Maximize free tier usage
- Plan for paid tiers if needed
* Long-term: Local Inference Path
** Minimum Viable Setup
- Hardware: RTX 4090 or Apple Silicon M3 Max
- Software: Ollama + OpenClaw integration
- Cost: ~$2000-4000 one-time
- Break-even: 3-6 months vs. API costs
** Full Self-Hosted
- Hardware: Dual RTX 4090 or 2x Mac Studio
- Models: Llama 3 70B, Mixtral 8x22B
- Cost: ~$8000-12000
- For: Privacy, unlimited inference, control

39
projects/token-optimization/quick-start.org Normal file
View File

@@ -0,0 +1,39 @@
#+TITLE: Token Optimization - Quick Start
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:28]
#+DATE: 2026-03-04
* Quick Reference for Daily Use
** Rule of Thumb
| What you need | Use this | Cost |
|---------------|----------|------|
| Quick answer, formatting, lookup | Gemini Flash | FREE |
| Code review, analysis | Gemini Pro | FREE |
| Complex problem solving | Claude Haiku / Qwen | $ |
| Critical architecture decision | GPT-4o | $$ |
** Free Tier Limits (Daily)
| Provider | Tokens | Requests | Reset |
|----------|--------|----------|-------|
| Google AI Studio | 300,000 | 60/min | Daily |
| OpenRouter Free | Varies | Limited | - |
** Current Recommendation
→ *Use Google Gemini exclusively* until hitting 250K tokens/day
→ Then add OpenRouter fallback
→ Only use GPT-4 for final reviews
** This will reduce token costs by ~90%
** Next Steps
1. Configure Gemini as primary (already partially done)
2. Add quota tracking
3. Set alerts at 80% of free limits
4. Implement tiered routing
** Savings Potential: $100-500/month → $10-50/month

67
projects/token-optimization/research.org Normal file
View File

@@ -0,0 +1,67 @@
#+TITLE: Token Management & Model Optimization Research
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:28]
#+DATE: 2026-03-04
#+FILETAGS: :research:token:optimization:models
* Token Management Strategy Research
** Initial Findings
*** OpenRouter Free Tier
- URL: https://openrouter.ai/collections/free-models
- Providers moving from free to paid-only models
- Belief: "Free models play crucial role in democratizing access"
*** Google AI Studio (Gemini)
- Free tier available
- Limits: 60 requests/minute, 300K tokens/day
- No credit card required
- Every API key gets these limits
** Research Questions
1. Which providers offer free or low-cost tiers?
2. What are the rate limits and quotas?
3. Which models are best for which use cases?
4. How to optimize context windows?
5. What is the cost per token breakdown?
** To Research Further
| Provider | Free Tier | Paid Tier | Best For |
|----------|-----------|-----------|----------|
| Google Gemini | 300K tokens/day | Pay per use? | General, coding |
| OpenRouter | Varies by model | Per-request | Routing, variety |
| OpenAI | ? | ? | GPT-4 quality |
| Anthropic | ? | ? | Claude capabilities |
| Mistral | ? | ? | Open weights |
| Local | Hardware cost | Free | Privacy, control |
** Token Optimization Strategies to Explore
1. *Tiered Model Usage*
- Simple tasks: Fast/cheap models
- Complex tasks: Stronger models
- Fallback: Lower tier if higher fails
2. *Context Compression*
- Summarize long contexts
- Use RAG instead of full context
- Prune old conversation
3. *Caching*
- Cache common responses
- Reuse embeddings
- Batch requests
4. *Hybrid Approach*
- Local models for simple queries
- Cloud APIs for complex tasks
- Manual review for critical outputs
** X Account Access
*Pending:* X account access via Google login
*Blocker:* Requires OTP from user per security rule (SOUL.md)
*Action needed:* User provides OTP, I complete OAuth, access bookmarks

30
projects/zotero_org_import_tool/README.org Normal file
View File

@@ -0,0 +1,30 @@
#+title: Zotero to Org-mode Import Tool Project
#+author: Amero Garcia
#+created: [2026-03-16 Mon 14:05]
#+begin_comment
Project documentation for Zotero to Org-mode Import Tool Project
#+end_comment
* Zotero to Org-mode Import Tool Project
*Goal:** To develop a "makeshift" tool for importing a Zotero library into Org-mode, with key functionalities:
1. *Academic Papers:** Accurately track and link to attached academic papers.
2. *Web Bookmarks:** Create web snapshots of imported web bookmarks for persistence and future reference.
*Initial Scope:**
- Understanding Zotero's data export formats (e.g., BibTeX, JSON, RIS).
- Determining the optimal way to represent Zotero items (publications, notes, web pages) within Org-mode.
- Designing a mechanism to locate and link to attached PDF/other files from Org-mode entries.
- Identifying tools or methods for creating persistent web snapshots (e.g., using a web archiving service, local archiving).
*Information Needed from Amr:**
- Preferred Zotero export format for data import.
- Desired Org-mode structure for imported Zotero entries.
- How you envision the links to academic papers working within Org-mode (e.g., direct file link, Zotero URI).
- Any specific web archiving tools or methods you have in mind for the web snapshots.
- Whether this tool should be Emacs Lisp-based, Python, or another language.
*Next Steps:**
1. Gather Amr's requirements for Zotero data handling and Org-mode output.
2. Research Zotero API/export capabilities and web archiving strategies.
3. Propose a high-level design for the import tool.