memex/system/plans/flight-plan-task-orchestrator.org

Flight Plan: Consolidation VI - Task Orchestrator

• Background & Motivation
• Scope & Impact
• Proposed Solution
• Alternatives Considered
• Implementation Plan
  • Phase C: Success (Testing)
  • Phase D: Build (Engineering)
• Verification & Rollback

Background & Motivation

The `cognitive-cycle` pipeline (Consolidation V) established the functional gates (`perceive`, `neuro`, `consensus`, `decide`, `dispatch`). However, the `consensus-gate` remains a pass-through. Furthermore, the `opencortex` currently lacks the ability to formally verify GTD state transitions (Task Integrity) and hand off complex logic to background threads (Delegation). The Task Orchestrator integrates these three capabilities.

Scope & Impact

• Target Files: `projects/passepartout/src/core.lisp`, `projects/passepartout/src/neuro.lisp`, `projects/passepartout/src/symbolic.lisp`.
• Impact: The Probabilistic Engine layer will spawn multiple threads to query providers simultaneously. Deterministic Engine will enforce GTD state integrity and manage sub-agent delegation.
• Dependencies: Requires `bordeaux-threads` for sub-agent management and the existing `org-gtd` v4.0 DAG.

Proposed Solution

1. Parallel Probabilistic Engine (`neuro.lisp`): Modify `ask-neuro` to dispatch asynchronous requests to the `*provider-cascade*` using `bt:make-thread`.
2. Deterministic Consensus (`core.lisp`): Implement `consensus-gate` to wait for all parallel Probabilistic Engine proposals. If they match, select the candidate. If they differ, query a fast, deterministic model (e.g., `groq`) to judge and select the safest Lisp `(:REQUEST …)` form.
3. Task Integrity (`symbolic.lisp`): Add `task-integrity-check` inside `decide-gate`. If a task transition (e.g., `TODO` -> `DONE`) violates semantic rules (e.g., active children exist), reject the action and return an error stimulus to Probabilistic Engine.
4. Delegation Hooks: Define the `delegate-task` action type within the `dispatch-gate`, allowing the core to spawn isolated Lisp environments for long-running sub-agents.

Alternatives Considered

• Fast Parallel Race: First LLM to return a valid Lisp request wins. Rejected because it prioritizes speed over accuracy, violating the PSF safety mandates.

Implementation Plan

Phase C: Success (Testing)

• Draft `projects/passepartout/tests/orchestrator-test.lisp`.
• Assert that `consensus-gate` correctly handles 3 diverging proposals and selects the safest one.
• Assert that `task-integrity-check` rejects closing a parent task with active children.

Phase D: Build (Engineering)

• Update `neuro.lisp` with `bt:make-thread` logic for `ask-neuro`.
• Implement `consensus-gate` in `core.lisp`.
• Implement `task-integrity-check` in `symbolic.lisp` and hook it into `decide-gate`.
• Implement `delegate-task` in `core.lisp` dispatch logic.

Verification & Rollback

• Verification: Hot-load the kernel and trigger a conflicting prompt. Verify the consensus judge resolves it.
• Rollback: The pipeline is functional. If the new `consensus-gate` fails, we can instantly revert it to the `(first proposals)` pass-through via Git and Swank hot-reloading.