FEAT: Verify all LLM providers and fix Gemini parsing

docs/rca/rca-provider-verification.org

@@ -0,0 +1,40 @@
+#+TITLE: Root Cause Analysis: Individual Provider Track Verification
+#+DATE: 2026-04-11
+#+FILETAGS: :rca:providers:llm:testing:psf:
+
+* Executive Summary
+Verified the unified LLM gateway implementation for all 6 individual provider tracks (Anthropic, Gemini, Groq, OpenAI, OpenRouter, Ollama). Identified and resolved critical parsing failures in the Gemini track and integration gaps in the system build definition.
+
+* 1. Issue: Fragile Response Parsing (Gemini)
+** Symptoms
+Gemini API responses were returning `NIL` content during mocked unit tests, despite the JSON structure being seemingly correct.
+** Root Cause
+Recursive `assoc` / `car` / `cdr` chains were hardcoded and brittle. Specifically, the Gemini extraction logic was incorrectly attempting to treat a single alist pair as a list of pairs, causing `assoc` to fail on the `:TEXT` key.
+** Resolution
+Implemented a robust `get-nested` helper function that safely traverses both nested objects (alists) and arrays (lists of alists). This normalized the extraction logic across all providers.
+
+* 2. Issue: Decoupled Build Configuration
+** Symptoms
+Provider logic was present in the codebase but inaccessible during tests and runtime.
+** Root Cause
+The `credentials-vault.lisp` and `llm-gateway.lisp` files (consolidated in a previous session) were never added to the `org-agent.asd` system definition. Furthermore, an incorrect loading order caused `UNDEFINED-FUNCTION` errors for `register-neuro-backend`.
+** Resolution
+1. Added both files to `org-agent.asd`.
+2. Enforced strict loading order: `neuro` (defines registry) -> `credentials-vault` -> `llm-gateway` (uses registry).
+
+* 3. Issue: Credential Key Mismatch
+** Symptoms
+Gemini requests failed with "API Key missing" even when environment variables were set.
+** Root Cause
+`llm-gateway` requested secrets for the `:gemini-api` provider, but the `credentials-vault` fallback logic only recognized the `:gemini` keyword.
+** Resolution
+Updated `vault-get-secret` to map both `:gemini` and `:gemini-api` to the same `GEMINI_API_KEY` environment variable.
+
+* 4. PSF Mandate Alignment
+** Invariant Check
+- *High-Integrity Memory:* All individual provider tracks are now backed by automated unit tests (`llm-gateway-tests.lisp`).
+- *Literate Programming:* Updated `org-skill-llm-gateway.org` to reflect the improved `get-nested` utility.
+
+* 5. Permanent Learnings
+- **Tooling vs Source:** Tangled `.lisp` files are not enough; always ensure new modules are registered in the `.asd` file to be part of the official kernel build.
+- **Robustness over Brevity:** Use abstraction helpers like `get-nested` instead of deep `car/cdr` chains when dealing with external JSON structures that may have varying array/object nesting.