#+TITLE: Root Cause Analysis: Individual Provider Track Verification
#+DATE: 2026-04-11
#+FILETAGS: :rca:providers:llm:testing:autonomy:

* Executive Summary
Verified the unified LLM gateway implementation for all 6 individual provider tracks (Anthropic, Gemini, Groq, OpenAI, OpenRouter, Ollama). Identified and resolved critical parsing failures in the Gemini track and integration gaps in the system build definition.

* 1. Issue: Fragile Response Parsing (Gemini)
** Symptoms
Gemini API responses were returning `NIL` content during mocked unit tests, despite the JSON structure being seemingly correct.
** Root Cause
Recursive `assoc` / `car` / `cdr` chains were hardcoded and brittle. Specifically, the Gemini extraction logic was incorrectly attempting to treat a single alist pair as a list of pairs, causing `assoc` to fail on the `:TEXT` key.
** Resolution
Implemented a robust `get-nested` helper function that safely traverses both nested objects (alists) and arrays (lists of alists). This normalized the extraction logic across all providers.

* 2. Issue: Decoupled Build Configuration
** Symptoms
Provider logic was present in the codebase but inaccessible during tests and runtime.
** Root Cause
The `credentials-vault.lisp` and `llm-gateway.lisp` files (consolidated in a previous session) were never added to the `opencortex.asd` system definition. Furthermore, an incorrect loading order caused `UNDEFINED-FUNCTION` errors for `register-probabilistic-backend`.
** Resolution
1. Added both files to `opencortex.asd`.
2. Enforced strict loading order: `probabilistic` (defines registry) -> `credentials-vault` -> `llm-gateway` (uses registry).

* 3. Issue: Credential Key Mismatch
** Symptoms
Gemini requests failed with "API Key missing" even when environment variables were set.
** Root Cause
`llm-gateway` requested secrets for the `:gemini-api` provider, but the `credentials-vault` fallback logic only recognized the `:gemini` keyword.
** Resolution
Updated `vault-get-secret` to map both `:gemini` and `:gemini-api` to the same `GEMINI_API_KEY` environment variable.

* 4. opencortex Mandate Alignment
** Invariant Check
- *High-Integrity Memory:* All individual provider tracks are now backed by automated unit tests (`llm-gateway-tests.lisp`).
- *Literate Programming:* Updated `org-skill-llm-gateway.org` to reflect the improved `get-nested` utility.

* 5. Permanent Learnings
- **Tooling vs Source:** Tangled `.lisp` files are not enough; always ensure new modules are registered in the `.asd` file to be part of the official kernel build.
- **Robustness over Brevity:** Use abstraction helpers like `get-nested` instead of deep `car/cdr` chains when dealing with external JSON structures that may have varying array/object nesting.