:PROPERTIES: :ID: 6cd0b4d7-e7cf-49ba-a847-e81c4ace0f76 :END: #+TITLE: SKILL: Embedding Generator (Universal Literate Note) #+STARTUP: content #+FILETAGS: :embeddings:neural:psf: * Overview The *Embedding Generator* provides vector representations of text for semantic search and neural memory. * Phase A: Demand (PRD) :PROPERTIES: :STATUS: FROZEN :END: ** 1. Purpose Generate embeddings for text strings. ** 2. User Needs - *Provider Choice:* Support for local (Ollama) or remote (Gemini, OpenAI) providers. - *Batching:* Efficiency through batching text (future-proof). * Phase B: Blueprint (PROTOCOL) :PROPERTIES: :STATUS: SIGNED :END: ** Phase B: Blueprint (PROTOCOL) :PROPERTIES: :STATUS: DRAFT :END: *** 1. Architectural Intent This system aims for a flexible and extensible architecture that can accommodate different embedding providers (local or remote) while maintaining a consistent interface for the user. It also seeks to optimize for batch processing to improve throughput. The core design principle is *provider abstraction*. *** 2. Semantic Interfaces #+BEGIN_SRC lisp ;;; Primary Function: Generate Embeddings ;;; Input: A list of strings, and provider configuration ;;; Output: A list of embedding vectors (lists of floats). (defun generate-embeddings (texts provider-config) """Generates embeddings for a list of texts using the specified provider.""") ;;; Provider Configuration Structure ;;; This plist defines the provider to use, and any necessary credentials or parameters. ;;; Example: Local Ollama provider ;;; (:provider :ollama :model "mistralai/Mistral-7B-Instruct-v0.2") ;;; Example: Remote Gemini provider ;;; (:provider :gemini :api-key "YOUR_API_KEY" :model "gemini-1.5-pro") ;;; Example: Remote OpenAI provider. ;;; (:provider :openai :api-key "YOUR_API_KEY" :model "text-embedding-ada-002") ;;; Sub-Function: (Abstract) Provider-Specific Embedding Generation ;;; This function is implemented differently for each provider. (defgeneric generate-embeddings-from-provider (texts provider-config)) ;;; Example implementation for :ollama provider (defmethod generate-embeddings-from-provider (texts (provider-config (eql (getf provider-config :provider :ollama)))) """Generates embeddings using a local Ollama server.""") ;;; Example implementation for :gemini provider (defmethod generate-embeddings-from-provider (texts (provider-config (eql (getf provider-config :provider :gemini)))) """Generates embeddings using the Gemini API.""") ;;; Example implementation for :openai provider (defmethod generate-embeddings-from-provider (texts (provider-config (eql (getf provider-config :provider :openai)))) """Generates embeddings using the OpenAI API.""") #+END_SRC