memex/notes/org-skill-embedding-generator.org

SKILL: Embedding Generator (Universal Literate Note)

• Overview
• Phase A: Demand (PRD)
  • 1. Purpose
  • 2. User Needs
• Phase B: Blueprint (PROTOCOL)
  • Phase B: Blueprint (PROTOCOL)
    • 1. Architectural Intent
    • 2. Semantic Interfaces

Overview

The Embedding Generator provides vector representations of text for semantic search and neural memory.

Phase A: Demand (PRD)

1. Purpose

Generate embeddings for text strings.

2. User Needs

• Provider Choice: Support for local (Ollama) or remote (Gemini, OpenAI) providers.
• Batching: Efficiency through batching text (future-proof).

Phase B: Blueprint (PROTOCOL)

Phase B: Blueprint (PROTOCOL)

1. Architectural Intent

This system aims for a flexible and extensible architecture that can accommodate different embedding providers (local or remote) while maintaining a consistent interface for the user. It also seeks to optimize for batch processing to improve throughput. The core design principle is provider abstraction.

2. Semantic Interfaces

;;; Primary Function: Generate Embeddings
;;; Input: A list of strings, and provider configuration
;;; Output: A list of embedding vectors (lists of floats).

(defun generate-embeddings (texts provider-config)
  """Generates embeddings for a list of texts using the specified provider.""")

;;; Provider Configuration Structure
;;; This plist defines the provider to use, and any necessary credentials or parameters.

;;; Example: Local Ollama provider
;;; (:provider :ollama :model "mistralai/Mistral-7B-Instruct-v0.2")

;;; Example: Remote Gemini provider
;;; (:provider :gemini :api-key "YOUR_API_KEY" :model "gemini-1.5-pro")

;;; Example: Remote OpenAI provider.
;;; (:provider :openai :api-key "YOUR_API_KEY" :model "text-embedding-ada-002")

;;; Sub-Function: (Abstract) Provider-Specific Embedding Generation
;;; This function is implemented differently for each provider.

(defgeneric generate-embeddings-from-provider (texts provider-config))

;;; Example implementation for :ollama provider
(defmethod generate-embeddings-from-provider (texts (provider-config (eql (getf provider-config :provider :ollama))))
  """Generates embeddings using a local Ollama server.""")

;;; Example implementation for :gemini provider
(defmethod generate-embeddings-from-provider (texts (provider-config (eql (getf provider-config :provider :gemini))))
 """Generates embeddings using the Gemini API.""")

;;; Example implementation for :openai provider
(defmethod generate-embeddings-from-provider (texts (provider-config (eql (getf provider-config :provider :openai))))
 """Generates embeddings using the OpenAI API.""")