RAG POC

2 Squirrels AI

Technology Stack

Framework LangChain
Vector DB ChromaDB
Vision OpenCLIP
LLM OpenAI GPT-4
Text Embeddings text-embedding-ada

Architecture Note: Multi-modal RAG system supporting both text and image embeddings for cross-modal retrieval and semantic understanding.

Multi-Modal Embedding Components

📝
Text Embeddings
OpenAI text-embedding-ada
🖼️
Image Embeddings
OpenCLIP ViT-B/32
🔗
Cross-Modal
Unified Vector Space
🗄️
Vector Store
ChromaDB Collections

Basic RAG Pipeline

%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#C17852', 'primaryTextColor': '#F0F6FC', 'primaryBorderColor': '#4A5E32', 'lineColor': '#E6C98F', 'secondaryColor': '#161B22', 'tertiaryColor': '#0D1117', 'background': '#0D1117', 'mainBkg': '#161B22', 'nodeBorder': '#4A5E32', 'clusterBkg': '#161B22', 'clusterBorder': '#4A5E32', 'titleColor': '#E6C98F', 'edgeLabelBackground': '#161B22'}}}%%
flowchart LR
    subgraph Input["📥 Input"]
        Q["🔍 User Query"]
    end

    subgraph Embed["🔢 Embedding"]
        QE["🔢 Query Embedding
text-embedding-ada"]
    end

    subgraph Retrieve["📚 Retrieval"]
        VS[("🗄️ ChromaDB")]
        SS["🎯 Similarity Search
top_k=5"]
        RD["📄 Retrieved Docs"]
    end

    subgraph Augment["📝 Augmentation"]
        CT["📋 Context Template"]
        PR["📝 Prompt Builder"]
    end

    subgraph Generate["🧠 Generation"]
        LLM["🤖 GPT-4"]
        RS["✅ Response"]
    end

    Q --> QE
    QE --> VS
    VS --> SS
    SS --> RD
    RD --> CT
    CT --> PR
    Q --> PR
    PR --> LLM
    LLM --> RS

Multi-Modal Document Ingestion

%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#C17852', 'primaryTextColor': '#F0F6FC', 'primaryBorderColor': '#4A5E32', 'lineColor': '#E6C98F', 'secondaryColor': '#161B22', 'tertiaryColor': '#0D1117', 'background': '#0D1117', 'mainBkg': '#161B22', 'nodeBorder': '#4A5E32', 'clusterBkg': '#161B22', 'clusterBorder': '#4A5E32', 'titleColor': '#E6C98F', 'edgeLabelBackground': '#161B22'}}}%%
flowchart TB
    subgraph Sources["📥 Data Sources"]
        PDF["📄 PDF Documents"]
        IMG["🖼️ Images"]
        TXT["📝 Text Files"]
    end

    subgraph Parse["⚙️ Parsing"]
        PL["📄 PDF Loader"]
        IL["🖼️ Image Loader"]
        TL["📝 Text Loader"]
    end

    subgraph Chunk["✂️ Chunking"]
        TS["✂️ Text Splitter
RecursiveCharacter"]
        IC["🖼️ Image Chunks
Caption + Embed"]
    end

    subgraph Embed["🔢 Embedding"]
        TE["📝 Text Embeddings
OpenAI"]
        IE["🖼️ Image Embeddings
OpenCLIP"]
    end

    subgraph Store["💾 Storage"]
        TC[("📝 Text Collection")]
        IC2[("🖼️ Image Collection")]
        META[("📋 Metadata")]
    end

    PDF --> PL
    IMG --> IL
    TXT --> TL
    PL --> TS
    IL --> IC
    TL --> TS
    TS --> TE
    IC --> IE
    TE --> TC
    IE --> IC2
    TE --> META
    IE --> META

Text Vectorization Pipeline

%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#C17852', 'primaryTextColor': '#F0F6FC', 'primaryBorderColor': '#4A5E32', 'lineColor': '#E6C98F', 'secondaryColor': '#161B22', 'tertiaryColor': '#0D1117', 'background': '#0D1117', 'mainBkg': '#161B22', 'nodeBorder': '#4A5E32', 'clusterBkg': '#161B22', 'clusterBorder': '#4A5E32', 'titleColor': '#E6C98F', 'edgeLabelBackground': '#161B22'}}}%%
flowchart LR
    subgraph Doc["📄 Document"]
        RAW["📜 Raw Text"]
    end

    subgraph Preprocess["⚙️ Preprocessing"]
        CLN["🧹 Clean Text"]
        NRM["📐 Normalize"]
    end

    subgraph Split["✂️ Splitting"]
        RC["✂️ RecursiveCharacter
chunk=1000, overlap=200"]
        CH["📦 Chunks"]
    end

    subgraph Embed["🔢 Embedding"]
        API["🔌 OpenAI API"]
        VEC["📊 1536-dim Vectors"]
    end

    subgraph Store["💾 ChromaDB"]
        COL[("📦 Collection")]
        IDX["🔍 HNSW Index"]
    end

    RAW --> CLN
    CLN --> NRM
    NRM --> RC
    RC --> CH
    CH --> API
    API --> VEC
    VEC --> COL
    COL --> IDX

Cross-Modal Retrieval

%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#C17852', 'primaryTextColor': '#F0F6FC', 'primaryBorderColor': '#4A5E32', 'lineColor': '#E6C98F', 'secondaryColor': '#161B22', 'tertiaryColor': '#0D1117', 'background': '#0D1117', 'mainBkg': '#161B22', 'nodeBorder': '#4A5E32', 'clusterBkg': '#161B22', 'clusterBorder': '#4A5E32', 'titleColor': '#E6C98F', 'edgeLabelBackground': '#161B22'}}}%%
flowchart TB
    subgraph Input["📥 Query Input"]
        TQ["📝 Text Query"]
        IQ["🖼️ Image Query"]
    end

    subgraph Embed["🔢 Query Embedding"]
        TE["📝 Text Embedding
OpenAI"]
        IE["🖼️ Image Embedding
OpenCLIP"]
    end

    subgraph Project["🔗 Projection"]
        UP["📐 Unified Space
CLIP Alignment"]
    end

    subgraph Search["🔍 Parallel Search"]
        ST["📝 Search Text Collection"]
        SI["🖼️ Search Image Collection"]
    end

    subgraph Merge["🔗 Result Fusion"]
        RR["🎯 Reciprocal Rank Fusion"]
        FD["📊 Fused Documents"]
    end

    subgraph Output["📤 Output"]
        RS["✅ Multi-Modal Results"]
    end

    TQ --> TE
    IQ --> IE
    TE --> UP
    IE --> UP
    UP --> ST
    UP --> SI
    ST --> RR
    SI --> RR
    RR --> FD
    FD --> RS

Retrieval with Reranking

%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#C17852', 'primaryTextColor': '#F0F6FC', 'primaryBorderColor': '#4A5E32', 'lineColor': '#E6C98F', 'secondaryColor': '#161B22', 'tertiaryColor': '#0D1117', 'background': '#0D1117', 'mainBkg': '#161B22', 'nodeBorder': '#4A5E32', 'clusterBkg': '#161B22', 'clusterBorder': '#4A5E32', 'titleColor': '#E6C98F', 'edgeLabelBackground': '#161B22'}}}%%
flowchart LR
    subgraph Query["📥 Query"]
        Q["🔍 User Question"]
    end

    subgraph Initial["📚 Initial Retrieval"]
        VS[("🗄️ ChromaDB")]
        IR["🎯 top_k=20
Candidates"]
    end

    subgraph Rerank["🔄 Reranking"]
        CR["🔍 Cross-Encoder
Relevance Score"]
        TP["📊 Rerank Scores"]
        SR["🏆 Top N Results"]
    end

    subgraph Generate["🧠 Generation"]
        CTX["📝 Context Window"]
        LLM["🤖 GPT-4"]
    end

    subgraph Output["📤 Output"]
        ANS["✅ Answer"]
    end

    Q --> VS
    VS --> IR
    IR --> CR
    Q --> CR
    CR --> TP
    TP --> SR
    SR --> CTX
    CTX --> LLM
    LLM --> ANS

Complete Multi-Modal RAG Architecture

%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#C17852', 'primaryTextColor': '#F0F6FC', 'primaryBorderColor': '#4A5E32', 'lineColor': '#E6C98F', 'secondaryColor': '#161B22', 'tertiaryColor': '#0D1117', 'background': '#0D1117', 'mainBkg': '#161B22', 'nodeBorder': '#4A5E32', 'clusterBkg': '#161B22', 'clusterBorder': '#4A5E32', 'titleColor': '#E6C98F', 'edgeLabelBackground': '#161B22'}}}%%
flowchart TB
    subgraph User["👤 User Interface"]
        UI["💻 Chat Interface"]
    end

    subgraph Query["🔍 Query Processing"]
        QA["📝 Query Analysis"]
        QE["🔢 Query Embedding"]
    end

    subgraph Retrieval["📚 Retrieval Layer"]
        direction TB
        TC[("📝 Text
Collection")]
        IC[("🖼️ Image
Collection")]
        HY["🔗 Hybrid Search"]
    end

    subgraph Embeddings["🔢 Embedding Models"]
        OAI["📝 OpenAI
text-embedding-ada"]
        CLIP["🖼️ OpenCLIP
ViT-B/32"]
    end

    subgraph Generation["🧠 Generation"]
        CTX["📋 Context Builder"]
        GPT["🤖 GPT-4"]
    end

    subgraph Output["📤 Response"]
        ANS["✅ Answer"]
        SRC["📎 Sources"]
    end

    UI --> QA
    QA --> QE
    QE --> OAI
    QE --> CLIP
    OAI --> TC
    CLIP --> IC
    TC --> HY
    IC --> HY
    HY --> CTX
    CTX --> GPT
    GPT --> ANS
    GPT --> SRC
    ANS --> UI
    SRC --> UI