# Qdrant: Embedded vs Server Mode

## Overview

This document explains the differences between running Qdrant in **Embedded Mode** (current implementation) vs **Server Mode**, the changes required, and migration strategy.

---

## Table of Contents

1. [Key Differences](#key-differences)
2. [Architecture Comparison](#architecture-comparison)
3. [Code Changes Required](#code-changes-required)
4. [API Methods Mapping](#api-methods-mapping)
5. [Data Migration](#data-migration)
6. [APIs to Expose](#apis-to-expose)
7. [Performance Comparison](#performance-comparison)
8. [Breaking Changes](#breaking-changes)
9. [Rollout Strategy](#rollout-strategy)
10. [Decision Matrix](#decision-matrix)

---

## Key Differences

### Current State: Embedded Qdrant (File-Based)

**Architecture:**
```
Your Rust App
    ↓
EmbeddedQdrantVectorStore (file-based simulation)
    ↓
Local File System
    ↓
./qdrant-data/
    ├── chat_history-documents.json
    ├── chat_history-vectors.bin
    ├── chat_history-vector-index.json
    └── aws_estate-documents.json
```

**Characteristics:**

| Feature | Status | Notes |
|---------|--------|-------|
| Process Model | ✅ Single process | Everything in your app |
| Network Calls | ✅ None | Direct file I/O |
| Speed | ✅ Fast | No network latency |
| Data Storage | ✅ JSON files | Easy to inspect |
| Scalability | ❌ Limited | Single machine only |
| Concurrent Access | ❌ No | One app at a time |
| Web UI | ❌ No | No built-in dashboard |
| Vector Search | ⚠️ Simulated | Basic cosine similarity |
| Production Ready | ⚠️ Dev/Testing | Not for scale |

---

### Future State: Qdrant Server (Client-Server)

**Architecture:**
```
Your Rust App (Client)
    ↓ HTTP/gRPC (Network)
Qdrant Server (EC2/Docker)
    ↓
Real Vector Database
    ↓
/qdrant/storage/
    ├── collections/
    │   ├── chat_history/
    │   │   ├── segments/
    │   │   └── wal/
    │   └── aws_estate/
    │       ├── segments/
    │       └── wal/
    └── meta/
```

**Characteristics:**

| Feature | Status | Notes |
|---------|--------|-------|
| Process Model | ✅ Client-Server | Separate processes |
| Network Calls | ✅ REST/gRPC | HTTP or gRPC protocol |
| Speed | ✅ Very Fast | Optimized HNSW indexing |
| Data Storage | ✅ Native format | Binary segments + WAL |
| Scalability | ✅ High | Horizontal scaling possible |
| Concurrent Access | ✅ Yes | Multiple clients supported |
| Web UI | ✅ Yes | Built-in dashboard at `:6333/dashboard` |
| Vector Search | ✅ Production | HNSW algorithm |
| Production Ready | ✅ Yes | Battle-tested |

---

## Architecture Comparison

### Embedded Mode (Current)

```rust
// All in one process
RagModule {
    vector_store: EmbeddedQdrantVectorStore {
        // Reads/writes JSON files directly
        chat_history_file: "./qdrant-data/chat_history-documents.json",
        vectors_file: "./qdrant-data/chat_history-vectors.bin",
    }
}

// Operations
save_chat_history()
    → Append to JSON file
    → Update vector index file
    → Save to disk
    → Done (no network)
```

**Pros:**
- ✅ Simple setup (no server needed)
- ✅ Fast for small datasets (<10k docs)
- ✅ Easy debugging (inspect JSON files)
- ✅ Works offline
- ✅ No infrastructure costs

**Cons:**
- ❌ Not scalable
- ❌ File I/O bottleneck
- ❌ Manual vector search (slow for large datasets)
- ❌ No concurrent access
- ❌ No monitoring tools

---

### Server Mode (Future)

```rust
// Client-server architecture
RagModule {
    vector_store: QdrantClientVectorStore {
        client: QdrantClient {
            url: "http://ec2-instance:6333",
        }
    }
}

// Operations
save_chat_history()
    → Serialize to PointStruct
    → HTTP POST to /collections/{name}/points
    → Server indexes with HNSW
    → Response received
    → Done
```

**Pros:**
- ✅ Highly scalable
- ✅ Fast vector search (HNSW)
- ✅ Multiple clients can connect
- ✅ Built-in web dashboard
- ✅ Monitoring & metrics
- ✅ Production-ready

**Cons:**
- ❌ Requires server setup
- ❌ Network latency (5-10ms)
- ❌ Infrastructure costs (EC2)
- ❌ More complex deployment
- ❌ Requires migration

---

## Code Changes Required

### 1. Add Dependencies

**Update `Cargo.toml`:**

```toml
[dependencies]
# Existing dependencies...

# Add Qdrant client
qdrant-client = "1.7"
tonic = "0.10"          # For gRPC
prost = "0.12"          # For Protocol Buffers
```

---

### 2. Create Qdrant Client Implementation

**Create new file: `src/db/qdrant_client.rs`**

```rust
//! Qdrant Client Vector Store Implementation
//! Connects to remote Qdrant server via HTTP/gRPC

use async_trait::async_trait;
use anyhow::{Result, anyhow};
use std::collections::HashMap;
use std::sync::Arc;
use uuid::Uuid;
use chrono::Utc;

use qdrant_client::prelude::*;
use qdrant_client::qdrant::{
    CreateCollection, VectorParams, Distance, PointStruct,
    SearchPoints, Filter, FieldCondition, Match, Condition,
    vectors_config::Config as VectorsConfig,
};

use crate::types::{Document, SearchOptions, SearchResult, SearchFilter};
use crate::services::EncryptionService;
use super::vector_store::{VectorStore, CollectionInfo, CollectionHealth};

/// Qdrant Client Vector Store
pub struct QdrantClientVectorStore {
    client: QdrantClient,
    encryption_service: Arc<EncryptionService>,
}

impl QdrantClientVectorStore {
    /// Create new Qdrant client connection
    ///
    /// # Arguments
    /// * `url` - Qdrant server URL (e.g., "http://localhost:6333")
    /// * `encryption_service` - Service for encrypting/decrypting content
    ///
    /// # Example
    /// ```rust
    /// let store = QdrantClientVectorStore::new(
    ///     "http://localhost:6333",
    ///     encryption_service,
    /// ).await?;
    /// ```
    pub async fn new(
        url: &str,
        encryption_service: Arc<EncryptionService>,
    ) -> Result<Self> {
        let client = QdrantClient::from_url(url).build()?;

        // Test connection
        client.health_check().await
            .map_err(|e| anyhow!("Failed to connect to Qdrant server at {}: {}", url, e))?;

        println!("✅ Connected to Qdrant server at: {}", url);

        Ok(Self {
            client,
            encryption_service,
        })
    }

    /// Create new with API key (for Qdrant Cloud or secured instances)
    pub async fn new_with_api_key(
        url: &str,
        api_key: &str,
        encryption_service: Arc<EncryptionService>,
    ) -> Result<Self> {
        let client = QdrantClient::from_url(url)
            .with_api_key(api_key)
            .build()?;

        client.health_check().await
            .map_err(|e| anyhow!("Failed to connect to Qdrant server: {}", e))?;

        println!("✅ Connected to Qdrant server with API key");

        Ok(Self {
            client,
            encryption_service,
        })
    }
}

#[async_trait]
impl VectorStore for QdrantClientVectorStore {
    fn as_any(&self) -> &dyn std::any::Any {
        self
    }

    async fn initialize(&self) -> Result<()> {
        // Client is already initialized in new()
        // Just verify connection
        self.client.health_check().await?;
        Ok(())
    }

    async fn is_initialized(&self) -> bool {
        self.client.health_check().await.is_ok()
    }

    async fn create_collection(&self, name: &str, dimension: usize) -> Result<()> {
        // Choose distance metric based on dimension
        let distance = if dimension == 1 {
            Distance::Euclid  // For 1D dummy vectors (chat_history)
        } else {
            Distance::Cosine  // For real embeddings (aws_estate)
        };

        let config = CreateCollection {
            collection_name: name.to_string(),
            vectors_config: Some(VectorsConfig::Params(VectorParams {
                size: dimension as u64,
                distance: distance.into(),
                ..Default::default()
            })),
            ..Default::default()
        };

        self.client.create_collection(&config).await?;

        println!("✅ Created collection '{}' with dimension {}", name, dimension);

        Ok(())
    }

    async fn collection_exists(&self, name: &str) -> Result<bool> {
        let collections = self.client.list_collections().await?;
        Ok(collections.collections.iter().any(|c| c.name == name))
    }

    async fn upsert_documents(
        &self,
        collection_name: &str,
        documents: &[Document],
    ) -> Result<()> {
        let mut points = Vec::new();

        for doc in documents {
            // Encrypt content
            let encrypted_content = self.encryption_service
                .encrypt_content(&doc.content)
                .await?;

            // Encrypt metadata
            let metadata_json = serde_json::to_value(&doc.metadata)?;
            let encrypted_metadata_json = self.encryption_service
                .encrypt_metadata(&metadata_json)
                .await?;

            // Build payload (Qdrant's metadata storage)
            let mut payload = serde_json::Map::new();
            payload.insert(
                "content".to_string(),
                serde_json::Value::String(encrypted_content),
            );
            payload.insert(
                "metadata".to_string(),
                encrypted_metadata_json,
            );
            payload.insert(
                "document_id".to_string(),
                serde_json::Value::String(doc.id.clone()),
            );
            payload.insert(
                "created_at".to_string(),
                serde_json::Value::String(doc.created_at.to_rfc3339()),
            );

            // Get embedding or generate dummy
            let vector = if let Some(ref embedding) = doc.embedding {
                embedding.clone()
            } else {
                vec![0.1]  // 1D dummy vector for chat_history
            };

            // Create point
            let point = PointStruct::new(
                Uuid::new_v4().to_string(),  // Unique point ID
                vector,
                payload.into(),
            );

            points.push(point);
        }

        // Upsert points to Qdrant
        self.client
            .upsert_points_blocking(collection_name, points, None)
            .await?;

        Ok(())
    }

    async fn search_documents(
        &self,
        collection_name: &str,
        query_vector: &[f32],
        options: &SearchOptions,
    ) -> Result<Vec<SearchResult>> {
        let limit = options.limit.unwrap_or(10);

        // Build filter if needed
        let filter = if let Some(ref search_filter) = options.filter {
            Some(self.build_qdrant_filter(search_filter)?)
        } else {
            None
        };

        // Perform search
        let search_result = self.client
            .search_points(&SearchPoints {
                collection_name: collection_name.to_string(),
                vector: query_vector.to_vec(),
                limit: limit as u64,
                filter,
                with_payload: Some(true.into()),
                ..Default::default()
            })
            .await?;

        // Decrypt and convert results
        let mut results = Vec::new();
        for point in search_result.result {
            let payload = point.payload;

            // Extract and decrypt content
            let encrypted_content = payload
                .get("content")
                .and_then(|v| v.as_str().map(String::from))
                .ok_or_else(|| anyhow!("Missing content in payload"))?;

            let decrypted_content = self.encryption_service
                .decrypt_content(&encrypted_content)
                .await?;

            // Extract and decrypt metadata
            let encrypted_metadata = payload
                .get("metadata")
                .ok_or_else(|| anyhow!("Missing metadata"))?;

            let decrypted_metadata = self.encryption_service
                .decrypt_metadata(encrypted_metadata)
                .await?;

            results.push(SearchResult {
                id: point.id.to_string(),
                score: point.score,
                content: decrypted_content,
                metadata: decrypted_metadata,
            });
        }

        Ok(results)
    }

    async fn delete_documents(
        &self,
        collection_name: &str,
        ids: &[String],
    ) -> Result<()> {
        use qdrant_client::qdrant::{PointsIdsList, PointsSelector};

        let points_selector = PointsSelector {
            points_selector_one_of: Some(
                qdrant_client::qdrant::points_selector::PointsSelectorOneOf::Points(
                    PointsIdsList {
                        ids: ids.iter().map(|id| id.clone().into()).collect(),
                    },
                ),
            ),
        };

        self.client
            .delete_points(collection_name, &points_selector, None)
            .await?;

        Ok(())
    }

    async fn get_collections_health(&self) -> Result<HashMap<String, CollectionHealth>> {
        let collections = self.client.list_collections().await?;
        let mut health_map = HashMap::new();

        for collection in collections.collections {
            let info = self.client
                .collection_info(&collection.name)
                .await?;

            let result = info.result.ok_or_else(|| anyhow!("No collection info"))?;

            health_map.insert(
                collection.name.clone(),
                CollectionHealth {
                    name: collection.name,
                    status: "green".to_string(),
                    points_count: result.points_count.unwrap_or(0),
                    segments_count: result.segments_count.unwrap_or(0),
                    disk_size: 0,  // Not directly available
                    ram_size: 0,   // Not directly available
                    last_updated: Utc::now(),
                },
            );
        }

        Ok(health_map)
    }
}

impl QdrantClientVectorStore {
    /// Convert SearchFilter to Qdrant Filter
    fn build_qdrant_filter(&self, search_filter: &SearchFilter) -> Result<Filter> {
        let mut conditions = Vec::new();

        // Example: Filter by context_id
        if let Some(context_id) = search_filter.get("context_id").and_then(|v| v.as_str()) {
            conditions.push(Condition {
                condition_one_of: Some(
                    qdrant_client::qdrant::condition::ConditionOneOf::Field(
                        FieldCondition {
                            key: "metadata.context_id".to_string(),
                            r#match: Some(Match {
                                match_value: Some(
                                    qdrant_client::qdrant::r#match::MatchValue::Keyword(
                                        context_id.to_string()
                                    )
                                ),
                            }),
                            ..Default::default()
                        },
                    ),
                ),
            });
        }

        Ok(Filter {
            must: conditions,
            ..Default::default()
        })
    }
}
```

---

### 3. Update `src/db/mod.rs`

```rust
pub mod vector_store;
pub mod embedded_qdrant;
pub mod local_file_store;
pub mod qdrant_client;  // ← Add this

pub use vector_store::{VectorStore, CollectionInfo, CollectionHealth};
pub use embedded_qdrant::EmbeddedQdrantVectorStore;
pub use local_file_store::LocalFileVectorStore;
pub use qdrant_client::QdrantClientVectorStore;  // ← Add this
```

---

### 4. Update `src/lib.rs` - RagModule Initialization

**Add configuration enum:**

```rust
/// Vector store mode configuration
pub enum VectorStoreMode {
    /// Use embedded file-based storage (current)
    Embedded { path: PathBuf },

    /// Connect to remote Qdrant server
    Remote { url: String, api_key: Option<String> },
}

impl Default for VectorStoreMode {
    fn default() -> Self {
        VectorStoreMode::Embedded {
            path: PathBuf::from("./qdrant-data"),
        }
    }
}
```

**Update RagModule initialization:**

```rust
impl RagModule {
    /// Create new RAG module with automatic mode detection
    pub async fn new(base_path: &str) -> Result<Self> {
        // Check environment variable for mode
        let mode = if let Ok(qdrant_url) = std::env::var("QDRANT_URL") {
            let api_key = std::env::var("QDRANT_API_KEY").ok();
            VectorStoreMode::Remote { url: qdrant_url, api_key }
        } else {
            let qdrant_path = PathBuf::from(base_path).join("qdrant-data");
            VectorStoreMode::Embedded { path: qdrant_path }
        };

        Self::new_with_mode(base_path, mode).await
    }

    /// Create new RAG module with explicit mode
    pub async fn new_with_mode(
        base_path: &str,
        mode: VectorStoreMode,
    ) -> Result<Self> {
        let config = Config::load(base_path)?;
        let base_path = PathBuf::from(base_path);

        // Initialize encryption service
        let encryption_service = Arc::new(
            EncryptionService::new(&config.encryption, base_path.to_str().unwrap()).await?
        );

        // Initialize vector store based on mode
        let vector_store: Arc<dyn VectorStore> = match mode {
            VectorStoreMode::Embedded { path } => {
                println!("💾 Using embedded Qdrant (local file storage)");
                println!("   Data path: {}", path.display());

                Arc::new(
                    EmbeddedQdrantVectorStore::new(
                        path,
                        Arc::clone(&encryption_service),
                    ).await?
                )
            }
            VectorStoreMode::Remote { url, api_key } => {
                println!("🌐 Connecting to remote Qdrant server");
                println!("   URL: {}", url);

                let client_store = if let Some(key) = api_key {
                    QdrantClientVectorStore::new_with_api_key(
                        &url,
                        &key,
                        Arc::clone(&encryption_service),
                    ).await?
                } else {
                    QdrantClientVectorStore::new(
                        &url,
                        Arc::clone(&encryption_service),
                    ).await?
                };

                Arc::new(client_store)
            }
        };

        // Rest of initialization...
        // (embedding_service, collection_manager, etc.)

        Ok(Self {
            vector_store,
            encryption_service,
            // ... other fields
        })
    }
}
```

---

## API Methods Mapping

### Collection Operations

| Your Method | Embedded Implementation | Qdrant Server API | HTTP Endpoint |
|-------------|------------------------|-------------------|---------------|
| `create_collection(name, dim)` | Create JSON files | `CreateCollection` with `VectorParams` | `PUT /collections/{name}` |
| `collection_exists(name)` | Check if file exists | List collections and find | `GET /collections` |
| `delete_collection(name)` | Delete JSON files | Delete collection | `DELETE /collections/{name}` |

**Example API Call:**

```rust
// Embedded (current)
async fn create_collection(&self, name: &str, dimension: usize) -> Result<()> {
    let file_path = self.base_path.join(format!("{}-documents.json", name));
    fs::write(&file_path, "[]")?;
    Ok(())
}

// Server (new)
async fn create_collection(&self, name: &str, dimension: usize) -> Result<()> {
    self.client.create_collection(&CreateCollection {
        collection_name: name.to_string(),
        vectors_config: Some(VectorParams {
            size: dimension as u64,
            distance: Distance::Cosine.into(),
        }.into()),
    }).await?;
    Ok(())
}
```

---

### Document Operations

| Your Method | Embedded Implementation | Qdrant Server API | HTTP Endpoint |
|-------------|------------------------|-------------------|---------------|
| `upsert_documents(docs)` | Append to JSON file | `upsert_points()` | `PUT /collections/{name}/points` |
| `search_documents(query)` | Loop + cosine similarity | `search_points()` with HNSW | `POST /collections/{name}/points/search` |
| `delete_documents(ids)` | Filter and rewrite JSON | `delete_points()` | `POST /collections/{name}/points/delete` |
| `get_document(id)` | Parse JSON and find | `retrieve_points()` | `POST /collections/{name}/points` |

**Example API Call:**

```rust
// Search with filters
let results = client.search_points(&SearchPoints {
    collection_name: "chat_history".to_string(),
    vector: query_embedding,
    limit: 10,
    filter: Some(Filter {
        must: vec![
            Condition {
                condition_one_of: Some(
                    FieldCondition {
                        key: "context_id".to_string(),
                        r#match: Some(Match {
                            match_value: Some(
                                MatchValue::Keyword("test123".to_string())
                            ),
                        }),
                    }.into()
                ),
            },
        ],
        ..Default::default()
    }),
    with_payload: Some(true.into()),
}).await?;
```

---

### Health & Monitoring

| Your Method | Embedded Implementation | Qdrant Server API | HTTP Endpoint |
|-------------|------------------------|-------------------|---------------|
| `is_initialized()` | Check files exist | `health_check()` | `GET /healthz` |
| `get_collections_health()` | Read file sizes | `collection_info()` | `GET /collections/{name}` |
| `get_stats()` | Count JSON entries | Built-in metrics | `GET /metrics` |

---

## Data Migration

When switching from embedded to server mode, you need to migrate existing data.

### Migration Strategy

#### Option 1: Full Migration Script

**Create `examples/migrate_to_qdrant_server.rs`:**

```rust
//! Migrate data from embedded storage to Qdrant server
//!
//! Usage:
//! ```bash
//! export QDRANT_URL="http://localhost:6333"
//! cargo run --example migrate_to_qdrant_server
//! ```

use anyhow::Result;
use rag_module::{RagModule, VectorStoreMode};
use std::path::PathBuf;

#[tokio::main]
async fn main() -> Result<()> {
    println!("🔄 Migrating data from embedded to Qdrant server...\n");

    // 1. Read from embedded storage
    println!("📖 Step 1: Loading data from embedded storage...");
    let embedded_rag = RagModule::new_with_mode(
        "./data",
        VectorStoreMode::Embedded {
            path: PathBuf::from("./data/qdrant-data"),
        },
    ).await?;

    let user_id = "your-user-id";  // Replace with actual user ID
    let encrypted_docs = embedded_rag.get_encrypted_chat_documents(user_id).await?;
    println!("   ✅ Loaded {} documents", encrypted_docs.len());

    // 2. Connect to Qdrant server
    println!("\n🌐 Step 2: Connecting to Qdrant server...");
    let qdrant_url = std::env::var("QDRANT_URL")
        .expect("QDRANT_URL environment variable not set");

    let server_rag = RagModule::new_with_mode(
        "./data",
        VectorStoreMode::Remote {
            url: qdrant_url.clone(),
            api_key: None,
        },
    ).await?;
    println!("   ✅ Connected to {}", qdrant_url);

    // 3. Create collections on server
    println!("\n🏗️  Step 3: Creating collections...");
    server_rag.initialize().await?;
    println!("   ✅ Collections created");

    // 4. Migrate documents
    println!("\n📤 Step 4: Migrating documents...");

    // Get decrypted data from embedded
    let decrypted_data = embedded_rag.get_decrypted_chat_history(user_id).await?;

    // Re-save to server (this will encrypt and store)
    let contexts = decrypted_data.get("contexts")
        .and_then(|v| v.as_object())
        .ok_or_else(|| anyhow::anyhow!("No contexts found"))?;

    let mut total_messages = 0;
    for (context_id, context_data) in contexts {
        let conversations = context_data.get("conversations")
            .and_then(|v| v.as_array())
            .ok_or_else(|| anyhow::anyhow!("No conversations"))?;

        println!("   Migrating context: {}", context_id);

        for conv in conversations {
            // Extract prompt and response
            let prompt = conv.get("prompt").ok_or_else(|| anyhow::anyhow!("No prompt"))?;
            let response = conv.get("response").ok_or_else(|| anyhow::anyhow!("No response"))?;

            // Get content
            let prompt_content = prompt.get("content")
                .and_then(|v| v.as_str())
                .ok_or_else(|| anyhow::anyhow!("No prompt content"))?;
            let response_content = response.get("content")
                .and_then(|v| v.as_str())
                .ok_or_else(|| anyhow::anyhow!("No response content"))?;

            // Save to server
            // You'll need to implement this based on your actual data structure
            total_messages += 2;
        }
    }

    println!("   ✅ Migrated {} messages", total_messages);

    // 5. Verify migration
    println!("\n✅ Step 5: Verifying migration...");
    let server_data = server_rag.get_decrypted_chat_history(user_id).await?;
    let server_contexts = server_data.get("contexts")
        .and_then(|v| v.as_object())
        .ok_or_else(|| anyhow::anyhow!("No contexts in server"))?;

    println!("   Embedded contexts: {}", contexts.len());
    println!("   Server contexts: {}", server_contexts.len());

    if contexts.len() == server_contexts.len() {
        println!("\n🎉 Migration completed successfully!");
    } else {
        println!("\n⚠️  Warning: Context count mismatch!");
    }

    Ok(())
}
```

**Run migration:**

```bash
# Start Qdrant server first
docker compose up -d

# Set environment variable
export QDRANT_URL="http://localhost:6333"

# Run migration
cargo run --example migrate_to_qdrant_server
```

---

#### Option 2: Gradual Migration (Dual Mode)

Support both modes simultaneously during transition:

```rust
pub struct RagModule {
    // Support both stores
    embedded_store: Option<Arc<EmbeddedQdrantVectorStore>>,
    server_store: Option<Arc<QdrantClientVectorStore>>,

    // Which one is active
    active_mode: VectorStoreMode,
}

impl RagModule {
    /// Read from embedded, write to server
    pub async fn migrate_on_write(&self, doc: &Document) -> Result<()> {
        // Write to server
        if let Some(ref server) = self.server_store {
            server.upsert_documents("chat_history", &[doc.clone()]).await?;
        }

        // Also write to embedded (for backup)
        if let Some(ref embedded) = self.embedded_store {
            embedded.upsert_documents("chat_history", &[doc.clone()]).await?;
        }

        Ok(())
    }
}
```

---

## APIs to Expose

Your backend service should expose these REST APIs to the frontend:

### 1. Chat History APIs

```
GET /api/chat/history/:user_id
```

**Description**: Get all chat history for a user

**Response:**
```json
{
  "user_id": "cudewvsbxakj1n",
  "contexts": {
    "context_abc123": {
      "context_id": "context_abc123",
      "session_id": "session_xyz",
      "chat_title": "Discussion about REST vs GraphQL",
      "conversations": [
        {
          "prompt": {
            "role": "user",
            "content": "What is REST?",
            "timestamp": "2025-11-03T10:00:00Z",
            "document_id": "doc_123"
          },
          "response": {
            "role": "assistant",
            "content": "REST is...",
            "timestamp": "2025-11-03T10:00:05Z",
            "document_id": "doc_124"
          }
        }
      ]
    }
  }
}
```

---

```
GET /api/chat/history/:user_id/:context_id
```

**Description**: Get specific conversation by context ID

**Response:**
```json
{
  "context_id": "context_abc123",
  "chat_title": "Discussion about REST vs GraphQL",
  "messages": [
    {
      "role": "user",
      "content": "What is REST?",
      "timestamp": "2025-11-03T10:00:00Z"
    },
    {
      "role": "assistant",
      "content": "REST is...",
      "timestamp": "2025-11-03T10:00:05Z"
    }
  ]
}
```

---

```
POST /api/chat/messages
```

**Description**: Add a new message to conversation

**Request Body:**
```json
{
  "user_id": "cudewvsbxakj1n",
  "session_id": "session_xyz",
  "context_id": "context_abc123",
  "role": "user",
  "content": "How does GraphQL work?"
}
```

**Response:**
```json
{
  "success": true,
  "message_id": "msg_789",
  "timestamp": "2025-11-03T10:05:00Z"
}
```

---

```
POST /api/chat/search
```

**Description**: Search chat history

**Request Body:**
```json
{
  "user_id": "cudewvsbxakj1n",
  "query": "GraphQL caching",
  "context_id": "context_abc123",
  "limit": 10
}
```

**Response:**
```json
{
  "results": [
    {
      "score": 0.95,
      "content": "GraphQL caching is complex because...",
      "context_id": "context_abc123",
      "timestamp": "2025-11-03T10:00:00Z"
    }
  ]
}
```

---

### 2. AWS Estate APIs

```
POST /api/estate/search
```

**Description**: Search AWS resources

**Request Body:**
```json
{
  "query": "EC2 instances in us-east-1",
  "filters": {
    "resource_type": "ec2",
    "region": "us-east-1"
  },
  "limit": 20
}
```

**Response:**
```json
{
  "results": [
    {
      "resource_id": "i-1234567890",
      "resource_type": "ec2",
      "score": 0.92,
      "metadata": {
        "instance_type": "t3.medium",
        "region": "us-east-1",
        "state": "running"
      }
    }
  ],
  "total": 45
}
```

---

```
GET /api/estate/resources/:resource_id
```

**Description**: Get specific resource details

**Response:**
```json
{
  "resource_id": "i-1234567890",
  "resource_type": "ec2",
  "content": "EC2 instance running web application",
  "metadata": {
    "instance_type": "t3.medium",
    "region": "us-east-1",
    "tags": {
      "Environment": "production",
      "Application": "web-app"
    }
  },
  "created_at": "2025-11-01T00:00:00Z"
}
```

---

```
POST /api/estate/resources
```

**Description**: Add or update AWS resource

**Request Body:**
```json
{
  "resource_id": "i-1234567890",
  "resource_type": "ec2",
  "content": "EC2 instance for web application",
  "metadata": {
    "instance_type": "t3.medium",
    "region": "us-east-1"
  }
}
```

**Response:**
```json
{
  "success": true,
  "resource_id": "i-1234567890",
  "indexed": true
}
```

---

### 3. Health & Monitoring APIs

```
GET /api/health
```

**Description**: Check backend health status

**Response:**
```json
{
  "status": "healthy",
  "timestamp": "2025-11-03T12:00:00Z",
  "services": {
    "qdrant": {
      "connected": true,
      "mode": "server",
      "url": "http://ec2-instance:6333",
      "version": "1.7.4"
    },
    "encryption": {
      "status": "ok",
      "keychain": "available"
    }
  },
  "collections": {
    "chat_history": {
      "documents": 1523,
      "status": "green"
    },
    "aws_estate": {
      "documents": 8947,
      "status": "green"
    }
  }
}
```

---

```
GET /api/stats
```

**Description**: Get usage statistics

**Response:**
```json
{
  "user_id": "cudewvsbxakj1n",
  "chat_history": {
    "total_conversations": 45,
    "total_messages": 892,
    "storage_mb": 12.5,
    "last_activity": "2025-11-03T11:30:00Z"
  },
  "aws_estate": {
    "total_resources": 8947,
    "resource_types": {
      "ec2": 1234,
      "s3": 567,
      "lambda": 890
    },
    "storage_mb": 156.8,
    "last_indexed": "2025-11-03T10:00:00Z"
  }
}
```

---

```
GET /api/collections/:name/health
```

**Description**: Get detailed collection health

**Response:**
```json
{
  "collection": "chat_history",
  "status": "green",
  "points_count": 1523,
  "segments_count": 4,
  "disk_size_mb": 12.5,
  "ram_size_mb": 8.2,
  "last_updated": "2025-11-03T11:45:00Z",
  "optimization_status": "optimized"
}
```

---

## Performance Comparison

### Benchmark: Insert Operation

| Metric | Embedded (Current) | Server (New) | Notes |
|--------|-------------------|--------------|-------|
| Insert 1 document | ~1ms | ~5-10ms | Network latency |
| Insert 100 documents | ~50ms | ~30ms | Batch optimization |
| Insert 1000 documents | ~500ms | ~150ms | Server parallelization |

### Benchmark: Search Operation

| Metric | Embedded (Current) | Server (New) | Notes |
|--------|-------------------|--------------|-------|
| Search 1k docs | ~50ms | ~5ms | HNSW vs linear |
| Search 10k docs | ~500ms | ~8ms | HNSW advantage |
| Search 100k docs | ~5s | ~12ms | HNSW scales better |
| Search 1M docs | ~50s | ~15ms | Only server feasible |

### Memory Usage

| Dataset Size | Embedded (Current) | Server (New) | Notes |
|--------------|-------------------|--------------|-------|
| 1k documents | ~10MB | ~50MB | Server overhead |
| 10k documents | ~100MB | ~150MB | Server more efficient |
| 100k documents | ~1GB | ~500MB | HNSW compression |
| 1M documents | ~10GB | ~2GB | Only server practical |

### Concurrent Requests

| Scenario | Embedded (Current) | Server (New) |
|----------|-------------------|--------------|
| 1 client | ✅ Fast | ✅ Fast |
| 5 concurrent clients | ❌ Sequential | ✅ Parallel |
| 10 concurrent clients | ❌ Slow | ✅ Fast |
| 50 concurrent clients | ❌ Not supported | ✅ Handles well |

---

## Breaking Changes

### What Breaks

1. **❌ File Paths**
   - **Before**: `./qdrant-data/chat_history-documents.json`
   - **After**: No direct file access
   - **Impact**: Can't manually inspect/edit JSON files

2. **❌ Offline Mode**
   - **Before**: Works without network
   - **After**: Requires connection to Qdrant server
   - **Impact**: Need server running for development

3. **❌ Direct File I/O**
   - **Before**: Read/write JSON directly
   - **After**: All operations via Qdrant API
   - **Impact**: Can't use standard file tools

4. **❌ Data Format**
   - **Before**: JSON files with specific structure
   - **After**: Qdrant native binary format
   - **Impact**: Need migration script

5. **❌ Deployment Complexity**
   - **Before**: Single binary deployment
   - **After**: Need to manage Qdrant server
   - **Impact**: More DevOps work

---

### What Stays the Same

1. **✅ Application API**
   - All your methods: `save_chat_history()`, `search_chat_history()`, etc.
   - Same function signatures
   - **No changes to your application logic**

2. **✅ Encryption**
   - Still encrypted before storage
   - Same encryption keys
   - **Security model unchanged**

3. **✅ Data Structure**
   - Same document format
   - Same metadata structure
   - **Business logic unchanged**

4. **✅ User Experience**
   - Same functionality
   - Possibly faster searches
   - **End-user sees no difference**

---

## Rollout Strategy

### Phase 1: Add Server Support (No Breaking Changes)

**Goal**: Support both modes without breaking existing installations

**Duration**: 2-3 weeks

**Tasks**:
1. ✅ Add `qdrant-client` dependency
2. ✅ Create `QdrantClientVectorStore` implementation
3. ✅ Add `VectorStoreMode` enum
4. ✅ Update `RagModule::new()` to support both modes
5. ✅ Environment variable detection (`QDRANT_URL`)
6. ✅ Add tests for server mode
7. ✅ Update documentation

**Result**: Application supports both modes, embedded is still default

**Usage**:
```bash
# Embedded mode (current, still works)
cargo run

# Server mode (new, opt-in)
export QDRANT_URL="http://localhost:6333"
cargo run
```

---

### Phase 2: Migration Tools & Testing

**Goal**: Provide tools and confidence for migration

**Duration**: 1-2 weeks

**Tasks**:
1. ✅ Create migration script (`migrate_to_qdrant_server.rs`)
2. ✅ Add data validation tools
3. ✅ Performance benchmarks
4. ✅ Integration tests with real Qdrant server
5. ✅ Document migration process
6. ✅ Internal testing with real data

**Result**: Users can safely migrate data

**Usage**:
```bash
# Test migration (dry-run)
cargo run --example migrate_to_qdrant_server -- --dry-run

# Actual migration
cargo run --example migrate_to_qdrant_server
```

---

### Phase 3: Server as Default (Recommended)

**Goal**: Make server mode the recommended option

**Duration**: 1 week

**Tasks**:
1. ✅ Update README to recommend server mode
2. ✅ Add Docker Compose quickstart
3. ✅ Update examples to use server mode
4. ✅ Keep embedded as fallback
5. ✅ Add "deprecated" warning to embedded mode

**Result**: New users start with server, existing users can stay on embedded

---

### Phase 4: Deprecation (Optional)

**Goal**: Eventually remove embedded mode

**Duration**: 6+ months after Phase 3

**Tasks**:
1. ⚠️ Add deprecation warnings to embedded mode
2. ⚠️ Set end-of-life date
3. ⚠️ Help users migrate
4. ⚠️ Remove embedded code in next major version

---

## Decision Matrix

### When to Use Embedded Mode

✅ **Use Embedded If:**
- Single user application
- Development/testing environment
- No network available (offline work)
- Simple use case (<10k documents)
- Don't want to manage server
- Quick prototyping
- Educational purposes

---

### When to Use Server Mode

✅ **Use Server If:**
- Production deployment
- Multiple users/clients
- Large datasets (>100k documents)
- Need concurrent access
- Want better search performance
- Need monitoring/metrics
- Want web dashboard
- Scaling is important

---

## Quick Comparison Table

| Feature | Embedded | Server |
|---------|----------|--------|
| **Setup Complexity** | ⭐ Easy | ⭐⭐⭐ Complex |
| **Performance (<10k docs)** | ⭐⭐⭐ Fast | ⭐⭐ Good |
| **Performance (>100k docs)** | ⭐ Slow | ⭐⭐⭐ Fast |
| **Scalability** | ⭐ Limited | ⭐⭐⭐ Excellent |
| **Concurrent Users** | ❌ No | ✅ Yes |
| **Monitoring** | ⭐ Basic | ⭐⭐⭐ Advanced |
| **Deployment** | ⭐⭐⭐ Simple | ⭐⭐ Moderate |
| **Cost** | ✅ Free | 💰 EC2 costs |
| **Maintenance** | ⭐⭐⭐ Low | ⭐⭐ Medium |
| **Production Ready** | ⚠️ Dev/Small | ✅ Yes |

---

## Example: Running Both Modes

### Development Setup

```bash
# Use embedded for local development
cargo run

# Data stored in ./qdrant-data/ (easy to inspect)
```

### Staging/Testing Setup

```bash
# Use server for staging
export QDRANT_URL="http://staging-qdrant:6333"
cargo run

# Data on staging server
```

### Production Setup

```bash
# Use server with API key
export QDRANT_URL="https://prod-qdrant.example.com:6333"
export QDRANT_API_KEY="your-secret-key"
cargo run

# Data on production server with auth
```

---

## FAQ

### Q: Can I run both modes simultaneously?

**A**: Yes! You can support both during transition. Read from embedded, write to server (dual-write pattern).

### Q: What happens to my existing data?

**A**: Your embedded data stays in `./qdrant-data/`. Use the migration script to copy it to the server.

### Q: Can I switch back to embedded after migrating?

**A**: Yes, but you'll need to export data from server and recreate the JSON files.

### Q: Do I need to change my frontend code?

**A**: No! Your REST APIs stay the same. Only the backend storage changes.

### Q: What if the Qdrant server goes down?

**A**: Your application will fail to connect. Consider implementing a circuit breaker pattern or fallback to embedded mode.

### Q: Can I use Qdrant Cloud instead of EC2?

**A**: Yes! Just use the Qdrant Cloud URL and API key. Same client code works.

---

## Next Steps

1. **Add Qdrant client dependency** to `Cargo.toml`
2. **Create `qdrant_client.rs`** implementation
3. **Update `RagModule`** to support both modes
4. **Test with local Qdrant server** (Docker)
5. **Create migration script**
6. **Deploy Qdrant to EC2**
7. **Migrate production data**

---

## Resources

- **Qdrant Documentation**: https://qdrant.tech/documentation/
- **Qdrant Rust Client**: https://docs.rs/qdrant-client/
- **Docker Setup Guide**: See `docs/QDRANT_EC2_DEPLOYMENT.md`
- **Migration Script**: See `examples/migrate_to_qdrant_server.rs`

---

## Support

For questions or issues:
- Check Qdrant docs: https://qdrant.tech/documentation/
- Open an issue in the repo
- Contact the backend team

---

**Document Version**: 1.0
**Last Updated**: 2025-11-03
**Author**: RAG Module Team
