# Self-Hosted Qdrant Cluster Deployment on AWS

## Table of Contents
- [Overview](#overview)
- [Deployment Options Comparison](#deployment-options-comparison)
- [Option 1: EC2 with Docker (Simplest)](#option-1-ec2-with-docker-simplest)
- [Option 2: ECS Fargate (Managed Containers)](#option-2-ecs-fargate-managed-containers)
- [Option 3: EKS Kubernetes (Enterprise)](#option-3-eks-kubernetes-enterprise)
- [Option 4: EC2 Auto Scaling Group](#option-4-ec2-auto-scaling-group)
- [Networking and Security](#networking-and-security)
- [Storage Configuration](#storage-configuration)
- [Load Balancing](#load-balancing)
- [Monitoring and Logging](#monitoring-and-logging)
- [Backup and Disaster Recovery](#backup-and-disaster-recovery)
- [Cost Optimization](#cost-optimization)
- [Connecting Your RAG Module](#connecting-your-rag-module)
- [Troubleshooting](#troubleshooting)

---

## Overview

This guide covers **4 ways** to deploy a self-hosted Qdrant cluster on AWS, from simplest to most enterprise-grade.

### Why Self-Host on AWS?

✅ **Full control** over infrastructure
✅ **Data sovereignty** - keep data in your AWS account
✅ **Custom networking** - integrate with existing VPCs
✅ **Cost optimization** - use reserved instances, spot instances
✅ **Compliance** - meet specific regulatory requirements

### Architecture Overview

```
                    Internet Gateway
                           │
                    ┌──────▼──────┐
                    │   Route 53  │ (DNS)
                    └──────┬──────┘
                           │
                  ┌────────▼────────┐
                  │ Application LB  │ (Port 6334)
                  └────────┬────────┘
                           │
        ┌──────────────────┼──────────────────┐
        │                  │                  │
   ┌────▼────┐       ┌────▼────┐       ┌────▼────┐
   │ Qdrant  │       │ Qdrant  │       │ Qdrant  │
   │ Node 1  │←─────→│ Node 2  │←─────→│ Node 3  │
   │ (EC2/ECS)│      │ (EC2/ECS)│      │ (EC2/ECS)│
   └────┬────┘       └────┬────┘       └────┬────┘
        │                  │                  │
   ┌────▼────┐       ┌────▼────┐       ┌────▼────┐
   │  EBS/   │       │  EBS/   │       │  EBS/   │
   │   EFS   │       │   EFS   │       │   EFS   │
   └─────────┘       └─────────┘       └─────────┘
```

---

## Deployment Options Comparison

| Feature | EC2 + Docker | ECS Fargate | EKS Kubernetes | EC2 ASG |
|---------|--------------|-------------|----------------|---------|
| **Complexity** | Low | Medium | High | Medium |
| **Setup Time** | 1-2 hours | 2-4 hours | 4-8 hours | 2-3 hours |
| **Management** | Manual | Semi-managed | Manual | Semi-automated |
| **Scaling** | Manual | Auto-scaling | Auto-scaling | Auto-scaling |
| **Cost (3 nodes)** | $200-300/mo | $250-400/mo | $400-600/mo | $200-350/mo |
| **Best For** | Quick start, Dev/Test | Production, Low ops | Enterprise, Large scale | Production, Cost-effective |
| **AWS Services** | EC2, EBS, ALB | ECS, EFS, ALB | EKS, EBS, ALB | EC2, ASG, ALB |

---

## Option 1: EC2 with Docker (Simplest)

**⏱️ Setup Time:** 1-2 hours
**💰 Cost:** ~$200-300/month
**🎯 Best For:** Quick start, development, small production

### Architecture

```
3 EC2 Instances (t3.large) + EBS volumes + Application Load Balancer
```

### Step 1: Launch EC2 Instances

#### 1.1 Create Security Group

```bash
# Create security group
aws ec2 create-security-group \
  --group-name qdrant-cluster-sg \
  --description "Security group for Qdrant cluster" \
  --vpc-id vpc-xxxxx

# Get the security group ID
SG_ID=sg-xxxxx

# Allow HTTP (port 6333) from ALB
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID \
  --protocol tcp \
  --port 6333 \
  --source-group <ALB-SG-ID>

# Allow gRPC (port 6334) from ALB
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID \
  --protocol tcp \
  --port 6334 \
  --source-group <ALB-SG-ID>

# Allow P2P (port 6335) between cluster nodes
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID \
  --protocol tcp \
  --port 6335 \
  --source-group $SG_ID

# Allow SSH (port 22) from your IP
aws ec2 authorize-security-group-ingress \
  --group-id $SG_ID \
  --protocol tcp \
  --port 22 \
  --cidr <YOUR-IP>/32
```

#### 1.2 Launch EC2 Instances

```bash
# Launch 3 instances
for i in 1 2 3; do
  aws ec2 run-instances \
    --image-id ami-0c55b159cbfafe1f0 \
    --instance-type t3.large \
    --key-name your-key-pair \
    --security-group-ids $SG_ID \
    --subnet-id subnet-xxxxx \
    --block-device-mappings '[
      {
        "DeviceName": "/dev/sda1",
        "Ebs": {
          "VolumeSize": 100,
          "VolumeType": "gp3",
          "Iops": 3000,
          "DeleteOnTermination": false
        }
      }
    ]' \
    --tag-specifications "ResourceType=instance,Tags=[
      {Key=Name,Value=qdrant-node-$i},
      {Key=Role,Value=qdrant-cluster}
    ]"
done
```

**Instance Specifications:**
- **Instance Type:** `t3.large` (2 vCPU, 8GB RAM)
- **Storage:** 100GB gp3 EBS (3000 IOPS)
- **Cost:** ~$65/month per instance = $195/month for 3 nodes

#### 1.3 Note Instance IPs

```bash
# Get private IPs
aws ec2 describe-instances \
  --filters "Name=tag:Role,Values=qdrant-cluster" \
  --query 'Reservations[].Instances[].[InstanceId,PrivateIpAddress,Tags[?Key==`Name`].Value|[0]]' \
  --output table

# Example output:
# Node 1: 10.0.1.10
# Node 2: 10.0.1.11
# Node 3: 10.0.1.12
```

### Step 2: Install Docker on All Nodes

Connect to each instance and run:

```bash
# SSH to each instance
ssh -i your-key.pem ec2-user@<instance-ip>

# Update system
sudo yum update -y

# Install Docker
sudo yum install docker -y

# Start Docker
sudo systemctl start docker
sudo systemctl enable docker

# Add ec2-user to docker group
sudo usermod -a -G docker ec2-user

# Log out and back in for group changes
exit
```

### Step 3: Deploy Qdrant Cluster

#### 3.1 Create Qdrant Configuration

On **Node 1** (10.0.1.10):

```bash
# Create config directory
mkdir -p ~/qdrant-config

# Create config file
cat > ~/qdrant-config/config.yaml <<'EOF'
cluster:
  enabled: true
  p2p:
    port: 6335
  consensus:
    tick_period_ms: 100

service:
  host: "0.0.0.0"
  http_port: 6333
  grpc_port: 6334

storage:
  storage_path: "/qdrant/storage"

log_level: INFO
EOF
```

Copy this config to Node 2 and Node 3:

```bash
# From your local machine
scp -i your-key.pem ~/qdrant-config/config.yaml ec2-user@10.0.1.11:~/qdrant-config/
scp -i your-key.pem ~/qdrant-config/config.yaml ec2-user@10.0.1.12:~/qdrant-config/
```

#### 3.2 Start Qdrant on Node 1 (Bootstrap Node)

```bash
ssh -i your-key.pem ec2-user@10.0.1.10

docker run -d \
  --name qdrant-node1 \
  --restart unless-stopped \
  -p 6333:6333 \
  -p 6334:6334 \
  -p 6335:6335 \
  -v ~/qdrant-config:/qdrant/config \
  -v ~/qdrant-storage:/qdrant/storage \
  qdrant/qdrant:latest

# Check logs
docker logs -f qdrant-node1
```

#### 3.3 Start Qdrant on Node 2 and Join Cluster

```bash
ssh -i your-key.pem ec2-user@10.0.1.11

docker run -d \
  --name qdrant-node2 \
  --restart unless-stopped \
  -p 6333:6333 \
  -p 6334:6334 \
  -p 6335:6335 \
  -v ~/qdrant-config:/qdrant/config \
  -v ~/qdrant-storage:/qdrant/storage \
  -e QDRANT__CLUSTER__ENABLED=true \
  -e QDRANT__CLUSTER__P2P__PORT=6335 \
  qdrant/qdrant:latest

# Wait 10 seconds, then join the cluster
sleep 10

# Get the peer ID from Node 1
NODE1_PEER_ID=$(curl -s http://10.0.1.10:6333/cluster | jq -r '.result.peer_id')

# Add Node 2 to cluster
curl -X POST "http://10.0.1.10:6333/cluster/peer" \
  -H "Content-Type: application/json" \
  -d "{\"uri\": \"http://10.0.1.11:6335\", \"peer_id\": null}"
```

#### 3.4 Start Qdrant on Node 3 and Join Cluster

```bash
ssh -i your-key.pem ec2-user@10.0.1.12

docker run -d \
  --name qdrant-node3 \
  --restart unless-stopped \
  -p 6333:6333 \
  -p 6334:6334 \
  -p 6335:6335 \
  -v ~/qdrant-config:/qdrant/config \
  -v ~/qdrant-storage:/qdrant/storage \
  -e QDRANT__CLUSTER__ENABLED=true \
  -e QDRANT__CLUSTER__P2P__PORT=6335 \
  qdrant/qdrant:latest

sleep 10

# Add Node 3 to cluster
curl -X POST "http://10.0.1.10:6333/cluster/peer" \
  -H "Content-Type: application/json" \
  -d "{\"uri\": \"http://10.0.1.12:6335\", \"peer_id\": null}"
```

#### 3.5 Verify Cluster

```bash
# Check cluster status
curl http://10.0.1.10:6333/cluster | jq

# Expected output:
{
  "result": {
    "status": "enabled",
    "peer_id": "xxxxx",
    "peers": {
      "xxxxx": {
        "uri": "http://10.0.1.10:6335"
      },
      "yyyyy": {
        "uri": "http://10.0.1.11:6335"
      },
      "zzzzz": {
        "uri": "http://10.0.1.12:6335"
      }
    }
  },
  "status": "ok",
  "time": 0.001
}
```

### Step 4: Create Application Load Balancer

#### 4.1 Create Target Group

```bash
# Create target group for gRPC (port 6334)
aws elbv2 create-target-group \
  --name qdrant-cluster-tg \
  --protocol TCP \
  --port 6334 \
  --vpc-id vpc-xxxxx \
  --health-check-protocol HTTP \
  --health-check-path /health \
  --health-check-port 6333 \
  --health-check-interval-seconds 30 \
  --health-check-timeout-seconds 5 \
  --healthy-threshold-count 2 \
  --unhealthy-threshold-count 2

# Get target group ARN
TG_ARN=$(aws elbv2 describe-target-groups \
  --names qdrant-cluster-tg \
  --query 'TargetGroups[0].TargetGroupArn' \
  --output text)

# Register instances
aws elbv2 register-targets \
  --target-group-arn $TG_ARN \
  --targets \
    Id=<node1-instance-id> \
    Id=<node2-instance-id> \
    Id=<node3-instance-id>
```

#### 4.2 Create Network Load Balancer

```bash
# Create NLB
aws elbv2 create-load-balancer \
  --name qdrant-cluster-nlb \
  --type network \
  --scheme internet-facing \
  --subnets subnet-xxxxx subnet-yyyyy subnet-zzzzz \
  --tags Key=Name,Value=qdrant-cluster-nlb

# Get NLB ARN
NLB_ARN=$(aws elbv2 describe-load-balancers \
  --names qdrant-cluster-nlb \
  --query 'LoadBalancers[0].LoadBalancerArn' \
  --output text)

# Create listener for gRPC (6334)
aws elbv2 create-listener \
  --load-balancer-arn $NLB_ARN \
  --protocol TCP \
  --port 6334 \
  --default-actions Type=forward,TargetGroupArn=$TG_ARN

# Get NLB DNS name
NLB_DNS=$(aws elbv2 describe-load-balancers \
  --names qdrant-cluster-nlb \
  --query 'LoadBalancers[0].DNSName' \
  --output text)

echo "Your Qdrant cluster is available at: $NLB_DNS:6334"
```

#### 4.3 (Optional) Create Route 53 DNS Record

```bash
# Create hosted zone record
aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890ABC \
  --change-batch '{
    "Changes": [{
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "qdrant.yourdomain.com",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{"Value": "'$NLB_DNS'"}]
      }
    }]
  }'
```

### Step 5: Test Your Cluster

```bash
# Test health endpoint
curl http://$NLB_DNS:6333/health

# Test cluster status
curl http://$NLB_DNS:6333/cluster

# Create a test collection
curl -X PUT "http://$NLB_DNS:6333/collections/test" \
  -H "Content-Type: application/json" \
  -d '{
    "vectors": {
      "size": 4,
      "distance": "Cosine"
    }
  }'

# Insert test point
curl -X PUT "http://$NLB_DNS:6333/collections/test/points" \
  -H "Content-Type: application/json" \
  -d '{
    "points": [
      {
        "id": 1,
        "vector": [0.1, 0.2, 0.3, 0.4],
        "payload": {"test": "data"}
      }
    ]
  }'

# Search
curl -X POST "http://$NLB_DNS:6333/collections/test/points/search" \
  -H "Content-Type: application/json" \
  -d '{
    "vector": [0.1, 0.2, 0.3, 0.4],
    "limit": 5
  }'
```

### Step 6: Connect Your Rust Application

Update your configuration:

```yaml
# config/production.yaml
vector_store:
  backend: "qdrant-server"
  connection:
    url: "http://qdrant.yourdomain.com:6334"  # Or NLB DNS
    api_key: null  # No API key needed (use security groups)
    timeout_secs: 60
```

Or use environment variables:

```bash
export QDRANT_URL="http://qdrant.yourdomain.com:6334"
```

Your Rust code works without changes:

```rust
// Automatically uses QDRANT_URL from env or config
let rag = RagModule::new("./data").await?;
rag.set_user_context("user123").await?;

// Now using AWS-hosted cluster!
let results = rag.search("aws_estate", "find EC2 instances", None).await?;
```

---

## Option 2: ECS Fargate (Managed Containers)

**⏱️ Setup Time:** 2-4 hours
**💰 Cost:** ~$250-400/month
**🎯 Best For:** Production with less operational overhead

### Architecture

```
ECS Cluster → Fargate Tasks (3) → EFS for persistent storage → ALB
```

### Benefits
- ✅ No EC2 instances to manage
- ✅ Automatic scaling
- ✅ Pay only for what you use
- ✅ Integrated with AWS services

### Step 1: Create EFS File System

```bash
# Create EFS
aws efs create-file-system \
  --performance-mode generalPurpose \
  --throughput-mode bursting \
  --encrypted \
  --tags Key=Name,Value=qdrant-cluster-efs

# Get EFS ID
EFS_ID=$(aws efs describe-file-systems \
  --query 'FileSystems[?Name==`qdrant-cluster-efs`].FileSystemId' \
  --output text)

# Create mount targets in each subnet
for subnet in subnet-xxxxx subnet-yyyyy subnet-zzzzz; do
  aws efs create-mount-target \
    --file-system-id $EFS_ID \
    --subnet-id $subnet \
    --security-groups sg-xxxxx
done
```

### Step 2: Create ECS Cluster

```bash
# Create cluster
aws ecs create-cluster \
  --cluster-name qdrant-cluster \
  --capacity-providers FARGATE \
  --default-capacity-provider-strategy \
    capacityProvider=FARGATE,weight=1
```

### Step 3: Create Task Definition

Create file `qdrant-task-definition.json`:

```json
{
  "family": "qdrant-cluster",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "2048",
  "memory": "8192",
  "containerDefinitions": [
    {
      "name": "qdrant",
      "image": "qdrant/qdrant:latest",
      "essential": true,
      "portMappings": [
        {
          "containerPort": 6333,
          "protocol": "tcp"
        },
        {
          "containerPort": 6334,
          "protocol": "tcp"
        },
        {
          "containerPort": 6335,
          "protocol": "tcp"
        }
      ],
      "environment": [
        {
          "name": "QDRANT__CLUSTER__ENABLED",
          "value": "true"
        },
        {
          "name": "QDRANT__CLUSTER__P2P__PORT",
          "value": "6335"
        },
        {
          "name": "QDRANT__SERVICE__HTTP_PORT",
          "value": "6333"
        },
        {
          "name": "QDRANT__SERVICE__GRPC_PORT",
          "value": "6334"
        }
      ],
      "mountPoints": [
        {
          "sourceVolume": "qdrant-storage",
          "containerPath": "/qdrant/storage"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/qdrant-cluster",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "qdrant"
        }
      }
    }
  ],
  "volumes": [
    {
      "name": "qdrant-storage",
      "efsVolumeConfiguration": {
        "fileSystemId": "fs-xxxxx",
        "transitEncryption": "ENABLED",
        "authorizationConfig": {
          "iam": "ENABLED"
        }
      }
    }
  ]
}
```

Register task definition:

```bash
aws ecs register-task-definition \
  --cli-input-json file://qdrant-task-definition.json
```

### Step 4: Create ECS Service

```bash
# Create service with 3 tasks
aws ecs create-service \
  --cluster qdrant-cluster \
  --service-name qdrant-service \
  --task-definition qdrant-cluster:1 \
  --desired-count 3 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={
    subnets=[subnet-xxxxx,subnet-yyyyy,subnet-zzzzz],
    securityGroups=[sg-xxxxx],
    assignPublicIp=DISABLED
  }" \
  --load-balancers "targetGroupArn=$TG_ARN,containerName=qdrant,containerPort=6334"
```

### Step 5: Enable Service Auto Scaling

```bash
# Register scalable target
aws application-autoscaling register-scalable-target \
  --service-namespace ecs \
  --resource-id service/qdrant-cluster/qdrant-service \
  --scalable-dimension ecs:service:DesiredCount \
  --min-capacity 3 \
  --max-capacity 10

# Create scaling policy
aws application-autoscaling put-scaling-policy \
  --service-namespace ecs \
  --resource-id service/qdrant-cluster/qdrant-service \
  --scalable-dimension ecs:service:DesiredCount \
  --policy-name qdrant-cpu-scaling \
  --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration '{
    "TargetValue": 70.0,
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ECSServiceAverageCPUUtilization"
    },
    "ScaleInCooldown": 300,
    "ScaleOutCooldown": 60
  }'
```

---

## Option 3: EKS Kubernetes (Enterprise)

**⏱️ Setup Time:** 4-8 hours
**💰 Cost:** ~$400-600/month
**🎯 Best For:** Enterprise, large scale, existing Kubernetes infrastructure

### Quick Start with eksctl

```bash
# Install eksctl
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin

# Create EKS cluster
eksctl create cluster \
  --name qdrant-cluster \
  --version 1.28 \
  --region us-east-1 \
  --nodegroup-name qdrant-nodes \
  --node-type m5.xlarge \
  --nodes 3 \
  --nodes-min 3 \
  --nodes-max 6 \
  --managed
```

### Deploy Qdrant with Helm

```bash
# Add Qdrant Helm repo
helm repo add qdrant https://qdrant.to/helm
helm repo update

# Create values file
cat > qdrant-values.yaml <<'EOF'
replicaCount: 3

cluster:
  enabled: true

resources:
  requests:
    memory: "4Gi"
    cpu: "1000m"
  limits:
    memory: "8Gi"
    cpu: "2000m"

persistence:
  enabled: true
  size: 100Gi
  storageClass: gp3

service:
  type: LoadBalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"

ingress:
  enabled: false

metrics:
  enabled: true
  serviceMonitor:
    enabled: true
EOF

# Install Qdrant
helm install qdrant qdrant/qdrant \
  --namespace qdrant \
  --create-namespace \
  -f qdrant-values.yaml
```

### Get Service Endpoint

```bash
# Get external IP/hostname
kubectl get svc -n qdrant qdrant -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
```

For detailed Kubernetes deployment, see [REMOTE_QDRANT_CLUSTER_GUIDE.md](./REMOTE_QDRANT_CLUSTER_GUIDE.md).

---

## Option 4: EC2 Auto Scaling Group

**⏱️ Setup Time:** 2-3 hours
**💰 Cost:** ~$200-350/month
**🎯 Best For:** Production with automated scaling and recovery

### Step 1: Create Launch Template

```bash
# Create user data script
cat > qdrant-userdata.sh <<'EOF'
#!/bin/bash
yum update -y
yum install docker -y
systemctl start docker
systemctl enable docker

# Pull Qdrant image
docker pull qdrant/qdrant:latest

# Create config
mkdir -p /opt/qdrant/config
cat > /opt/qdrant/config/config.yaml <<'YAML'
cluster:
  enabled: true
  p2p:
    port: 6335
service:
  host: "0.0.0.0"
  http_port: 6333
  grpc_port: 6334
storage:
  storage_path: "/qdrant/storage"
YAML

# Run Qdrant
docker run -d \
  --name qdrant \
  --restart unless-stopped \
  -p 6333:6333 \
  -p 6334:6334 \
  -p 6335:6335 \
  -v /opt/qdrant/config:/qdrant/config \
  -v /opt/qdrant/storage:/qdrant/storage \
  qdrant/qdrant:latest
EOF

# Create launch template
aws ec2 create-launch-template \
  --launch-template-name qdrant-cluster-lt \
  --version-description "Qdrant cluster node" \
  --launch-template-data '{
    "ImageId": "ami-0c55b159cbfafe1f0",
    "InstanceType": "t3.large",
    "KeyName": "your-key-pair",
    "SecurityGroupIds": ["sg-xxxxx"],
    "BlockDeviceMappings": [{
      "DeviceName": "/dev/sda1",
      "Ebs": {
        "VolumeSize": 100,
        "VolumeType": "gp3",
        "Iops": 3000,
        "DeleteOnTermination": false
      }
    }],
    "UserData": "'$(base64 -w 0 qdrant-userdata.sh)'"
  }'
```

### Step 2: Create Auto Scaling Group

```bash
aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name qdrant-cluster-asg \
  --launch-template LaunchTemplateName=qdrant-cluster-lt \
  --min-size 3 \
  --max-size 6 \
  --desired-capacity 3 \
  --vpc-zone-identifier "subnet-xxxxx,subnet-yyyyy,subnet-zzzzz" \
  --health-check-type ELB \
  --health-check-grace-period 300 \
  --target-group-arns $TG_ARN \
  --tags "Key=Name,Value=qdrant-cluster-node"
```

### Step 3: Configure Scaling Policies

```bash
# CPU-based scaling
aws autoscaling put-scaling-policy \
  --auto-scaling-group-name qdrant-cluster-asg \
  --policy-name cpu-scale-out \
  --policy-type TargetTrackingScaling \
  --target-tracking-configuration '{
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ASGAverageCPUUtilization"
    },
    "TargetValue": 70.0
  }'
```

---

## Networking and Security

### VPC Configuration

```bash
# Create VPC
aws ec2 create-vpc --cidr-block 10.0.0.0/16

# Create subnets in 3 AZs
aws ec2 create-subnet --vpc-id vpc-xxxxx --cidr-block 10.0.1.0/24 --availability-zone us-east-1a
aws ec2 create-subnet --vpc-id vpc-xxxxx --cidr-block 10.0.2.0/24 --availability-zone us-east-1b
aws ec2 create-subnet --vpc-id vpc-xxxxx --cidr-block 10.0.3.0/24 --availability-zone us-east-1c
```

### Security Best Practices

**1. Use Private Subnets**
- Deploy Qdrant nodes in private subnets
- Only ALB in public subnet
- Use NAT Gateway for outbound traffic

**2. Security Group Rules**
```bash
# Qdrant nodes security group
# Allow gRPC from ALB only
--port 6334 --source-group <ALB-SG>

# Allow P2P between nodes only
--port 6335 --source-group <QDRANT-SG>

# No SSH from internet (use bastion or Systems Manager)
```

**3. Enable Encryption**
- EBS encryption at rest
- EFS encryption at rest
- TLS for data in transit (ALB → Qdrant)

**4. IAM Roles**
```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::qdrant-backups/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:PutMetricData"
      ],
      "Resource": "*"
    }
  ]
}
```

---

## Storage Configuration

### EBS vs EFS

| Feature | EBS | EFS |
|---------|-----|-----|
| **Use Case** | EC2, Single node | ECS Fargate, Multi-node |
| **Performance** | Higher IOPS | Moderate IOPS |
| **Cost** | $0.08/GB-month | $0.30/GB-month |
| **Shared** | No | Yes |
| **Best For** | EC2 deployments | ECS deployments |

### Optimize EBS for Qdrant

```bash
# Use gp3 with provisioned IOPS
--volume-type gp3 \
--iops 3000 \
--throughput 250

# Enable EBS optimization on instance
--ebs-optimized
```

### Backup Strategy

**Automated EBS Snapshots:**

```bash
# Create snapshot lifecycle policy
aws dlm create-lifecycle-policy \
  --description "Daily Qdrant backups" \
  --state ENABLED \
  --execution-role-arn arn:aws:iam::123456789012:role/AWSDataLifecycleManagerDefaultRole \
  --policy-details '{
    "PolicyType": "EBS_SNAPSHOT_MANAGEMENT",
    "ResourceTypes": ["VOLUME"],
    "TargetTags": [{"Key": "Role", "Value": "qdrant-cluster"}],
    "Schedules": [{
      "Name": "Daily snapshots",
      "CreateRule": {
        "Interval": 24,
        "IntervalUnit": "HOURS",
        "Times": ["03:00"]
      },
      "RetainRule": {
        "Count": 7
      },
      "CopyTags": true
    }]
  }'
```

---

## Load Balancing

### Network Load Balancer (Recommended)

**Why NLB for Qdrant:**
- ✅ Supports TCP (gRPC needs TCP)
- ✅ Lower latency than ALB
- ✅ Preserves source IP
- ✅ Higher throughput

**Configuration:**

```bash
# NLB with cross-zone load balancing
aws elbv2 create-load-balancer \
  --name qdrant-nlb \
  --type network \
  --scheme internet-facing \
  --subnets subnet-xxxxx subnet-yyyyy subnet-zzzzz \
  --tags Key=Name,Value=qdrant-cluster-nlb

# Enable cross-zone load balancing
aws elbv2 modify-load-balancer-attributes \
  --load-balancer-arn $NLB_ARN \
  --attributes Key=load_balancing.cross_zone.enabled,Value=true
```

### Application Load Balancer (Alternative)

For HTTP/HTTPS only:

```bash
aws elbv2 create-load-balancer \
  --name qdrant-alb \
  --type application \
  --scheme internet-facing \
  --subnets subnet-xxxxx subnet-yyyyy subnet-zzzzz \
  --security-groups sg-xxxxx
```

---

## Monitoring and Logging

### CloudWatch Metrics

**Enable detailed monitoring:**

```bash
# Enable for EC2
aws ec2 monitor-instances --instance-ids i-xxxxx

# Create custom metrics for Qdrant
# Add to cron job on each node:
*/5 * * * * /opt/qdrant-metrics.sh
```

**Metrics script (`qdrant-metrics.sh`):**

```bash
#!/bin/bash
INSTANCE_ID=$(ec2-metadata --instance-id | cut -d " " -f 2)
REGION=$(ec2-metadata --availability-zone | cut -d " " -f 2 | sed 's/[a-z]$//')

# Get collection count
COLLECTIONS=$(curl -s http://localhost:6333/collections | jq '.result.collections | length')

# Get total points
POINTS=$(curl -s http://localhost:6333/collections | jq '[.result.collections[].points_count] | add')

# Push to CloudWatch
aws cloudwatch put-metric-data \
  --region $REGION \
  --namespace "Qdrant/Cluster" \
  --metric-name CollectionCount \
  --value $COLLECTIONS \
  --dimensions Instance=$INSTANCE_ID

aws cloudwatch put-metric-data \
  --region $REGION \
  --namespace "Qdrant/Cluster" \
  --metric-name TotalPoints \
  --value $POINTS \
  --dimensions Instance=$INSTANCE_ID
```

### CloudWatch Logs

**Configure Docker logging:**

```bash
docker run -d \
  --log-driver=awslogs \
  --log-opt awslogs-region=us-east-1 \
  --log-opt awslogs-group=/aws/qdrant/cluster \
  --log-opt awslogs-stream=node-1 \
  qdrant/qdrant:latest
```

### CloudWatch Alarms

```bash
# High CPU alarm
aws cloudwatch put-metric-alarm \
  --alarm-name qdrant-high-cpu \
  --alarm-description "Qdrant CPU > 80%" \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --statistic Average \
  --period 300 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=AutoScalingGroupName,Value=qdrant-cluster-asg \
  --evaluation-periods 2 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:qdrant-alerts

# Health check alarm
aws cloudwatch put-metric-alarm \
  --alarm-name qdrant-unhealthy-targets \
  --alarm-description "Qdrant unhealthy targets" \
  --metric-name UnHealthyHostCount \
  --namespace AWS/NetworkELB \
  --statistic Average \
  --period 60 \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --dimensions Name=TargetGroup,Value=$TG_ARN \
  --evaluation-periods 2 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:qdrant-alerts
```

---

## Backup and Disaster Recovery

### Automated Backup Strategy

**1. EBS Snapshots (for EC2 deployments)**

Already covered in [Storage Configuration](#storage-configuration)

**2. Qdrant Native Snapshots to S3**

```bash
# Create S3 bucket
aws s3 mb s3://qdrant-backups-your-company

# Enable versioning
aws s3api put-bucket-versioning \
  --bucket qdrant-backups-your-company \
  --versioning-configuration Status=Enabled

# Create backup script
cat > /opt/qdrant-backup.sh <<'EOF'
#!/bin/bash
DATE=$(date +%Y%m%d_%H%M%S)
COLLECTIONS=$(curl -s http://localhost:6333/collections | jq -r '.result.collections[].name')

for collection in $COLLECTIONS; do
  # Create snapshot
  SNAPSHOT=$(curl -s -X POST "http://localhost:6333/collections/$collection/snapshots" | jq -r '.result.name')

  # Download snapshot
  curl -s "http://localhost:6333/collections/$collection/snapshots/$SNAPSHOT" > /tmp/${collection}_${SNAPSHOT}

  # Upload to S3
  aws s3 cp /tmp/${collection}_${SNAPSHOT} s3://qdrant-backups-your-company/${collection}/${DATE}/${SNAPSHOT}

  # Clean up local
  rm /tmp/${collection}_${SNAPSHOT}

  # Delete remote snapshot
  curl -X DELETE "http://localhost:6333/collections/$collection/snapshots/$SNAPSHOT"
done
EOF

chmod +x /opt/qdrant-backup.sh

# Add to cron (daily at 2 AM)
echo "0 2 * * * /opt/qdrant-backup.sh >> /var/log/qdrant-backup.log 2>&1" | crontab -
```

### Disaster Recovery Procedure

**1. Restore from Snapshot:**

```bash
# Download snapshot from S3
aws s3 cp s3://qdrant-backups-your-company/aws_estate/20240115_020000/snapshot.tar.gz /tmp/

# Restore collection
curl -X PUT "http://qdrant-nlb:6333/collections/aws_estate/snapshots/upload" \
  --data-binary @/tmp/snapshot.tar.gz
```

**2. Cross-Region Replication:**

```bash
# Enable S3 cross-region replication
aws s3api put-bucket-replication \
  --bucket qdrant-backups-your-company \
  --replication-configuration file://replication-config.json
```

---

## Cost Optimization

### Strategies

**1. Use Reserved Instances**
- 1-year term: ~40% savings
- 3-year term: ~60% savings

```bash
# Purchase reserved instance
aws ec2 purchase-reserved-instances-offering \
  --reserved-instances-offering-id xxxxx \
  --instance-count 3
```

**2. Use Spot Instances for Dev/Test**

```bash
# Launch template with spot
aws ec2 create-launch-template \
  --launch-template-data '{
    ...
    "InstanceMarketOptions": {
      "MarketType": "spot",
      "SpotOptions": {
        "MaxPrice": "0.05",
        "SpotInstanceType": "persistent"
      }
    }
  }'
```

**3. Right-Size Instances**

Start with t3.large, monitor, then adjust:
- Low CPU/memory: Downgrade to t3.medium
- High CPU/memory: Upgrade to m5.xlarge

**4. Use gp3 Instead of gp2**
- Same performance
- 20% cheaper

**5. Lifecycle Policies**
- Delete old snapshots after 30 days
- Move infrequent data to S3 Glacier

### Cost Calculator

**Monthly costs for 3-node cluster:**

```
EC2 Option (t3.large):
- Instances (3 × $65):        $195
- EBS (3 × 100GB gp3):        $24
- NLB:                        $35
- Data transfer:              $10
- Snapshots (100GB × 7):      $3
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Total:                        ~$267/month

With Reserved (1-year):       ~$170/month (36% savings)
With Reserved (3-year):       ~$135/month (49% savings)
```

---

## Connecting Your RAG Module

### Configuration for AWS Deployment

**Method 1: Environment Variables**

```bash
# .env.production
QDRANT_URL=http://qdrant-nlb-xxxxx.elb.us-east-1.amazonaws.com:6334
# or
QDRANT_URL=http://qdrant.yourdomain.com:6334
```

**Method 2: Config File**

```yaml
# config/production.yaml
vector_store:
  backend: "qdrant-server"
  connection:
    url: "${QDRANT_URL}"
    api_key: null
    timeout_secs: 60
  storage_path: null  # No local storage needed
```

### Your Rust Code (No Changes!)

```rust
use rag_module::RagModule;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Load config (reads QDRANT_URL from env)
    let rag = RagModule::new("./data").await?;

    // Set user context
    rag.set_user_context("user123").await?;

    // Use cluster - same code!
    let results = rag.search(
        "aws_estate",
        "find all EC2 instances in production",
        None
    ).await?;

    println!("Found {} results", results.len());

    Ok(())
}
```

### Connection Pooling (Optional Enhancement)

For high-concurrency applications:

```rust
// Enhanced connection with connection pool
impl QdrantServerVectorStore {
    pub async fn new_with_pool(
        cluster_urls: Vec<String>,  // Multiple NLB endpoints
        pool_size: usize,
        // ... other params
    ) -> Result<Self> {
        let mut clients = Vec::new();

        for url in &cluster_urls {
            for _ in 0..pool_size {
                let client = Qdrant::from_url(url)
                    .timeout(Duration::from_secs(60))
                    .build()?;
                clients.push(client);
            }
        }

        // Round-robin load balancing
        // ... rest of implementation
    }
}
```

---

## Troubleshooting

### Common Issues

**Issue 1: Nodes Can't Join Cluster**

```bash
# Check P2P connectivity
telnet 10.0.1.10 6335

# Check security group
aws ec2 describe-security-groups --group-ids sg-xxxxx

# Check cluster logs
docker logs qdrant-node1 | grep -i "cluster"
```

**Solution:**
- Verify port 6335 is open between nodes
- Check security group allows traffic from cluster SG

**Issue 2: High Latency**

```bash
# Check network latency
ping -c 10 <qdrant-node-ip>

# Check query performance
curl -w "@curl-format.txt" http://qdrant-nlb:6333/collections

# curl-format.txt:
time_total: %{time_total}s
```

**Solution:**
- Enable cross-zone load balancing
- Place nodes and clients in same AZ when possible
- Increase timeout in config

**Issue 3: Out of Memory**

```bash
# Check memory usage
docker stats qdrant-node1

# Check collection sizes
curl http://localhost:6333/collections | jq '.result.collections[] | {name, points_count}'
```

**Solution:**
- Upgrade instance type (more RAM)
- Implement data retention policy
- Shard collections across more nodes

**Issue 4: ELB Health Checks Failing**

```bash
# Test health endpoint
curl http://10.0.1.10:6333/health

# Check target health
aws elbv2 describe-target-health --target-group-arn $TG_ARN
```

**Solution:**
- Verify Qdrant is running (`docker ps`)
- Check security group allows health check port (6333)
- Increase health check interval/timeout

### Useful Commands

```bash
# View cluster status
curl http://qdrant-nlb:6333/cluster | jq

# List collections
curl http://qdrant-nlb:6333/collections | jq

# Get collection info
curl http://qdrant-nlb:6333/collections/aws_estate | jq

# View metrics
curl http://qdrant-nlb:6333/metrics

# Check node logs
docker logs -f --tail 100 qdrant-node1

# SSH to node via Systems Manager (no SSH key needed)
aws ssm start-session --target i-xxxxx
```

---

## Summary

### Recommended Approach by Use Case

| **Use Case** | **Recommended Option** | **Why** |
|--------------|----------------------|---------|
| **Quick start, Dev/Test** | EC2 + Docker | Simple, fast setup, full control |
| **Production, Low ops** | ECS Fargate | Managed, auto-scaling, no servers |
| **Enterprise, Large scale** | EKS | Advanced orchestration, existing K8s |
| **Production, Cost-optimized** | EC2 ASG | Automated, cost-effective, flexible |

### Cost Comparison Summary

```
EC2 + Docker:     $267/month (can reduce to $135 with 3-year RI)
ECS Fargate:      $350/month (fully managed)
EKS:              $550/month (control plane + nodes)
EC2 ASG:          $280/month (can reduce to $145 with 3-year RI)

Compare to:
Qdrant Cloud:     $11-105/month (no ops overhead!)
```

### Key Takeaways

1. **Start Simple:** Begin with EC2 + Docker for learning
2. **Scale Smart:** Move to ECS Fargate for production
3. **Go Big:** Use EKS for enterprise requirements
4. **Consider Managed:** Qdrant Cloud is often cheaper when factoring ops time

### Your Code is Ready

Your `QdrantServerVectorStore` works with ALL these options:
- Just change the URL in config
- No code modifications needed
- Same API, different infrastructure

---

## Next Steps

1. **Choose deployment option** based on your needs
2. **Follow setup instructions** for chosen option
3. **Test connection** from your Rust app
4. **Set up monitoring** and alerts
5. **Configure backups** and disaster recovery
6. **Optimize costs** with reserved instances

**Need help?** Check related documentation:
- [HORIZONTAL_SCALING_GUIDE.md](./HORIZONTAL_SCALING_GUIDE.md) - When and why to scale
- [REMOTE_QDRANT_CLUSTER_GUIDE.md](./REMOTE_QDRANT_CLUSTER_GUIDE.md) - Cluster architecture details
- [HOW_TO_RUN.md](./HOW_TO_RUN.md) - Running examples

Happy clustering! 🚀
