# rstmdb Kubernetes Operator - Technical Specification

## Overview

Build a Kubernetes Operator to manage rstmdb database instances declaratively using Custom Resource Definitions (CRDs). The operator will handle the full lifecycle of rstmdb deployments including provisioning, scaling, configuration, backups, and monitoring.

**Repository**: `github.com/rstmdb/rstmdb-operator`
**Language**: Go (using Kubebuilder/controller-runtime)
**Target**: Kubernetes 1.26+

---

## Goals

1. **Declarative Management** - Deploy and manage rstmdb clusters via CRDs
2. **Lifecycle Automation** - Handle creation, updates, scaling, and deletion
3. **High Availability** - Support multi-replica deployments with leader election
4. **Configuration Management** - Manage rstmdb config, TLS, and authentication
5. **Observability** - Integrate with Prometheus metrics and Grafana dashboards
6. **Backup/Restore** - Automated WAL snapshots and point-in-time recovery
7. **State Machine Management** - Deploy state machine definitions as Kubernetes resources

---

## Custom Resource Definitions (CRDs)

### 1. RstmdbCluster

Primary resource for deploying an rstmdb database cluster.

```yaml
apiVersion: rstmdb.io/v1alpha1
kind: RstmdbCluster
metadata:
  name: my-rstmdb
  namespace: default
spec:
  # Replicas (1 for standalone, 3+ for HA)
  replicas: 3

  # Image configuration
  image:
    repository: rstmdb/rstmdb
    tag: "0.1.0"
    pullPolicy: IfNotPresent

  # Resource requirements
  resources:
    requests:
      cpu: "500m"
      memory: "512Mi"
    limits:
      cpu: "2"
      memory: "2Gi"

  # Storage configuration
  storage:
    className: standard
    size: 10Gi
    # WAL storage (optional, separate volume)
    walSize: 5Gi

  # Network configuration
  service:
    type: ClusterIP  # ClusterIP | LoadBalancer | NodePort
    port: 7401
    metricsPort: 9090

  # TLS configuration
  tls:
    enabled: true
    # Auto-generate using cert-manager
    certManager:
      issuerRef:
        name: letsencrypt-prod
        kind: ClusterIssuer
    # Or provide existing secret
    secretName: rstmdb-tls
    # mTLS for client authentication
    clientAuth: require  # none | request | require

  # Authentication
  auth:
    enabled: true
    # Secret containing tokens
    secretName: rstmdb-auth
    # Keys in secret: admin-token, readonly-token

  # Configuration overrides
  config:
    walFsyncInterval: "100ms"
    maxConnections: 1000
    idleTimeout: "300s"
    snapshotInterval: 10000

  # Prometheus metrics
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
      interval: "30s"
      labels:
        release: prometheus

  # Backup configuration
  backup:
    enabled: true
    schedule: "0 2 * * *"  # Daily at 2 AM
    retention: 7  # Keep 7 backups
    storage:
      s3:
        bucket: rstmdb-backups
        region: us-east-1
        secretName: s3-credentials
      # Or GCS
      gcs:
        bucket: rstmdb-backups
        secretName: gcs-credentials

  # Pod configuration
  podTemplate:
    annotations:
      prometheus.io/scrape: "true"
    labels:
      app.kubernetes.io/part-of: my-app
    nodeSelector:
      kubernetes.io/os: linux
    tolerations: []
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: rstmdb
              topologyKey: kubernetes.io/hostname

status:
  phase: Running  # Pending | Running | Failed | Updating
  replicas: 3
  readyReplicas: 3
  leader: my-rstmdb-0
  conditions:
    - type: Ready
      status: "True"
      lastTransitionTime: "2024-01-15T10:00:00Z"
    - type: TLSReady
      status: "True"
    - type: BackupReady
      status: "True"
  endpoints:
    - my-rstmdb-0.my-rstmdb-headless:7401
    - my-rstmdb-1.my-rstmdb-headless:7401
    - my-rstmdb-2.my-rstmdb-headless:7401
  walOffset: 1042
  lastBackup: "2024-01-15T02:00:00Z"
```

### 2. RstmdbStateMachine

Deploy state machine definitions to rstmdb clusters.

```yaml
apiVersion: rstmdb.io/v1alpha1
kind: RstmdbStateMachine
metadata:
  name: order-workflow
  namespace: default
spec:
  # Target cluster
  clusterRef:
    name: my-rstmdb
    namespace: default  # Optional, defaults to same namespace

  # Machine name and version
  machine: order
  version: 1

  # State machine definition
  definition:
    states:
      - pending
      - confirmed
      - shipped
      - delivered
      - cancelled
    initial: pending
    transitions:
      - from: pending
        event: CONFIRM
        to: confirmed
      - from: pending
        event: CANCEL
        to: cancelled
      - from: confirmed
        event: SHIP
        to: shipped
        guard: "$.tracking_number != null"
      - from: shipped
        event: DELIVER
        to: delivered
    meta:
      description: "Order fulfillment workflow"
      version: "1.0.0"

status:
  phase: Deployed  # Pending | Deployed | Failed
  checksum: "8f3a2b1c"
  deployedAt: "2024-01-15T10:05:00Z"
  conditions:
    - type: Deployed
      status: "True"
      lastTransitionTime: "2024-01-15T10:05:00Z"
```

### 3. RstmdbBackup

Trigger manual backups or view backup status.

```yaml
apiVersion: rstmdb.io/v1alpha1
kind: RstmdbBackup
metadata:
  name: my-rstmdb-backup-20240115
  namespace: default
spec:
  clusterRef:
    name: my-rstmdb
  # Optional: specific WAL offset to backup up to
  walOffset: 1042

status:
  phase: Completed  # Pending | Running | Completed | Failed
  startedAt: "2024-01-15T10:00:00Z"
  completedAt: "2024-01-15T10:05:00Z"
  walOffset: 1042
  size: "256Mi"
  location: "s3://rstmdb-backups/my-rstmdb/2024-01-15T10:00:00Z"
```

### 4. RstmdbRestore

Restore from a backup.

```yaml
apiVersion: rstmdb.io/v1alpha1
kind: RstmdbRestore
metadata:
  name: my-rstmdb-restore
  namespace: default
spec:
  # Target cluster (will be stopped during restore)
  clusterRef:
    name: my-rstmdb

  # Source backup
  backupRef:
    name: my-rstmdb-backup-20240115

  # Or restore from specific location
  source:
    s3:
      bucket: rstmdb-backups
      path: my-rstmdb/2024-01-15T10:00:00Z
      secretName: s3-credentials

status:
  phase: Completed  # Pending | Running | Completed | Failed
  startedAt: "2024-01-15T12:00:00Z"
  completedAt: "2024-01-15T12:10:00Z"
  restoredWalOffset: 1042
```

---

## Operator Architecture

### Components

```
┌─────────────────────────────────────────────────────────────┐
│                    rstmdb-operator                          │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────────┐  ┌─────────────────┐                  │
│  │ Cluster         │  │ StateMachine    │                  │
│  │ Controller      │  │ Controller      │                  │
│  └────────┬────────┘  └────────┬────────┘                  │
│           │                    │                            │
│  ┌────────┴────────┐  ┌────────┴────────┐                  │
│  │ Backup          │  │ Restore         │                  │
│  │ Controller      │  │ Controller      │                  │
│  └────────┬────────┘  └────────┬────────┘                  │
│           │                    │                            │
│  ┌────────┴────────────────────┴────────┐                  │
│  │         Reconciliation Loop          │                  │
│  └──────────────────────────────────────┘                  │
│                                                             │
│  ┌──────────────────────────────────────┐                  │
│  │         rstmdb Client Library        │                  │
│  │         (for health checks)          │                  │
│  └──────────────────────────────────────┘                  │
│                                                             │
└─────────────────────────────────────────────────────────────┘
```

### Controller Responsibilities

#### RstmdbCluster Controller

1. **Create/Update Resources**:
   - StatefulSet for rstmdb pods
   - Headless Service for pod discovery
   - ClusterIP/LoadBalancer Service for client access
   - ConfigMap for rstmdb configuration
   - Secret for authentication tokens
   - PodDisruptionBudget for HA
   - ServiceMonitor for Prometheus

2. **Reconciliation Logic**:
   ```
   func Reconcile(ctx, req):
     cluster = get RstmdbCluster

     # Ensure ConfigMap
     reconcileConfigMap(cluster)

     # Ensure Secrets (TLS, Auth)
     reconcileSecrets(cluster)

     # Ensure StatefulSet
     reconcileStatefulSet(cluster)

     # Ensure Services
     reconcileServices(cluster)

     # Ensure PodDisruptionBudget
     reconcilePDB(cluster)

     # Ensure ServiceMonitor
     reconcileServiceMonitor(cluster)

     # Update status
     updateStatus(cluster)
   ```

3. **Health Checks**:
   - Connect to rstmdb and call `ping`
   - Verify leader election status
   - Check WAL health
   - Monitor connection count

4. **Scaling**:
   - Scale up: Add replicas, wait for sync
   - Scale down: Graceful shutdown, ensure data replicated

#### RstmdbStateMachine Controller

1. **Deploy Definitions**:
   - Connect to target cluster
   - Call `put_machine` API
   - Verify checksum matches

2. **Reconciliation**:
   ```
   func Reconcile(ctx, req):
     sm = get RstmdbStateMachine
     cluster = get referenced RstmdbCluster

     if cluster not ready:
       return requeue

     client = connect to cluster

     result = client.put_machine(
       sm.spec.machine,
       sm.spec.version,
       sm.spec.definition
     )

     update status with checksum
   ```

#### Backup Controller

1. **Scheduled Backups**:
   - Watch CronJob-like schedule
   - Create RstmdbBackup resources on schedule

2. **Backup Execution**:
   - Trigger WAL snapshot
   - Upload to object storage (S3/GCS)
   - Record backup metadata

3. **Retention**:
   - Delete old backups based on retention policy

#### Restore Controller

1. **Restore Process**:
   - Stop rstmdb cluster
   - Download backup from storage
   - Restore data directory
   - Restart cluster
   - Verify health

---

## Implementation Plan

### Phase 1: Core Operator (MVP)

**Duration**: 2-3 weeks

1. **Project Setup**
   - Initialize with Kubebuilder
   - Define CRD schemas
   - Setup CI/CD pipeline

2. **RstmdbCluster Controller**
   - Basic StatefulSet management
   - Service creation
   - ConfigMap management
   - Status updates

3. **Basic Features**
   - Single-replica deployment
   - Persistent storage
   - Basic health checks

**Deliverables**:
- Deploy standalone rstmdb via CRD
- Basic CRUD operations working

### Phase 2: Security & Configuration

**Duration**: 2 weeks

1. **TLS Support**
   - Self-signed certificates
   - cert-manager integration
   - mTLS for clients

2. **Authentication**
   - Token-based auth
   - Secret management
   - Token rotation

3. **Configuration**
   - Full config override support
   - Config validation
   - Rolling updates on config change

**Deliverables**:
- Secure deployments with TLS
- Authentication working
- Configuration management

### Phase 3: High Availability

**Duration**: 2-3 weeks

1. **Multi-Replica Support**
   - Leader election
   - Replica synchronization
   - Automatic failover

2. **Pod Disruption**
   - PodDisruptionBudget
   - Graceful shutdown
   - Rolling updates

3. **Topology Awareness**
   - Pod anti-affinity
   - Zone spreading
   - Node selection

**Deliverables**:
- HA cluster deployment
- Automatic failover
- Zero-downtime updates

### Phase 4: State Machine Management

**Duration**: 1-2 weeks

1. **RstmdbStateMachine CRD**
   - Schema definition
   - Validation webhooks
   - Status tracking

2. **Controller Implementation**
   - Deploy definitions
   - Version management
   - Checksum verification

**Deliverables**:
- Deploy state machines via CRD
- GitOps-friendly workflow

### Phase 5: Backup & Restore

**Duration**: 2-3 weeks

1. **Backup Implementation**
   - Manual backups
   - Scheduled backups (CronJob)
   - S3/GCS upload

2. **Restore Implementation**
   - Point-in-time restore
   - Cluster recovery
   - Data verification

3. **Retention Management**
   - Automatic cleanup
   - Retention policies

**Deliverables**:
- Automated backups
- Disaster recovery capability

### Phase 6: Observability & Polish

**Duration**: 1-2 weeks

1. **Prometheus Integration**
   - ServiceMonitor creation
   - Custom metrics
   - Alert rules

2. **Grafana Dashboard**
   - Pre-configured dashboard ConfigMap
   - Link to existing dashboard (ID: 24778)

3. **Documentation**
   - API reference
   - User guide
   - Examples

**Deliverables**:
- Full observability stack
- Production-ready documentation

---

## Technical Requirements

### Dependencies

```go
// go.mod
module github.com/rstmdb/rstmdb-operator

go 1.21

require (
    k8s.io/api v0.29.0
    k8s.io/apimachinery v0.29.0
    k8s.io/client-go v0.29.0
    sigs.k8s.io/controller-runtime v0.17.0

    // rstmdb client (Go bindings or raw TCP)
    github.com/rstmdb/rstmdb-go v0.1.0

    // Object storage
    github.com/aws/aws-sdk-go-v2 v1.24.0
    cloud.google.com/go/storage v1.36.0

    // TLS
    github.com/cert-manager/cert-manager v1.14.0
)
```

### Project Structure

```
rstmdb-operator/
├── api/
│   └── v1alpha1/
│       ├── rstmdbcluster_types.go
│       ├── rstmdbstatemachine_types.go
│       ├── rstmdbbackup_types.go
│       ├── rstmdbrestore_types.go
│       ├── groupversion_info.go
│       └── zz_generated.deepcopy.go
├── cmd/
│   └── main.go
├── config/
│   ├── crd/
│   │   └── bases/
│   ├── manager/
│   ├── rbac/
│   ├── samples/
│   └── default/
├── internal/
│   ├── controller/
│   │   ├── rstmdbcluster_controller.go
│   │   ├── rstmdbstatemachine_controller.go
│   │   ├── rstmdbbackup_controller.go
│   │   └── rstmdbrestore_controller.go
│   ├── resources/
│   │   ├── statefulset.go
│   │   ├── service.go
│   │   ├── configmap.go
│   │   ├── secret.go
│   │   └── pdb.go
│   ├── rstmdb/
│   │   └── client.go  # rstmdb client wrapper
│   └── backup/
│       ├── s3.go
│       └── gcs.go
├── test/
│   ├── e2e/
│   └── integration/
├── Dockerfile
├── Makefile
├── go.mod
└── README.md
```

### RBAC Requirements

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: rstmdb-operator
rules:
  # Core resources
  - apiGroups: [""]
    resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
    verbs: [get, list, watch, create, update, patch, delete]

  - apiGroups: ["apps"]
    resources: [statefulsets]
    verbs: [get, list, watch, create, update, patch, delete]

  - apiGroups: ["policy"]
    resources: [poddisruptionbudgets]
    verbs: [get, list, watch, create, update, patch, delete]

  # rstmdb CRDs
  - apiGroups: ["rstmdb.io"]
    resources: [rstmdbclusters, rstmdbstatemachines, rstmdbbackups, rstmdbrestores]
    verbs: [get, list, watch, create, update, patch, delete]

  - apiGroups: ["rstmdb.io"]
    resources: [rstmdbclusters/status, rstmdbstatemachines/status, rstmdbbackups/status, rstmdbrestores/status]
    verbs: [get, update, patch]

  # Prometheus ServiceMonitor
  - apiGroups: ["monitoring.coreos.com"]
    resources: [servicemonitors]
    verbs: [get, list, watch, create, update, patch, delete]

  # cert-manager (optional)
  - apiGroups: ["cert-manager.io"]
    resources: [certificates]
    verbs: [get, list, watch, create, update, patch, delete]
```

---

## Testing Strategy

### Unit Tests

- Controller reconciliation logic
- Resource generation
- Status calculations

### Integration Tests

- CRD validation
- Controller with fake client
- Webhook validation

### E2E Tests

```go
var _ = Describe("RstmdbCluster", func() {
    Context("When creating a cluster", func() {
        It("Should create StatefulSet with correct replicas", func() {
            cluster := &v1alpha1.RstmdbCluster{
                Spec: v1alpha1.RstmdbClusterSpec{
                    Replicas: 3,
                },
            }
            Expect(k8sClient.Create(ctx, cluster)).To(Succeed())

            Eventually(func() int32 {
                sts := &appsv1.StatefulSet{}
                k8sClient.Get(ctx, key, sts)
                return *sts.Spec.Replicas
            }).Should(Equal(int32(3)))
        })
    })
})
```

### Test Environment

- Kind cluster for local testing
- GitHub Actions for CI
- Real Kubernetes for E2E

---

## Deployment

### Helm Chart

```yaml
# values.yaml
image:
  repository: rstmdb/rstmdb-operator
  tag: v0.1.0
  pullPolicy: IfNotPresent

resources:
  limits:
    cpu: 500m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 128Mi

# Leader election for HA
leaderElection:
  enabled: true

# Metrics
metrics:
  enabled: true
  port: 8080

# Webhook (for validation)
webhook:
  enabled: true
  port: 9443
```

### OLM (Operator Lifecycle Manager)

- Create ClusterServiceVersion
- Publish to OperatorHub
- Support for OLM upgrades

---

## Success Criteria

1. **Functional**
   - [ ] Deploy single-node rstmdb cluster
   - [ ] Deploy HA rstmdb cluster (3+ replicas)
   - [ ] Configure TLS and authentication
   - [ ] Deploy state machine definitions
   - [ ] Perform backup and restore
   - [ ] Scale cluster up/down

2. **Operational**
   - [ ] Zero-downtime rolling updates
   - [ ] Automatic failover in HA mode
   - [ ] Prometheus metrics exported
   - [ ] Grafana dashboard working

3. **Quality**
   - [ ] 80%+ unit test coverage
   - [ ] E2E tests passing
   - [ ] Documentation complete
   - [ ] Published to OperatorHub

---

## Open Questions

1. **Replication**: Does rstmdb support native replication, or should the operator handle it at the storage level?

2. **Leader Election**: How does rstmdb handle leader election in multi-replica setup? Does the operator need to manage this?

3. **Go Client**: Is there a Go client library for rstmdb, or should we implement one / use raw TCP?

4. **Backup Format**: What's the best way to backup rstmdb? WAL + snapshots? Full data directory?

---

## References

- [Kubebuilder Book](https://book.kubebuilder.io/)
- [Operator SDK](https://sdk.operatorframework.io/)
- [rstmdb Documentation](https://github.com/rstmdb/rstmdb)
- [rstmdb Helm Chart](https://artifacthub.io/packages/helm/rstmdb/rstmdb)
- [rstmdb Grafana Dashboard](https://grafana.com/grafana/dashboards/24778-rstmdb-dashboard/)
