# mecha10-diagnostics

Topic-based diagnostics and performance monitoring service for Mecha10.

## Overview

The diagnostics service provides comprehensive performance monitoring through the framework's pub/sub system. All diagnostics are published to topics under the `/diagnostics` namespace, allowing real-time monitoring via CLI, dashboard, or custom subscribers.

## Features

- **Topic-based architecture**: Seamlessly integrates with Mecha10's pub/sub system
- **Streaming pipeline metrics**: Frame pipeline, latency, encoding performance, bandwidth
- **WebRTC metrics**: Connection stats, quality metrics (RTT, packet loss, jitter)
- **WebSocket metrics**: Connection tracking, message rates
- **Redis metrics**: Connection pool, operation latency and throughput
- **Docker metrics**: Container CPU, memory, network, and I/O stats
- **System metrics**: Host CPU, memory, disk, and network usage
- **Low overhead**: Atomic counters on hot paths, background aggregation

## Diagnostic Topics

All diagnostics use the `/diagnostics` namespace:

```
/diagnostics/streaming/pipeline     # Frame pipeline metrics
/diagnostics/streaming/latency      # Latency measurements
/diagnostics/streaming/encoding     # Encoding performance
/diagnostics/streaming/bandwidth    # Bandwidth usage
/diagnostics/webrtc/connections     # WebRTC peer connections
/diagnostics/webrtc/quality         # RTT, packet loss, jitter
/diagnostics/websocket/connections  # WebSocket connections
/diagnostics/websocket/messages     # WebSocket message stats
/diagnostics/redis/pool             # Redis connection pool
/diagnostics/redis/operations       # Redis operation metrics
/diagnostics/docker/containers      # Docker container stats
/diagnostics/godot/performance      # Godot FPS, frame time, physics
/diagnostics/godot/scene            # Godot node count, memory
/diagnostics/godot/connection       # Godot WebSocket health
/diagnostics/system/resources       # System-wide resources
/diagnostics/node/{id}/health       # Per-node health
```

## Usage

### Publishing Streaming Diagnostics

```rust
use mecha10_diagnostics::prelude::*;
use mecha10_core::prelude::*;

// Create collector
let collector = StreamingCollector::new("simulation-bridge");

// Record metrics on hot path (minimal overhead)
collector.record_frame_received();
collector.record_frame_encoded(5000); // 5ms encode time
collector.record_frame_sent(10000);   // 10KB frame

// Publish aggregated metrics periodically (every 1-5 seconds)
collector.publish_all(&ctx, 400_000).await?;
```

### Subscribing to Diagnostics

```rust
use mecha10_diagnostics::prelude::*;
use mecha10_core::prelude::*;

// Subscribe to streaming pipeline metrics
let mut rx = ctx.subscribe(
    Topic::<DiagnosticMessage<StreamingPipelineMetrics>>::new(
        TOPIC_DIAGNOSTICS_STREAMING_PIPELINE
    )
).await?;

while let Some(msg) = rx.recv().await {
    println!("FPS: {}, Frames dropped: {}",
        msg.payload.fps,
        msg.payload.frames_dropped
    );
}
```

### Docker Metrics

```rust
// Create Docker collector
let docker = DockerCollector::new("diagnostics-service").await;

// Collect metrics for all containers
docker.collect_all_containers(&ctx).await?;
```

### System Metrics

```rust
// Create system collector
let system = SystemCollector::new("diagnostics-service");

// Collect and publish system metrics
system.collect_metrics(&ctx).await?;
```

### Godot Metrics

```rust
// Create Godot collector
let godot = GodotCollector::new(
    "simulation-bridge",
    "ws://localhost:11008".to_string(),  // Control URL
    "ws://localhost:11009".to_string(),  // Camera URL
);

// Track connection events
godot.set_control_connected(true);
godot.set_camera_connected(true);

// Record messages
godot.record_control_message(now_micros());
godot.record_camera_frame(now_micros());

// Publish connection health
godot.publish_connection_metrics(&ctx).await?;

// Publish performance metrics (data from Godot)
godot.publish_performance_metrics(
    &ctx,
    60.0,   // fps
    60.0,   // target_fps
    16.6,   // frame_time_ms
    2.5,    // physics_time_ms
    10.0,   // render_time_ms
    4.1,    // idle_time_ms
).await?;
```

## CLI Integration

Monitor diagnostics in real-time:

```bash
# View all diagnostics
mecha10 diagnostics

# Filter by category
mecha10 diagnostics --category streaming
mecha10 diagnostics --category docker

# Live streaming metrics only
mecha10 diagnostics --streaming

# Historical query
mecha10 diagnostics --from "2024-01-01 12:00" --to "2024-01-01 13:00"

# Export to file
mecha10 diagnostics --export metrics.json
```

## Dashboard Integration

The diagnostics service integrates with the dashboard via WebSocket:

- Real-time charts and visualizations
- Bottleneck detection
- Alerting on anomalies
- Historical analysis

Access at: `http://localhost:3000/dashboard/diagnostics`

## Metric Types

### Counter
Monotonically increasing value (e.g., frames received)

```rust
let counter = Counter::new();
counter.inc();
counter.add(5);
let total = counter.get();
```

### Gauge
Current value that can go up or down (e.g., queue depth)

```rust
let gauge = Gauge::new();
gauge.set(10);
gauge.inc();
gauge.dec();
```

### Histogram
Distribution tracking with percentiles (e.g., encoding latency)

```rust
let mut hist = Histogram::for_latency()?;
hist.record(5000); // Record 5ms
let p95 = hist.p95();
let p99 = hist.p99();
```

## Performance

The diagnostics system is designed for minimal overhead:

- **Atomic counters**: Zero-cost increment on hot paths
- **Background aggregation**: Expensive operations (percentiles) run off hot path
- **Rate limiting**: Automatic 1-second intervals for publishing
- **Zero-copy**: Efficient Arc-based data sharing

Overhead on streaming pipeline: **< 1% CPU**, **< 1MB memory**

## Architecture

```
┌─────────────────────────────────────────┐
│        Diagnostic Publishers            │
│  (Simulation Bridge, Nodes, Services)   │
└────────────────┬────────────────────────┘
                 │
                 ▼
         ┌───────────────┐
         │ Redis Pub/Sub │
         └───────────────┘
                 │
        ┌────────┴────────┐
        ▼                 ▼
   ┌────────┐      ┌──────────┐
   │  CLI   │      │Dashboard │
   └────────┘      └──────────┘
        │                 │
        ▼                 ▼
┌──────────────────────────────┐
│    Telemetry Service         │
│    (Historical Archive)      │
└──────────────────────────────┘
```

## Future Enhancements

1. **Redis instrumentation**: Add detailed operation tracking to mecha10-core::Context
2. **WebRTC stats**: Extract metrics from WebRTC peer connections
3. **WebSocket tracking**: Instrument WebSocket connections
4. **Custom metrics**: Allow nodes to publish custom diagnostic metrics
5. **Alerting**: Automatic alerts on threshold violations
6. **Profiling integration**: Integration with performance profiling tools

## License

MIT
