    Finished `release` profile [optimized] target(s) in 1.70s
     Running tests/model_comparison.rs (/Users/jamessheen/.cargo-shared-target/kaironiq/release/deps/model_comparison-1238d4cb13964449)

running 1 test
test model_comparison_tests::test_model_comparison ... 
============================================================
  MODEL COMPARISON — Entity Recall Benchmark
============================================================

  SKIP: NuExtract-4B (template) (RQL_NUEXTRACT_MODEL_PATH not set)

  SKIP: NuExtract-4B (grounded) (RQL_NUEXTRACT_MODEL_PATH not set)

  SKIP: Phi-4-mini (3-stage) (RQL_PHI4_MODEL_PATH not set)

  SKIP: Phi-4-mini (3-stage-nogrm) (RQL_PHI4_MODEL_PATH not set)

  Loading: Qwen-3B (template) (/Users/jamessheen/Documents/Projects/Ideas/ai-meeting-transcription/models/qwen2.5-3b-instruct-q4_k_m.gguf)
  Running fixtures for: Qwen-3B (template)
    Mock Interview — R:90% P:64% F1:75% (9/10 found, 14 extracted) in 23.8s  miss=["ziad"]
warn: failed to parse NuExtract JSON after repair: invalid type: string "PLAT-4521\"", expected struct NuExtractOutput
    Tech Standup — R:100% P:52% F1:68% (13/13 found, 25 extracted) in 26.1s  miss=[]
    Podcast Interview — R:100% P:43% F1:60% (15/15 found, 35 extracted) in 28.1s  miss=[]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Sales Call — R:100% P:43% F1:60% (12/12 found, 28 extracted) in 26.3s  miss=[]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Legal Deposition — R:88% P:32% F1:47% (14/16 found, 44 extracted) in 22.4s  miss=["clifford reeves", "february 14, 2024"]
    Medical Consultation — R:71% P:45% F1:56% (10/14 found, 22 extracted) in 24.9s  miss=["dr. patel", "st. catherine's general hospital", "hypertrophic cardiomyopathy", "type 2 diabetes"]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Academic Lecture — R:95% P:55% F1:70% (21/22 found, 38 extracted) in 4.5s  miss=["mit"]
    Customer Support — R:100% P:38% F1:55% (9/9 found, 24 extracted) in 25.1s  miss=[]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Board Meeting — R:88% P:27% F1:42% (15/17 found, 55 extracted) in 25.8s  miss=["niall o'brien", "verdant capital partners ltd."]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    News Article — R:94% P:58% F1:71% (15/16 found, 26 extracted) in 16.5s  miss=["washington d.c."]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Slack Thread — R:83% P:71% F1:77% (10/12 found, 14 extracted) in 24.3s  miss=["pagerduty", "grafana"]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Product Review — R:80% P:57% F1:67% (8/10 found, 14 extracted) in 24.0s  miss=["za-800", "active noise cancellation"]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Short Snippet — R:100% P:67% F1:80% (4/4 found, 6 extracted) in 6.7s  miss=[]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Long Report — R:95% P:40% F1:57% (36/38 found, 89 extracted) in 28.0s  miss=["münchener rückversicherungs-gesellschaft", "columbus, ohio"]

  Loading: Qwen-3B (3-stage) (/Users/jamessheen/Documents/Projects/Ideas/ai-meeting-transcription/models/qwen2.5-3b-instruct-q4_k_m.gguf)
  Running fixtures for: Qwen-3B (3-stage)
    Mock Interview — R:90% P:53% F1:67% (9/10 found, 17 extracted) in 42.3s  miss=["ziad"]
    Tech Standup — R:100% P:45% F1:62% (13/13 found, 29 extracted) in 18.9s  miss=[]
    Podcast Interview — R:100% P:34% F1:51% (15/15 found, 44 extracted) in 37.4s  miss=[]
    Sales Call — R:100% P:35% F1:52% (12/12 found, 34 extracted) in 21.2s  miss=[]
    Legal Deposition — R:88% P:26% F1:41% (14/16 found, 53 extracted) in 21.1s  miss=["clifford reeves", "february 14, 2024"]
    Medical Consultation — R:79% P:39% F1:52% (11/14 found, 28 extracted) in 38.1s  miss=["dr. patel", "st. catherine's general hospital", "type 2 diabetes"]
warn: skipping fact (Ferdinand de Saussure -> isA -> Person): database error: SQLite failure: `FOREIGN KEY constraint failed`
warn: skipping fact (Noam Chomsky -> isA -> Person): database error: SQLite failure: `FOREIGN KEY constraint failed`
warn: skipping fact (University of Chicago Press -> isA -> Organisation): database error: SQLite failure: `FOREIGN KEY constraint failed`
warn: skipping fact (Olusegun Bello -> isA -> Person): database error: SQLite failure: `FOREIGN KEY constraint failed`
    Academic Lecture — R:95% P:42% F1:58% (21/22 found, 50 extracted) in 42.8s  miss=["mit"]
warn: skipping fact (Dan Kowalski -> customer_has_account -> VSC-448821): database error: SQLite failure: `FOREIGN KEY constraint failed`
warn: skipping fact (Dan Kowalski -> customer_has_issue -> error code E-5503): database error: SQLite failure: `FOREIGN KEY constraint failed`
warn: skipping fact (Dan Kowalski -> customer_has_ticket -> SUP-109342): database error: SQLite failure: `FOREIGN KEY constraint failed`
    Customer Support — R:100% P:28% F1:44% (9/9 found, 32 extracted) in 25.5s  miss=[]
warn: skipping fact (Roland Strasser -> CEO_REPORT -> Q4 trading update): database error: SQLite failure: `FOREIGN KEY constraint failed`
warn: skipping fact (Beatriz Souza -> APPROVAL_OF_FY2024_BUDGET -> FY2024 budget): database error: SQLite failure: `FOREIGN KEY constraint failed`
warn: skipping fact (Ingrid Holmberg -> APPOINTMENT_TO_REMUNERATION_COMMITTEE -> Chair of the Remuneration Committee): database error: SQLite failure: `FOREIGN KEY constraint failed`
    Board Meeting — R:88% P:27% F1:41% (15/17 found, 56 extracted) in 34.6s  miss=["niall o'brien", "verdant capital partners ltd."]
    News Article — R:94% P:50% F1:65% (15/16 found, 30 extracted) in 50.0s  miss=["washington d.c."]
    Slack Thread — R:92% P:48% F1:63% (11/12 found, 23 extracted) in 42.0s  miss=["pagerduty"]
    Product Review — R:80% P:44% F1:57% (8/10 found, 18 extracted) in 53.0s  miss=["za-800", "active noise cancellation"]
    Short Snippet — R:100% P:44% F1:62% (4/4 found, 9 extracted) in 16.9s  miss=[]
    Long Report — R:95% P:35% F1:51% (36/38 found, 103 extracted) in 73.2s  miss=["münchener rückversicherungs-gesellschaft", "columbus, ohio"]

  SKIP: Gemma-4-E2B (3-stage) (RQL_GEMMA4_MODEL_PATH not set)

  SKIP: Gemma-4-E2B (3-stage-nogrm) (RQL_GEMMA4_MODEL_PATH not set)

  SKIP: Gemma-4-E2B (template) (RQL_GEMMA4_MODEL_PATH not set)

============================================================
  MODEL COMPARISON — Recall / Precision / F1 / Latency
============================================================


  Model                           |   Recall |  Precis. |       F1 | Total(s) |   Avg(s)
  --------------------------------+----------+----------+----------+----------+---------
  Qwen-3B (template)              |    92%   |    44%   |    60%   |  306.5  |   21.9 
  Qwen-3B (3-stage)               |    93%   |    37%   |    53%   |  517.1  |   36.9 
  --------------------------------+----------+----------+----------+----------+---------

ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 849.11s

