    Finished `release` profile [optimized] target(s) in 2.38s
     Running tests/model_comparison.rs (/Users/jamessheen/.cargo-shared-target/kaironiq/release/deps/model_comparison-1238d4cb13964449)

running 1 test
test model_comparison_tests::test_model_comparison ... 
============================================================
  MODEL COMPARISON — Entity Recall Benchmark
============================================================

  SKIP: NuExtract-4B (template) (RQL_NUEXTRACT_MODEL_PATH not set)

  SKIP: NuExtract-4B (grounded) (RQL_NUEXTRACT_MODEL_PATH not set)

  SKIP: Phi-4-mini (3-stage) (RQL_PHI4_MODEL_PATH not set)

  SKIP: Phi-4-mini (3-stage-nogrm) (RQL_PHI4_MODEL_PATH not set)

  SKIP: Qwen-3B (template) (RQL_QWEN3B_MODEL_PATH not set)

  SKIP: Qwen-3B (3-stage) (RQL_QWEN3B_MODEL_PATH not set)

  Loading: Gemma-4-E2B (3-stage) (/Users/jamessheen/Documents/Projects/Ideas/ai-meeting-transcription/models/gemma-4-E2B-it-Q4_K_M.gguf)
  Running fixtures for: Gemma-4-E2B (3-stage)
    Mock Interview — R:90% P:53% F1:67% (9/10 found, 17 extracted) in 26.5s  miss=["ziad"]
    Tech Standup — R:100% P:42% F1:59% (13/13 found, 31 extracted) in 51.2s  miss=[]
warn: skipping fact (Priya Sharma -> works for -> Marine Biology Institute at Stanford University): database error: SQLite failure: `FOREIGN KEY constraint failed`
    Podcast Interview — R:100% P:31% F1:47% (15/15 found, 49 extracted) in 54.3s  miss=[]
    Sales Call — R:100% P:32% F1:48% (12/12 found, 38 extracted) in 51.0s  miss=[]
    Legal Deposition — R:88% P:24% F1:37% (14/16 found, 59 extracted) in 48.0s  miss=["clifford reeves", "february 14, 2024"]
    Medical Consultation — R:93% P:43% F1:59% (13/14 found, 30 extracted) in 46.1s  miss=["st. catherine's general hospital"]
warn: skipping fact (Ferdinand de Saussure -> contributed_to -> foundational_claim_in): database error: SQLite failure: `FOREIGN KEY constraint failed`
warn: skipping fact (foundational_claim_in -> is_in -> Course in General Linguistics): database error: SQLite failure: `FOREIGN KEY constraint failed`
    Academic Lecture — R:95% P:54% F1:69% (21/22 found, 39 extracted) in 36.9s  miss=["mit"]
    Customer Support — R:100% P:28% F1:44% (9/9 found, 32 extracted) in 43.0s  miss=[]
    Board Meeting — R:94% P:23% F1:37% (16/17 found, 69 extracted) in 59.1s  miss=["niall o'brien"]
warn: skipping fact (Meridian Systems (Organisation) -> sponsors -> Meridian Cloud Services (Organisation)): database error: SQLite failure: `FOREIGN KEY constraint failed`
    News Article — R:100% P:41% F1:58% (16/16 found, 39 extracted) in 53.7s  miss=[]
    Slack Thread — R:100% P:50% F1:67% (12/12 found, 24 extracted) in 51.7s  miss=[]
warn: skipping fact (Zephyr Audio Connect app -> has_feature -> clunky equaliser UI): database error: SQLite failure: `FOREIGN KEY constraint failed`
warn: skipping fact (Sony Headphones Connect app -> handles_feature -> automatic volume levelling feature): database error: SQLite failure: `FOREIGN KEY constraint failed`
warn: skipping fact (Zephyr Audio ZA-900 Pro -> has_feature -> ANC): database error: SQLite failure: `FOREIGN KEY constraint failed`
    Product Review — R:80% P:40% F1:53% (8/10 found, 20 extracted) in 31.8s  miss=["za-800", "active noise cancellation"]
    Short Snippet — R:100% P:44% F1:62% (4/4 found, 9 extracted) in 13.7s  miss=[]
warn: skipping fact (NEXAGEN CORPORATION -> delivered revenues -> $312.4 million in Q3 FY2024): database error: SQLite failure: `FOREIGN KEY constraint failed`
    Long Report — R:95% P:40% F1:56% (36/38 found, 90 extracted) in 56.6s  miss=["münchener rückversicherungs-gesellschaft", "columbus, ohio"]

  Loading: Gemma-4-E2B (3-stage-nogrm) (/Users/jamessheen/Documents/Projects/Ideas/ai-meeting-transcription/models/gemma-4-E2B-it-Q4_K_M.gguf)
  Running fixtures for: Gemma-4-E2B (3-stage-nogrm)
    Mock Interview — R:90% P:53% F1:67% (9/10 found, 17 extracted) in 22.3s  miss=["ziad"]
warn: skipping fact (Sarah -> works_on -> Jira issue): database error: SQLite failure: `FOREIGN KEY constraint failed`
    Tech Standup — R:100% P:42% F1:59% (13/13 found, 31 extracted) in 39.0s  miss=[]
warn: skipping fact (Priya Sharma -> works for -> Marine Biology Institute at Stanford University): database error: SQLite failure: `FOREIGN KEY constraint failed`
    Podcast Interview — R:100% P:31% F1:47% (15/15 found, 49 extracted) in 24.1s  miss=[]
    Sales Call — R:100% P:32% F1:48% (12/12 found, 38 extracted) in 21.1s  miss=[]
    Legal Deposition — R:88% P:24% F1:37% (14/16 found, 59 extracted) in 23.5s  miss=["clifford reeves", "february 14, 2024"]
    Medical Consultation — R:93% P:43% F1:59% (13/14 found, 30 extracted) in 21.1s  miss=["st. catherine's general hospital"]
warn: skipping fact (Ferdinand de Saussure -> contributed_to -> foundational_claim_in): database error: SQLite failure: `FOREIGN KEY constraint failed`
warn: skipping fact (foundational_claim_in -> is_in -> Course in General Linguistics): database error: SQLite failure: `FOREIGN KEY constraint failed`
    Academic Lecture — R:95% P:54% F1:69% (21/22 found, 39 extracted) in 21.0s  miss=["mit"]
    Customer Support — R:100% P:28% F1:44% (9/9 found, 32 extracted) in 23.1s  miss=[]
    Board Meeting — R:94% P:23% F1:37% (16/17 found, 69 extracted) in 25.8s  miss=["niall o'brien"]
warn: skipping fact (Meridian Systems (Organisation) -> sponsors -> Meridian Cloud Services (Organisation)): database error: SQLite failure: `FOREIGN KEY constraint failed`
    News Article — R:100% P:41% F1:58% (16/16 found, 39 extracted) in 23.7s  miss=[]
    Slack Thread — R:100% P:50% F1:67% (12/12 found, 24 extracted) in 23.2s  miss=[]
warn: skipping fact (Zephyr Audio Connect app -> has_feature -> clunky equaliser UI): database error: SQLite failure: `FOREIGN KEY constraint failed`
warn: skipping fact (Sony Headphones Connect app -> handles_feature -> automatic volume levelling feature): database error: SQLite failure: `FOREIGN KEY constraint failed`
    Product Review — R:80% P:40% F1:53% (8/10 found, 20 extracted) in 22.3s  miss=["za-800", "active noise cancellation"]
warn: skipping fact (appointment -> announced_by -> James Whitfield): database error: SQLite failure: `FOREIGN KEY constraint failed`
    Short Snippet — R:100% P:44% F1:62% (4/4 found, 9 extracted) in 12.4s  miss=[]
warn: skipping fact (NEXAGEN CORPORATION -> delivered revenues -> $312.4 million in Q3 FY2024): database error: SQLite failure: `FOREIGN KEY constraint failed`
    Long Report — R:95% P:40% F1:56% (36/38 found, 90 extracted) in 56.1s  miss=["münchener rückversicherungs-gesellschaft", "columbus, ohio"]

  Loading: Gemma-4-E2B (template) (/Users/jamessheen/Documents/Projects/Ideas/ai-meeting-transcription/models/gemma-4-E2B-it-Q4_K_M.gguf)
  Running fixtures for: Gemma-4-E2B (template)
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Mock Interview — R:90% P:64% F1:75% (9/10 found, 14 extracted) in 15.2s  miss=["ziad"]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Tech Standup — R:100% P:52% F1:68% (13/13 found, 25 extracted) in 15.9s  miss=[]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Podcast Interview — R:100% P:43% F1:60% (15/15 found, 35 extracted) in 15.5s  miss=[]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Sales Call — R:100% P:43% F1:60% (12/12 found, 28 extracted) in 17.8s  miss=[]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Legal Deposition — R:88% P:32% F1:47% (14/16 found, 44 extracted) in 15.5s  miss=["clifford reeves", "february 14, 2024"]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Medical Consultation — R:71% P:45% F1:56% (10/14 found, 22 extracted) in 16.7s  miss=["dr. patel", "st. catherine's general hospital", "hypertrophic cardiomyopathy", "type 2 diabetes"]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Academic Lecture — R:95% P:55% F1:70% (21/22 found, 38 extracted) in 16.5s  miss=["mit"]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Customer Support — R:100% P:38% F1:55% (9/9 found, 24 extracted) in 16.2s  miss=[]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Board Meeting — R:88% P:27% F1:42% (15/17 found, 55 extracted) in 17.0s  miss=["niall o'brien", "verdant capital partners ltd."]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    News Article — R:94% P:58% F1:71% (15/16 found, 26 extracted) in 14.2s  miss=["washington d.c."]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Slack Thread — R:83% P:71% F1:77% (10/12 found, 14 extracted) in 14.7s  miss=["pagerduty", "grafana"]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Product Review — R:80% P:57% F1:67% (8/10 found, 14 extracted) in 15.7s  miss=["za-800", "active noise cancellation"]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Short Snippet — R:100% P:67% F1:80% (4/4 found, 6 extracted) in 5.7s  miss=[]
warn: failed to parse NuExtract JSON after repair: invalid type: sequence, expected a string
    Long Report — R:95% P:40% F1:57% (36/38 found, 89 extracted) in 17.0s  miss=["münchener rückversicherungs-gesellschaft", "columbus, ohio"]

============================================================
  MODEL COMPARISON — Recall / Precision / F1 / Latency
============================================================


  Model                           |   Recall |  Precis. |       F1 | Total(s) |   Avg(s)
  --------------------------------+----------+----------+----------+----------+---------
  Gemma-4-E2B (3-stage)           |    95%   |    36%   |    53%   |  623.8  |   44.6 
  Gemma-4-E2B (3-stage-nogrm)     |    95%   |    36%   |    53%   |  358.5  |   25.6 
  Gemma-4-E2B (template)          |    92%   |    44%   |    60%   |  213.5  |   15.3 
  --------------------------------+----------+----------+----------+----------+---------

ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1199.43s

-

ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1318.53s

