   Compiling metal-candle v1.2.0 (/Users/garthdb/Projects/metal-candle)
    Finished `bench` profile [optimized] target(s) in 5.16s
     Running benches/inference.rs (target/release/deps/inference-daeecb1e1d277378)
Gnuplot not found, using plotters backend
Benchmarking kv_cache_update/single_layer_update
Benchmarking kv_cache_update/single_layer_update: Warming up for 3.0000 s
Benchmarking kv_cache_update/single_layer_update: Collecting 100 samples in estimated 5.0000 s (429M iterations)
Benchmarking kv_cache_update/single_layer_update: Analyzing
kv_cache_update/single_layer_update
                        time:   [11.586 ns 11.602 ns 11.620 ns]
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) low severe
  3 (3.00%) low mild
  2 (2.00%) high mild
Benchmarking kv_cache_update/all_layers_update
Benchmarking kv_cache_update/all_layers_update: Warming up for 3.0000 s
Benchmarking kv_cache_update/all_layers_update: Collecting 100 samples in estimated 5.0008 s (16M iterations)
Benchmarking kv_cache_update/all_layers_update: Analyzing
kv_cache_update/all_layers_update
                        time:   [318.95 ns 320.03 ns 321.18 ns]

Benchmarking kv_cache_retrieval/retrieve_from_cache
Benchmarking kv_cache_retrieval/retrieve_from_cache: Warming up for 3.0000 s
Benchmarking kv_cache_retrieval/retrieve_from_cache: Collecting 100 samples in estimated 5.0000 s (398M iterations)
Benchmarking kv_cache_retrieval/retrieve_from_cache: Analyzing
kv_cache_retrieval/retrieve_from_cache
                        time:   [12.551 ns 12.583 ns 12.616 ns]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) low mild

Benchmarking sampling_strategies/greedy
Benchmarking sampling_strategies/greedy: Warming up for 3.0000 s
Benchmarking sampling_strategies/greedy: Collecting 100 samples in estimated 5.1553 s (45k iterations)
Benchmarking sampling_strategies/greedy: Analyzing
sampling_strategies/greedy
                        time:   [109.25 µs 109.52 µs 109.87 µs]
                        thrpt:  [291.26 Melem/s 292.20 Melem/s 292.91 Melem/s]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild
Benchmarking sampling_strategies/top_k/k_10
Benchmarking sampling_strategies/top_k/k_10: Warming up for 3.0000 s
Benchmarking sampling_strategies/top_k/k_10: Collecting 100 samples in estimated 6.8675 s (15k iterations)
Benchmarking sampling_strategies/top_k/k_10: Analyzing
sampling_strategies/top_k/k_10
                        time:   [449.06 µs 450.09 µs 451.16 µs]
                        thrpt:  [70.929 Melem/s 71.097 Melem/s 71.259 Melem/s]
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild
  3 (3.00%) high severe
Benchmarking sampling_strategies/top_k/k_40
Benchmarking sampling_strategies/top_k/k_40: Warming up for 3.0000 s
Benchmarking sampling_strategies/top_k/k_40: Collecting 100 samples in estimated 6.8811 s (15k iterations)
Benchmarking sampling_strategies/top_k/k_40: Analyzing
sampling_strategies/top_k/k_40
                        time:   [453.39 µs 454.56 µs 455.78 µs]
                        thrpt:  [70.209 Melem/s 70.397 Melem/s 70.580 Melem/s]
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) low mild
  3 (3.00%) high mild
Benchmarking sampling_strategies/top_k/k_50
Benchmarking sampling_strategies/top_k/k_50: Warming up for 3.0000 s
Benchmarking sampling_strategies/top_k/k_50: Collecting 100 samples in estimated 6.9706 s (15k iterations)
Benchmarking sampling_strategies/top_k/k_50: Analyzing
sampling_strategies/top_k/k_50
                        time:   [466.57 µs 469.03 µs 471.12 µs]
                        thrpt:  [67.923 Melem/s 68.227 Melem/s 68.586 Melem/s]
Benchmarking sampling_strategies/top_p/p_90
Benchmarking sampling_strategies/top_p/p_90: Warming up for 3.0000 s
Benchmarking sampling_strategies/top_p/p_90: Collecting 100 samples in estimated 6.2389 s (10k iterations)
Benchmarking sampling_strategies/top_p/p_90: Analyzing
sampling_strategies/top_p/p_90
                        time:   [609.70 µs 611.09 µs 612.51 µs]
                        thrpt:  [52.244 Melem/s 52.366 Melem/s 52.484 Melem/s]
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) low severe
  1 (1.00%) low mild
  2 (2.00%) high mild
Benchmarking sampling_strategies/top_p/p_95
Benchmarking sampling_strategies/top_p/p_95: Warming up for 3.0000 s
Benchmarking sampling_strategies/top_p/p_95: Collecting 100 samples in estimated 6.3076 s (10k iterations)
Benchmarking sampling_strategies/top_p/p_95: Analyzing
sampling_strategies/top_p/p_95
                        time:   [620.69 µs 622.04 µs 623.50 µs]
                        thrpt:  [51.323 Melem/s 51.444 Melem/s 51.556 Melem/s]
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) low severe
  3 (3.00%) low mild
  7 (7.00%) high mild
Benchmarking sampling_strategies/temperature/t_7
Benchmarking sampling_strategies/temperature/t_7: Warming up for 3.0000 s
Benchmarking sampling_strategies/temperature/t_7: Collecting 100 samples in estimated 5.6032 s (25k iterations)
Benchmarking sampling_strategies/temperature/t_7: Analyzing
sampling_strategies/temperature/t_7
                        time:   [215.51 µs 217.75 µs 220.03 µs]
                        thrpt:  [145.44 Melem/s 146.96 Melem/s 148.48 Melem/s]
Found 12 outliers among 100 measurements (12.00%)
  12 (12.00%) low mild
Benchmarking sampling_strategies/temperature/t_8
Benchmarking sampling_strategies/temperature/t_8: Warming up for 3.0000 s
Benchmarking sampling_strategies/temperature/t_8: Collecting 100 samples in estimated 5.5734 s (25k iterations)
Benchmarking sampling_strategies/temperature/t_8: Analyzing
sampling_strategies/temperature/t_8
                        time:   [222.60 µs 223.63 µs 224.54 µs]
                        thrpt:  [142.51 Melem/s 143.09 Melem/s 143.76 Melem/s]
Found 12 outliers among 100 measurements (12.00%)
  4 (4.00%) low severe
  8 (8.00%) low mild
Benchmarking sampling_strategies/temperature/t_10
Benchmarking sampling_strategies/temperature/t_10: Warming up for 3.0000 s
Benchmarking sampling_strategies/temperature/t_10: Collecting 100 samples in estimated 5.6376 s (25k iterations)
Benchmarking sampling_strategies/temperature/t_10: Analyzing
sampling_strategies/temperature/t_10
                        time:   [191.50 µs 193.62 µs 196.34 µs]
                        thrpt:  [162.98 Melem/s 165.27 Melem/s 167.10 Melem/s]

Benchmarking kv_cache_scaling/full_sequence_seq_512
Benchmarking kv_cache_scaling/full_sequence_seq_512: Warming up for 3.0000 s
Benchmarking kv_cache_scaling/full_sequence_seq_512: Collecting 10 samples in estimated 5.5427 s (110 iterations)
Benchmarking kv_cache_scaling/full_sequence_seq_512: Analyzing
kv_cache_scaling/full_sequence_seq_512
                        time:   [50.372 ms 50.465 ms 50.546 ms]
Benchmarking kv_cache_scaling/full_sequence_seq_1024
Benchmarking kv_cache_scaling/full_sequence_seq_1024: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 8.7s or enable flat sampling.
Benchmarking kv_cache_scaling/full_sequence_seq_1024: Collecting 10 samples in estimated 8.7486 s (55 iterations)
Benchmarking kv_cache_scaling/full_sequence_seq_1024: Analyzing
kv_cache_scaling/full_sequence_seq_1024
                        time:   [159.28 ms 159.53 ms 159.74 ms]
Benchmarking kv_cache_scaling/full_sequence_seq_2048
Benchmarking kv_cache_scaling/full_sequence_seq_2048: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 5.5s.
Benchmarking kv_cache_scaling/full_sequence_seq_2048: Collecting 10 samples in estimated 5.5199 s (10 iterations)
Benchmarking kv_cache_scaling/full_sequence_seq_2048: Analyzing
kv_cache_scaling/full_sequence_seq_2048
                        time:   [557.07 ms 559.17 ms 561.33 ms]

Benchmarking sampling_vocab_scaling/greedy/vocab_1000
Benchmarking sampling_vocab_scaling/greedy/vocab_1000: Warming up for 3.0000 s
Benchmarking sampling_vocab_scaling/greedy/vocab_1000: Collecting 100 samples in estimated 5.1488 s (50k iterations)
Benchmarking sampling_vocab_scaling/greedy/vocab_1000: Analyzing
sampling_vocab_scaling/greedy/vocab_1000
                        time:   [100.15 µs 101.22 µs 102.20 µs]
                        thrpt:  [9.7847 Melem/s 9.8797 Melem/s 9.9853 Melem/s]
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) low mild
Benchmarking sampling_vocab_scaling/top_k_50/vocab_1000
Benchmarking sampling_vocab_scaling/top_k_50/vocab_1000: Warming up for 3.0000 s
Benchmarking sampling_vocab_scaling/top_k_50/vocab_1000: Collecting 100 samples in estimated 5.0339 s (45k iterations)
Benchmarking sampling_vocab_scaling/top_k_50/vocab_1000: Analyzing
sampling_vocab_scaling/top_k_50/vocab_1000
                        time:   [109.85 µs 110.98 µs 111.96 µs]
                        thrpt:  [8.9319 Melem/s 9.0106 Melem/s 9.1033 Melem/s]
Found 15 outliers among 100 measurements (15.00%)
  10 (10.00%) low severe
  5 (5.00%) low mild
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_1000
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_1000: Warming up for 3.0000 s
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_1000: Collecting 100 samples in estimated 5.2510 s (45k iterations)
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_1000: Analyzing
sampling_vocab_scaling/top_p_0.9/vocab_1000
                        time:   [113.61 µs 114.97 µs 116.14 µs]
                        thrpt:  [8.6100 Melem/s 8.6979 Melem/s 8.8023 Melem/s]
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) low mild
Benchmarking sampling_vocab_scaling/greedy/vocab_10000
Benchmarking sampling_vocab_scaling/greedy/vocab_10000: Warming up for 3.0000 s
Benchmarking sampling_vocab_scaling/greedy/vocab_10000: Collecting 100 samples in estimated 5.1187 s (45k iterations)
Benchmarking sampling_vocab_scaling/greedy/vocab_10000: Analyzing
sampling_vocab_scaling/greedy/vocab_10000
                        time:   [111.38 µs 112.04 µs 112.54 µs]
                        thrpt:  [88.859 Melem/s 89.254 Melem/s 89.782 Melem/s]
Found 17 outliers among 100 measurements (17.00%)
  13 (13.00%) low severe
  3 (3.00%) low mild
  1 (1.00%) high severe
Benchmarking sampling_vocab_scaling/top_k_50/vocab_10000
Benchmarking sampling_vocab_scaling/top_k_50/vocab_10000: Warming up for 3.0000 s
Benchmarking sampling_vocab_scaling/top_k_50/vocab_10000: Collecting 100 samples in estimated 5.6855 s (25k iterations)
Benchmarking sampling_vocab_scaling/top_k_50/vocab_10000: Analyzing
sampling_vocab_scaling/top_k_50/vocab_10000
                        time:   [218.51 µs 223.58 µs 228.50 µs]
                        thrpt:  [43.763 Melem/s 44.728 Melem/s 45.765 Melem/s]
Found 11 outliers among 100 measurements (11.00%)
  7 (7.00%) low severe
  4 (4.00%) low mild
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_10000
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_10000: Warming up for 3.0000 s
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_10000: Collecting 100 samples in estimated 5.6401 s (20k iterations)
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_10000: Analyzing
sampling_vocab_scaling/top_p_0.9/vocab_10000
                        time:   [277.68 µs 278.87 µs 279.86 µs]
                        thrpt:  [35.732 Melem/s 35.859 Melem/s 36.013 Melem/s]
Found 13 outliers among 100 measurements (13.00%)
  3 (3.00%) low severe
  8 (8.00%) low mild
  2 (2.00%) high mild
Benchmarking sampling_vocab_scaling/greedy/vocab_32000
Benchmarking sampling_vocab_scaling/greedy/vocab_32000: Warming up for 3.0000 s
Benchmarking sampling_vocab_scaling/greedy/vocab_32000: Collecting 100 samples in estimated 5.0165 s (40k iterations)
Benchmarking sampling_vocab_scaling/greedy/vocab_32000: Analyzing
sampling_vocab_scaling/greedy/vocab_32000
                        time:   [123.66 µs 124.24 µs 124.71 µs]
                        thrpt:  [256.59 Melem/s 257.58 Melem/s 258.77 Melem/s]
Found 14 outliers among 100 measurements (14.00%)
  10 (10.00%) low severe
  2 (2.00%) low mild
  1 (1.00%) high mild
  1 (1.00%) high severe
Benchmarking sampling_vocab_scaling/top_k_50/vocab_32000
Benchmarking sampling_vocab_scaling/top_k_50/vocab_32000: Warming up for 3.0000 s
Benchmarking sampling_vocab_scaling/top_k_50/vocab_32000: Collecting 100 samples in estimated 5.0206 s (10k iterations)
Benchmarking sampling_vocab_scaling/top_k_50/vocab_32000: Analyzing
sampling_vocab_scaling/top_k_50/vocab_32000
                        time:   [493.71 µs 494.57 µs 495.40 µs]
                        thrpt:  [64.594 Melem/s 64.703 Melem/s 64.816 Melem/s]
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) low severe
  5 (5.00%) low mild
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_32000
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_32000: Warming up for 3.0000 s
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_32000: Collecting 100 samples in estimated 6.4696 s (10k iterations)
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_32000: Analyzing
sampling_vocab_scaling/top_p_0.9/vocab_32000
                        time:   [627.02 µs 629.65 µs 632.27 µs]
                        thrpt:  [50.611 Melem/s 50.822 Melem/s 51.035 Melem/s]
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) low mild
Benchmarking sampling_vocab_scaling/greedy/vocab_100000
Benchmarking sampling_vocab_scaling/greedy/vocab_100000: Warming up for 3.0000 s
Benchmarking sampling_vocab_scaling/greedy/vocab_100000: Collecting 100 samples in estimated 5.6886 s (35k iterations)
Benchmarking sampling_vocab_scaling/greedy/vocab_100000: Analyzing
sampling_vocab_scaling/greedy/vocab_100000
                        time:   [160.00 µs 160.91 µs 161.75 µs]
                        thrpt:  [618.25 Melem/s 621.48 Melem/s 625.01 Melem/s]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) low mild
Benchmarking sampling_vocab_scaling/top_k_50/vocab_100000
Benchmarking sampling_vocab_scaling/top_k_50/vocab_100000: Warming up for 3.0000 s

Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.8s, enable flat sampling, or reduce sample count to 50.
Benchmarking sampling_vocab_scaling/top_k_50/vocab_100000: Collecting 100 samples in estimated 7.7964 s (5050 iterations)
Benchmarking sampling_vocab_scaling/top_k_50/vocab_100000: Analyzing
sampling_vocab_scaling/top_k_50/vocab_100000
                        time:   [1.4981 ms 1.5032 ms 1.5095 ms]
                        thrpt:  [66.249 Melem/s 66.525 Melem/s 66.753 Melem/s]
Found 8 outliers among 100 measurements (8.00%)
  4 (4.00%) high mild
  4 (4.00%) high severe
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_100000
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_100000: Warming up for 3.0000 s
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_100000: Collecting 100 samples in estimated 5.1204 s (2300 iterations)
Benchmarking sampling_vocab_scaling/top_p_0.9/vocab_100000: Analyzing
sampling_vocab_scaling/top_p_0.9/vocab_100000
                        time:   [2.2042 ms 2.2090 ms 2.2135 ms]
                        thrpt:  [45.178 Melem/s 45.269 Melem/s 45.368 Melem/s]
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) low severe
  1 (1.00%) low mild
  1 (1.00%) high mild

Benchmarking kv_cache_memory/allocate_qwen_0.5B
Benchmarking kv_cache_memory/allocate_qwen_0.5B: Warming up for 3.0000 s
Benchmarking kv_cache_memory/allocate_qwen_0.5B: Collecting 10 samples in estimated 5.0000 s (242M iterations)
Benchmarking kv_cache_memory/allocate_qwen_0.5B: Analyzing
kv_cache_memory/allocate_qwen_0.5B
                        time:   [20.672 ns 20.706 ns 20.764 ns]
Found 2 outliers among 10 measurements (20.00%)
  1 (10.00%) high mild
  1 (10.00%) high severe
Benchmarking kv_cache_memory/allocate_qwen_1.5B
Benchmarking kv_cache_memory/allocate_qwen_1.5B: Warming up for 3.0000 s
Benchmarking kv_cache_memory/allocate_qwen_1.5B: Collecting 10 samples in estimated 5.0000 s (242M iterations)
Benchmarking kv_cache_memory/allocate_qwen_1.5B: Analyzing
kv_cache_memory/allocate_qwen_1.5B
                        time:   [20.659 ns 20.676 ns 20.699 ns]
Benchmarking kv_cache_memory/allocate_qwen_3B
Benchmarking kv_cache_memory/allocate_qwen_3B: Warming up for 3.0000 s
Benchmarking kv_cache_memory/allocate_qwen_3B: Collecting 10 samples in estimated 5.0000 s (240M iterations)
Benchmarking kv_cache_memory/allocate_qwen_3B: Analyzing
kv_cache_memory/allocate_qwen_3B
                        time:   [20.662 ns 20.672 ns 20.690 ns]
Found 2 outliers among 10 measurements (20.00%)
  1 (10.00%) low mild
  1 (10.00%) high mild

Benchmarking generation_simulation/token_generation_cycle
Benchmarking generation_simulation/token_generation_cycle: Warming up for 3.0000 s
Benchmarking generation_simulation/token_generation_cycle: Collecting 10 samples in estimated 5.0195 s (7975 iterations)
Benchmarking generation_simulation/token_generation_cycle: Analyzing
generation_simulation/token_generation_cycle
                        time:   [623.95 µs 625.30 µs 626.27 µs]

