   Compiling metal-candle v1.2.0 (/Users/garthdb/Projects/metal-candle)
    Finished `bench` profile [optimized] target(s) in 11.84s
     Running benches/training.rs (target/release/deps/training-893f96c187521b93)
Gnuplot not found, using plotters backend
Benchmarking lora_forward/forward/small_512x512_r8
Benchmarking lora_forward/forward/small_512x512_r8: Warming up for 3.0000 s
Benchmarking lora_forward/forward/small_512x512_r8: Collecting 100 samples in estimated 5.1040 s (15k iterations)
Benchmarking lora_forward/forward/small_512x512_r8: Analyzing
lora_forward/forward/small_512x512_r8
                        time:   [410.63 µs 418.90 µs 427.17 µs]
                        thrpt:  [76.709 Melem/s 78.224 Melem/s 79.800 Melem/s]
                 change:
                        time:   [-10.369% +0.1651% +12.397%] (p = 0.97 > 0.05)
                        thrpt:  [-11.029% -0.1648% +11.569%]
                        No change in performance detected.
Found 14 outliers among 100 measurements (14.00%)
  8 (8.00%) low severe
  1 (1.00%) low mild
  2 (2.00%) high mild
  3 (3.00%) high severe
Benchmarking lora_forward/forward/medium_1024x1024_r8
Benchmarking lora_forward/forward/medium_1024x1024_r8: Warming up for 3.0000 s
Benchmarking lora_forward/forward/medium_1024x1024_r8: Collecting 100 samples in estimated 8.1168 s (10k iterations)
Benchmarking lora_forward/forward/medium_1024x1024_r8: Analyzing
lora_forward/forward/medium_1024x1024_r8
                        time:   [2.4107 ms 2.5021 ms 2.5928 ms]
                        thrpt:  [25.276 Melem/s 26.192 Melem/s 27.186 Melem/s]
                 change:
                        time:   [-19.878% -0.1016% +27.033%] (p = 1.00 > 0.05)
                        thrpt:  [-21.281% +0.1017% +24.810%]
                        No change in performance detected.
Found 18 outliers among 100 measurements (18.00%)
  12 (12.00%) low mild
  3 (3.00%) high mild
  3 (3.00%) high severe
Benchmarking lora_forward/forward/large_2048x2048_r8
Benchmarking lora_forward/forward/large_2048x2048_r8: Warming up for 3.0000 s
Benchmarking lora_forward/forward/large_2048x2048_r8: Collecting 100 samples in estimated 5.1830 s (2100 iterations)
Benchmarking lora_forward/forward/large_2048x2048_r8: Analyzing
lora_forward/forward/large_2048x2048_r8
                        time:   [4.0160 ms 5.6738 ms 7.4428 ms]
                        thrpt:  [17.611 Melem/s 23.101 Melem/s 32.638 Melem/s]
                 change:
                        time:   [-34.310% +0.2899% +53.812%] (p = 0.98 > 0.05)
                        thrpt:  [-34.985% -0.2890% +52.231%]
                        No change in performance detected.
Found 18 outliers among 100 measurements (18.00%)
  18 (18.00%) high mild
Benchmarking lora_forward/forward/small_512x512_r16
Benchmarking lora_forward/forward/small_512x512_r16: Warming up for 3.0000 s
Benchmarking lora_forward/forward/small_512x512_r16: Collecting 100 samples in estimated 5.6716 s (600 iterations)
Benchmarking lora_forward/forward/small_512x512_r16: Analyzing
lora_forward/forward/small_512x512_r16
                        time:   [5.0148 ms 9.9908 ms 15.794 ms]
                        thrpt:  [2.0747 Melem/s 3.2798 Melem/s 6.5343 Melem/s]
                 change:
                        time:   [-54.057% +0.7852% +126.07%] (p = 0.91 > 0.05)
                        thrpt:  [-55.766% -0.7790% +117.66%]
                        No change in performance detected.
Found 15 outliers among 100 measurements (15.00%)
  3 (3.00%) high mild
  12 (12.00%) high severe

Benchmarking gradient_computation/forward_backward
Benchmarking gradient_computation/forward_backward: Warming up for 3.0000 s
Benchmarking gradient_computation/forward_backward: Collecting 100 samples in estimated 5.4647 s (1000 iterations)
Benchmarking gradient_computation/forward_backward: Analyzing
gradient_computation/forward_backward
                        time:   [5.0831 ms 5.8423 ms 6.8714 ms]
                        change: [-24.596% -4.2139% +22.398%] (p = 0.75 > 0.05)
                        No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
  4 (4.00%) low severe
  1 (1.00%) high mild
  6 (6.00%) high severe

Benchmarking optimizer_step/adamw_step
Benchmarking optimizer_step/adamw_step: Warming up for 3.0000 s

Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 10.0s, enable flat sampling, or reduce sample count to 40.
Benchmarking optimizer_step/adamw_step: Collecting 100 samples in estimated 9.9509 s (5050 iterations)
Benchmarking optimizer_step/adamw_step: Analyzing
optimizer_step/adamw_step
                        time:   [880.35 µs 1.0608 ms 1.3099 ms]
                        change: [-53.119% -21.245% +26.767%] (p = 0.49 > 0.05)
                        No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe

Benchmarking full_training_step/complete_iteration
Benchmarking full_training_step/complete_iteration: Warming up for 3.0000 s
Benchmarking full_training_step/complete_iteration: Collecting 10 samples in estimated 5.0250 s (1485 iterations)
Benchmarking full_training_step/complete_iteration: Analyzing
full_training_step/complete_iteration
                        time:   [3.3108 ms 3.3154 ms 3.3191 ms]
                        thrpt:  [39.491 Melem/s 39.535 Melem/s 39.590 Melem/s]
                 change:
                        time:   [-0.6521% -0.5061% -0.3303%] (p = 0.00 < 0.05)
                        thrpt:  [+0.3314% +0.5087% +0.6564%]
                        Change within noise threshold.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild

Benchmarking layer_operations/softmax_stable
Benchmarking layer_operations/softmax_stable: Warming up for 3.0000 s
Benchmarking layer_operations/softmax_stable: Collecting 100 samples in estimated 5.0620 s (116k iterations)
Benchmarking layer_operations/softmax_stable: Analyzing
layer_operations/softmax_stable
                        time:   [43.504 µs 43.647 µs 43.764 µs]
                        change: [-3.3105% -2.4373% -1.5715%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 12 outliers among 100 measurements (12.00%)
  10 (10.00%) low severe
  2 (2.00%) low mild
Benchmarking layer_operations/layer_norm
Benchmarking layer_operations/layer_norm: Warming up for 3.0000 s
Benchmarking layer_operations/layer_norm: Collecting 100 samples in estimated 5.1270 s (111k iterations)
Benchmarking layer_operations/layer_norm: Analyzing
layer_operations/layer_norm
                        time:   [47.519 µs 47.816 µs 48.047 µs]
                        change: [-2.2916% -1.5659% -0.9830%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 15 outliers among 100 measurements (15.00%)
  12 (12.00%) low severe
  3 (3.00%) low mild
Benchmarking layer_operations/rms_norm
Benchmarking layer_operations/rms_norm: Warming up for 3.0000 s
Benchmarking layer_operations/rms_norm: Collecting 100 samples in estimated 5.1182 s (202k iterations)
Benchmarking layer_operations/rms_norm: Analyzing
layer_operations/rms_norm
                        time:   [25.348 µs 25.472 µs 25.575 µs]
                        change: [-0.0280% +0.5924% +1.2612%] (p = 0.07 > 0.05)
                        No change in performance detected.
Found 15 outliers among 100 measurements (15.00%)
  8 (8.00%) low severe
  6 (6.00%) low mild
  1 (1.00%) high mild

Benchmarking lora_rank_scaling/forward/rank_4
Benchmarking lora_rank_scaling/forward/rank_4: Warming up for 3.0000 s
Benchmarking lora_rank_scaling/forward/rank_4: Collecting 100 samples in estimated 7.5013 s (10k iterations)
Benchmarking lora_rank_scaling/forward/rank_4: Analyzing
lora_rank_scaling/forward/rank_4
                        time:   [1.1825 ms 1.2262 ms 1.2694 ms]
                        change: [-17.614% +0.0147% +21.302%] (p = 1.00 > 0.05)
                        No change in performance detected.
Found 18 outliers among 100 measurements (18.00%)
  12 (12.00%) low mild
  3 (3.00%) high mild
  3 (3.00%) high severe
Benchmarking lora_rank_scaling/forward/rank_8
Benchmarking lora_rank_scaling/forward/rank_8: Warming up for 3.0000 s

Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.4s, enable flat sampling, or reduce sample count to 50.
Benchmarking lora_rank_scaling/forward/rank_8: Collecting 100 samples in estimated 7.4208 s (5050 iterations)
Benchmarking lora_rank_scaling/forward/rank_8: Analyzing
lora_rank_scaling/forward/rank_8
                        time:   [2.4094 ms 2.5872 ms 2.7587 ms]
                        change: [-18.151% +3.2500% +31.664%] (p = 0.80 > 0.05)
                        No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe
Benchmarking lora_rank_scaling/forward/rank_16
Benchmarking lora_rank_scaling/forward/rank_16: Warming up for 3.0000 s
Benchmarking lora_rank_scaling/forward/rank_16: Collecting 100 samples in estimated 5.1082 s (2000 iterations)
Benchmarking lora_rank_scaling/forward/rank_16: Analyzing
lora_rank_scaling/forward/rank_16
                        time:   [2.3205 ms 3.0785 ms 3.8547 ms]
                        change: [-41.080% -17.015% +17.637%] (p = 0.31 > 0.05)
                        No change in performance detected.
Benchmarking lora_rank_scaling/forward/rank_32
Benchmarking lora_rank_scaling/forward/rank_32: Warming up for 3.0000 s
Benchmarking lora_rank_scaling/forward/rank_32: Collecting 100 samples in estimated 5.1975 s (1400 iterations)
Benchmarking lora_rank_scaling/forward/rank_32: Analyzing
lora_rank_scaling/forward/rank_32
                        time:   [2.6838 ms 3.9116 ms 5.1542 ms]
                        change: [-55.429% -26.440% +25.962%] (p = 0.26 > 0.05)
                        No change in performance detected.
Benchmarking lora_rank_scaling/forward/rank_64
Benchmarking lora_rank_scaling/forward/rank_64: Warming up for 3.0000 s
Benchmarking lora_rank_scaling/forward/rank_64: Collecting 100 samples in estimated 5.1115 s (1100 iterations)
Benchmarking lora_rank_scaling/forward/rank_64: Analyzing
lora_rank_scaling/forward/rank_64
                        time:   [4.7030 ms 7.5506 ms 10.433 ms]
                        change: [-55.653% -22.080% +38.773%] (p = 0.39 > 0.05)
                        No change in performance detected.
Found 21 outliers among 100 measurements (21.00%)
  21 (21.00%) high severe

Benchmarking lazy_vs_eager_lora/eager_forward
Benchmarking lazy_vs_eager_lora/eager_forward: Warming up for 3.0000 s

Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.0s, enable flat sampling, or reduce sample count to 60.
Benchmarking lazy_vs_eager_lora/eager_forward: Collecting 100 samples in estimated 6.0141 s (5050 iterations)
Benchmarking lazy_vs_eager_lora/eager_forward: Analyzing
lazy_vs_eager_lora/eager_forward
                        time:   [1.8866 ms 2.1685 ms 2.5731 ms]
                        change: [-34.753% -5.9504% +37.330%] (p = 0.77 > 0.05)
                        No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe
Benchmarking lazy_vs_eager_lora/lazy_forward_sync
Benchmarking lazy_vs_eager_lora/lazy_forward_sync: Warming up for 3.0000 s
Benchmarking lazy_vs_eager_lora/lazy_forward_sync: Collecting 100 samples in estimated 5.0116 s (2100 iterations)
Benchmarking lazy_vs_eager_lora/lazy_forward_sync: Analyzing
lazy_vs_eager_lora/lazy_forward_sync
                        time:   [1.7358 ms 2.4556 ms 3.3506 ms]
                        change: [-20.510% +23.703% +96.742%] (p = 0.38 > 0.05)
                        No change in performance detected.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

