    Finished `bench` profile [optimized] target(s) in 0.54s
     Running benches/training.rs (target/release/deps/training-b91a46af1471bb99)
Gnuplot not found, using plotters backend
Benchmarking lora_forward/forward/small_512x512_r8
Benchmarking lora_forward/forward/small_512x512_r8: Warming up for 3.0000 s
Benchmarking lora_forward/forward/small_512x512_r8: Collecting 100 samples in estimated 5.1041 s (15k iterations)
Benchmarking lora_forward/forward/small_512x512_r8: Analyzing
lora_forward/forward/small_512x512_r8
                        time:   [412.10 µs 420.61 µs 428.92 µs]
                        thrpt:  [76.397 Melem/s 77.906 Melem/s 79.515 Melem/s]
                 change:
                        time:   [-11.044% +0.1809% +11.618%] (p = 0.96 > 0.05)
                        thrpt:  [-10.409% -0.1806% +12.415%]
                        No change in performance detected.
Found 14 outliers among 100 measurements (14.00%)
  8 (8.00%) low severe
  1 (1.00%) low mild
  2 (2.00%) high mild
  3 (3.00%) high severe
Benchmarking lora_forward/forward/medium_1024x1024_r8
Benchmarking lora_forward/forward/medium_1024x1024_r8: Warming up for 3.0000 s
Benchmarking lora_forward/forward/medium_1024x1024_r8: Collecting 100 samples in estimated 8.1684 s (10k iterations)
Benchmarking lora_forward/forward/medium_1024x1024_r8: Analyzing
lora_forward/forward/medium_1024x1024_r8
                        time:   [2.4109 ms 2.5020 ms 2.5923 ms]
                        thrpt:  [25.281 Melem/s 26.193 Melem/s 27.183 Melem/s]
                 change:
                        time:   [-21.050% +0.0166% +25.450%] (p = 1.00 > 0.05)
                        thrpt:  [-20.287% -0.0166% +26.662%]
                        No change in performance detected.
Found 18 outliers among 100 measurements (18.00%)
  12 (12.00%) low mild
  3 (3.00%) high mild
  3 (3.00%) high severe
Benchmarking lora_forward/forward/large_2048x2048_r8
Benchmarking lora_forward/forward/large_2048x2048_r8: Warming up for 3.0000 s
Benchmarking lora_forward/forward/large_2048x2048_r8: Collecting 100 samples in estimated 5.2087 s (2100 iterations)
Benchmarking lora_forward/forward/large_2048x2048_r8: Analyzing
lora_forward/forward/large_2048x2048_r8
                        time:   [4.0064 ms 5.6574 ms 7.4229 ms]
                        thrpt:  [17.658 Melem/s 23.168 Melem/s 32.715 Melem/s]
                 change:
                        time:   [-34.328% -0.0170% +53.236%] (p = 1.00 > 0.05)
                        thrpt:  [-34.741% +0.0170% +52.271%]
                        No change in performance detected.
Found 18 outliers among 100 measurements (18.00%)
  18 (18.00%) high mild
Benchmarking lora_forward/forward/small_512x512_r16
Benchmarking lora_forward/forward/small_512x512_r16: Warming up for 3.0000 s
Benchmarking lora_forward/forward/small_512x512_r16: Collecting 100 samples in estimated 5.6462 s (600 iterations)
Benchmarking lora_forward/forward/small_512x512_r16: Analyzing
lora_forward/forward/small_512x512_r16
                        time:   [4.9762 ms 9.9129 ms 15.672 ms]
                        thrpt:  [2.0909 Melem/s 3.3056 Melem/s 6.5850 Melem/s]
                 change:
                        time:   [-56.255% -0.3913% +120.51%] (p = 0.92 > 0.05)
                        thrpt:  [-54.651% +0.3928% +128.60%]
                        No change in performance detected.
Found 15 outliers among 100 measurements (15.00%)
  3 (3.00%) high mild
  12 (12.00%) high severe

Benchmarking gradient_computation/forward_backward
Benchmarking gradient_computation/forward_backward: Warming up for 3.0000 s
Benchmarking gradient_computation/forward_backward: Collecting 100 samples in estimated 5.0760 s (1000 iterations)
Benchmarking gradient_computation/forward_backward: Analyzing
gradient_computation/forward_backward
                        time:   [5.1167 ms 5.1229 ms 5.1290 ms]
                        change: [-20.402% -7.4352% +0.8254%] (p = 0.62 > 0.05)
                        No change in performance detected.

Benchmarking optimizer_step/adamw_step
Benchmarking optimizer_step/adamw_step: Warming up for 3.0000 s

Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 8.9s, enable flat sampling, or reduce sample count to 50.
Benchmarking optimizer_step/adamw_step: Collecting 100 samples in estimated 8.9204 s (5050 iterations)
Benchmarking optimizer_step/adamw_step: Analyzing
optimizer_step/adamw_step
                        time:   [1.1725 ms 1.3587 ms 1.5646 ms]
                        change: [-3.3027% +17.356% +48.874%] (p = 0.20 > 0.05)
                        No change in performance detected.
Found 22 outliers among 100 measurements (22.00%)
  19 (19.00%) low severe
  3 (3.00%) high severe

Benchmarking full_training_step/complete_iteration
Benchmarking full_training_step/complete_iteration: Warming up for 3.0000 s
Benchmarking full_training_step/complete_iteration: Collecting 10 samples in estimated 5.0189 s (1485 iterations)
Benchmarking full_training_step/complete_iteration: Analyzing
full_training_step/complete_iteration
                        time:   [3.0845 ms 3.0868 ms 3.0886 ms]
                        thrpt:  [42.438 Melem/s 42.461 Melem/s 42.494 Melem/s]
                 change:
                        time:   [-0.1885% -0.1216% -0.0523%] (p = 0.00 < 0.05)
                        thrpt:  [+0.0523% +0.1217% +0.1889%]
                        Change within noise threshold.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) low mild

Benchmarking layer_operations/softmax_stable
Benchmarking layer_operations/softmax_stable: Warming up for 3.0000 s
Benchmarking layer_operations/softmax_stable: Collecting 100 samples in estimated 5.2113 s (116k iterations)
Benchmarking layer_operations/softmax_stable: Analyzing
layer_operations/softmax_stable
                        time:   [44.347 µs 44.722 µs 45.023 µs]
                        change: [-0.9802% -0.2167% +0.5719%] (p = 0.59 > 0.05)
                        No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
  8 (8.00%) low severe
  1 (1.00%) low mild
  2 (2.00%) high mild
Benchmarking layer_operations/layer_norm
Benchmarking layer_operations/layer_norm: Warming up for 3.0000 s
Benchmarking layer_operations/layer_norm: Collecting 100 samples in estimated 5.1160 s (106k iterations)
Benchmarking layer_operations/layer_norm: Analyzing
layer_operations/layer_norm
                        time:   [48.022 µs 48.187 µs 48.318 µs]
                        change: [-0.6726% -0.1785% +0.3293%] (p = 0.48 > 0.05)
                        No change in performance detected.
Found 15 outliers among 100 measurements (15.00%)
  7 (7.00%) low severe
  6 (6.00%) low mild
  2 (2.00%) high mild
Benchmarking layer_operations/rms_norm
Benchmarking layer_operations/rms_norm: Warming up for 3.0000 s
Benchmarking layer_operations/rms_norm: Collecting 100 samples in estimated 5.0475 s (197k iterations)
Benchmarking layer_operations/rms_norm: Analyzing
layer_operations/rms_norm
                        time:   [25.573 µs 25.633 µs 25.685 µs]
                        change: [-0.0987% +0.3403% +0.8343%] (p = 0.17 > 0.05)
                        No change in performance detected.
Found 12 outliers among 100 measurements (12.00%)
  6 (6.00%) low severe
  4 (4.00%) low mild
  1 (1.00%) high mild
  1 (1.00%) high severe

Benchmarking lora_rank_scaling/forward/rank_4
Benchmarking lora_rank_scaling/forward/rank_4: Warming up for 3.0000 s
Benchmarking lora_rank_scaling/forward/rank_4: Collecting 100 samples in estimated 7.5082 s (10k iterations)
Benchmarking lora_rank_scaling/forward/rank_4: Analyzing
lora_rank_scaling/forward/rank_4
                        time:   [1.1831 ms 1.2269 ms 1.2696 ms]
                        change: [-18.041% +0.1125% +22.079%] (p = 0.99 > 0.05)
                        No change in performance detected.
Found 17 outliers among 100 measurements (17.00%)
  12 (12.00%) low mild
  2 (2.00%) high mild
  3 (3.00%) high severe
Benchmarking lora_rank_scaling/forward/rank_8
Benchmarking lora_rank_scaling/forward/rank_8: Warming up for 3.0000 s

Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.4s, enable flat sampling, or reduce sample count to 50.
Benchmarking lora_rank_scaling/forward/rank_8: Collecting 100 samples in estimated 7.4145 s (5050 iterations)
Benchmarking lora_rank_scaling/forward/rank_8: Analyzing
lora_rank_scaling/forward/rank_8
                        time:   [2.3425 ms 2.5144 ms 2.6811 ms]
                        change: [-21.142% +0.3412% +27.815%] (p = 0.98 > 0.05)
                        No change in performance detected.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe
Benchmarking lora_rank_scaling/forward/rank_16
Benchmarking lora_rank_scaling/forward/rank_16: Warming up for 3.0000 s
Benchmarking lora_rank_scaling/forward/rank_16: Collecting 100 samples in estimated 5.0675 s (2100 iterations)
Benchmarking lora_rank_scaling/forward/rank_16: Analyzing
lora_rank_scaling/forward/rank_16
                        time:   [2.7234 ms 3.6364 ms 4.5889 ms]
                        change: [-30.082% -0.2473% +45.648%] (p = 0.99 > 0.05)
                        No change in performance detected.
Benchmarking lora_rank_scaling/forward/rank_32
Benchmarking lora_rank_scaling/forward/rank_32: Warming up for 3.0000 s
Benchmarking lora_rank_scaling/forward/rank_32: Collecting 100 samples in estimated 5.0167 s (1000 iterations)
Benchmarking lora_rank_scaling/forward/rank_32: Analyzing
lora_rank_scaling/forward/rank_32
                        time:   [3.1788 ms 5.2720 ms 7.3659 ms]
                        change: [-42.916% -0.6152% +73.216%] (p = 0.93 > 0.05)
                        No change in performance detected.
Found 20 outliers among 100 measurements (20.00%)
  20 (20.00%) high severe
Benchmarking lora_rank_scaling/forward/rank_64
Benchmarking lora_rank_scaling/forward/rank_64: Warming up for 3.0000 s
Benchmarking lora_rank_scaling/forward/rank_64: Collecting 100 samples in estimated 5.0050 s (1000 iterations)
Benchmarking lora_rank_scaling/forward/rank_64: Analyzing
lora_rank_scaling/forward/rank_64
                        time:   [5.9254 ms 9.6425 ms 13.628 ms]
                        change: [-43.701% -0.2814% +77.935%] (p = 0.98 > 0.05)
                        No change in performance detected.
Found 21 outliers among 100 measurements (21.00%)
  1 (1.00%) high mild
  20 (20.00%) high severe

Benchmarking lazy_vs_eager_lora/eager_forward
Benchmarking lazy_vs_eager_lora/eager_forward: Warming up for 3.0000 s

Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.4s, enable flat sampling, or reduce sample count to 50.
Benchmarking lazy_vs_eager_lora/eager_forward: Collecting 100 samples in estimated 7.4172 s (5050 iterations)
Benchmarking lazy_vs_eager_lora/eager_forward: Analyzing
lazy_vs_eager_lora/eager_forward
                        time:   [2.2944 ms 2.9124 ms 3.9965 ms]
                        change: [-20.890% +6.2861% +42.839%] (p = 0.71 > 0.05)
                        No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe
Benchmarking lazy_vs_eager_lora/lazy_forward_sync
Benchmarking lazy_vs_eager_lora/lazy_forward_sync: Warming up for 3.0000 s
Benchmarking lazy_vs_eager_lora/lazy_forward_sync: Collecting 100 samples in estimated 5.1486 s (1600 iterations)
Benchmarking lazy_vs_eager_lora/lazy_forward_sync: Analyzing
lazy_vs_eager_lora/lazy_forward_sync
                        time:   [1.8016 ms 2.4930 ms 3.1990 ms]
                        change: [-50.864% -17.879% +38.094%] (p = 0.51 > 0.05)
                        No change in performance detected.

