    Finished `bench` profile [optimized] target(s) in 0.18s
     Running benches/training.rs (target/release/deps/training-b91a46af1471bb99)
Gnuplot not found, using plotters backend
Benchmarking lora_forward/forward/small_512x512_r8
Benchmarking lora_forward/forward/small_512x512_r8: Warming up for 3.0000 s
Benchmarking lora_forward/forward/small_512x512_r8: Collecting 100 samples in estimated 5.1080 s (15k iterations)
Benchmarking lora_forward/forward/small_512x512_r8: Analyzing
lora_forward/forward/small_512x512_r8
                        time:   [411.71 µs 420.17 µs 428.52 µs]
                        thrpt:  [76.467 Melem/s 77.988 Melem/s 79.590 Melem/s]
                 change:
                        time:   [-10.394% +0.3643% +11.419%] (p = 0.95 > 0.05)
                        thrpt:  [-10.249% -0.3630% +11.600%]
                        No change in performance detected.
Found 14 outliers among 100 measurements (14.00%)
  8 (8.00%) low severe
  1 (1.00%) low mild
  2 (2.00%) high mild
  3 (3.00%) high severe
Benchmarking lora_forward/forward/medium_1024x1024_r8
Benchmarking lora_forward/forward/medium_1024x1024_r8: Warming up for 3.0000 s
Benchmarking lora_forward/forward/medium_1024x1024_r8: Collecting 100 samples in estimated 8.1590 s (10k iterations)
Benchmarking lora_forward/forward/medium_1024x1024_r8: Analyzing
lora_forward/forward/medium_1024x1024_r8
                        time:   [2.4104 ms 2.5020 ms 2.5929 ms]
                        thrpt:  [25.276 Melem/s 26.193 Melem/s 27.189 Melem/s]
                 change:
                        time:   [-20.189% +0.0018% +26.347%] (p = 1.00 > 0.05)
                        thrpt:  [-20.853% -0.0018% +25.296%]
                        No change in performance detected.
Found 18 outliers among 100 measurements (18.00%)
  12 (12.00%) low mild
  3 (3.00%) high mild
  3 (3.00%) high severe
Benchmarking lora_forward/forward/large_2048x2048_r8
Benchmarking lora_forward/forward/large_2048x2048_r8: Warming up for 3.0000 s
Benchmarking lora_forward/forward/large_2048x2048_r8: Collecting 100 samples in estimated 5.2129 s (2100 iterations)
Benchmarking lora_forward/forward/large_2048x2048_r8: Analyzing
lora_forward/forward/large_2048x2048_r8
                        time:   [4.0066 ms 5.6578 ms 7.4184 ms]
                        thrpt:  [17.668 Melem/s 23.167 Melem/s 32.714 Melem/s]
                 change:
                        time:   [-34.172% +0.0071% +56.017%] (p = 1.00 > 0.05)
                        thrpt:  [-35.904% -0.0071% +51.912%]
                        No change in performance detected.
Found 18 outliers among 100 measurements (18.00%)
  18 (18.00%) high mild
Benchmarking lora_forward/forward/small_512x512_r16
Benchmarking lora_forward/forward/small_512x512_r16: Warming up for 3.0000 s
Benchmarking lora_forward/forward/small_512x512_r16: Collecting 100 samples in estimated 5.6500 s (600 iterations)
Benchmarking lora_forward/forward/small_512x512_r16: Analyzing
lora_forward/forward/small_512x512_r16
                        time:   [4.9754 ms 9.9131 ms 15.673 ms]
                        thrpt:  [2.0908 Melem/s 3.3055 Melem/s 6.5860 Melem/s]
                 change:
                        time:   [-58.111% +0.0028% +127.75%] (p = 0.98 > 0.05)
                        thrpt:  [-56.092% -0.0028% +138.73%]
                        No change in performance detected.
Found 14 outliers among 100 measurements (14.00%)
  2 (2.00%) high mild
  12 (12.00%) high severe

Benchmarking gradient_computation/forward_backward
Benchmarking gradient_computation/forward_backward: Warming up for 3.0000 s
Benchmarking gradient_computation/forward_backward: Collecting 100 samples in estimated 5.3557 s (1000 iterations)
Benchmarking gradient_computation/forward_backward: Analyzing
gradient_computation/forward_backward
                        time:   [5.0305 ms 5.0704 ms 5.1108 ms]
                        change: [-0.8135% -0.0423% +0.7093%] (p = 0.92 > 0.05)
                        No change in performance detected.
Found 15 outliers among 100 measurements (15.00%)
  8 (8.00%) low severe
  7 (7.00%) high severe

Benchmarking optimizer_step/adamw_step
Benchmarking optimizer_step/adamw_step: Warming up for 3.0000 s

Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 9.6s, enable flat sampling, or reduce sample count to 50.
Benchmarking optimizer_step/adamw_step: Collecting 100 samples in estimated 9.6304 s (5050 iterations)
Benchmarking optimizer_step/adamw_step: Analyzing
optimizer_step/adamw_step
                        time:   [1.0418 ms 1.2132 ms 1.4108 ms]
                        change: [-53.368% -25.854% +11.766%] (p = 0.37 > 0.05)
                        No change in performance detected.
Found 23 outliers among 100 measurements (23.00%)
  21 (21.00%) low severe
  1 (1.00%) low mild
  1 (1.00%) high severe

Benchmarking full_training_step/complete_iteration
Benchmarking full_training_step/complete_iteration: Warming up for 3.0000 s
Benchmarking full_training_step/complete_iteration: Collecting 10 samples in estimated 5.1067 s (1540 iterations)
Benchmarking full_training_step/complete_iteration: Analyzing
full_training_step/complete_iteration
                        time:   [3.0873 ms 3.0883 ms 3.0896 ms]
                        thrpt:  [42.424 Melem/s 42.441 Melem/s 42.455 Melem/s]
                 change:
                        time:   [-7.4192% -7.3519% -7.2986%] (p = 0.00 < 0.05)
                        thrpt:  [+7.8732% +7.9352% +8.0138%]
                        Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) low mild

Benchmarking layer_operations/softmax_stable
Benchmarking layer_operations/softmax_stable: Warming up for 3.0000 s
Benchmarking layer_operations/softmax_stable: Collecting 100 samples in estimated 5.2093 s (116k iterations)
Benchmarking layer_operations/softmax_stable: Analyzing
layer_operations/softmax_stable
                        time:   [44.792 µs 44.954 µs 45.094 µs]
                        change: [-0.3087% +0.4275% +1.1607%] (p = 0.28 > 0.05)
                        No change in performance detected.
Found 16 outliers among 100 measurements (16.00%)
  8 (8.00%) low severe
  6 (6.00%) low mild
  2 (2.00%) high severe
Benchmarking layer_operations/layer_norm
Benchmarking layer_operations/layer_norm: Warming up for 3.0000 s
Benchmarking layer_operations/layer_norm: Collecting 100 samples in estimated 5.1024 s (106k iterations)
Benchmarking layer_operations/layer_norm: Analyzing
layer_operations/layer_norm
                        time:   [48.054 µs 48.206 µs 48.337 µs]
                        change: [-1.2034% -0.6875% -0.1421%] (p = 0.01 < 0.05)
                        Change within noise threshold.
Found 16 outliers among 100 measurements (16.00%)
  9 (9.00%) low severe
  4 (4.00%) low mild
  3 (3.00%) high mild
Benchmarking layer_operations/rms_norm
Benchmarking layer_operations/rms_norm: Warming up for 3.0000 s
Benchmarking layer_operations/rms_norm: Collecting 100 samples in estimated 5.0485 s (197k iterations)
Benchmarking layer_operations/rms_norm: Analyzing
layer_operations/rms_norm
                        time:   [25.604 µs 25.646 µs 25.680 µs]
                        change: [+0.5391% +1.1388% +1.8427%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 11 outliers among 100 measurements (11.00%)
  7 (7.00%) low severe
  2 (2.00%) low mild
  1 (1.00%) high mild
  1 (1.00%) high severe

Benchmarking lora_rank_scaling/forward/rank_4
Benchmarking lora_rank_scaling/forward/rank_4: Warming up for 3.0000 s
Benchmarking lora_rank_scaling/forward/rank_4: Collecting 100 samples in estimated 7.4598 s (10k iterations)
Benchmarking lora_rank_scaling/forward/rank_4: Analyzing
lora_rank_scaling/forward/rank_4
                        time:   [1.1778 ms 1.2204 ms 1.2626 ms]
                        change: [-17.776% -0.2892% +20.902%] (p = 0.98 > 0.05)
                        No change in performance detected.
Found 17 outliers among 100 measurements (17.00%)
  12 (12.00%) low mild
  2 (2.00%) high mild
  3 (3.00%) high severe
Benchmarking lora_rank_scaling/forward/rank_8
Benchmarking lora_rank_scaling/forward/rank_8: Warming up for 3.0000 s

Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.4s, enable flat sampling, or reduce sample count to 50.
Benchmarking lora_rank_scaling/forward/rank_8: Collecting 100 samples in estimated 7.3833 s (5050 iterations)
Benchmarking lora_rank_scaling/forward/rank_8: Analyzing
lora_rank_scaling/forward/rank_8
                        time:   [2.3341 ms 2.5065 ms 2.6727 ms]
                        change: [-21.382% +0.2775% +27.928%] (p = 0.98 > 0.05)
                        No change in performance detected.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe
Benchmarking lora_rank_scaling/forward/rank_16
Benchmarking lora_rank_scaling/forward/rank_16: Warming up for 3.0000 s
Benchmarking lora_rank_scaling/forward/rank_16: Collecting 100 samples in estimated 5.0544 s (2100 iterations)
Benchmarking lora_rank_scaling/forward/rank_16: Analyzing
lora_rank_scaling/forward/rank_16
                        time:   [2.7303 ms 3.6476 ms 4.6084 ms]
                        change: [-30.090% -0.0156% +43.716%] (p = 0.99 > 0.05)
                        No change in performance detected.
Benchmarking lora_rank_scaling/forward/rank_32
Benchmarking lora_rank_scaling/forward/rank_32: Warming up for 3.0000 s
Benchmarking lora_rank_scaling/forward/rank_32: Collecting 100 samples in estimated 5.0529 s (1000 iterations)
Benchmarking lora_rank_scaling/forward/rank_32: Analyzing
lora_rank_scaling/forward/rank_32
                        time:   [3.2009 ms 5.3089 ms 7.4174 ms]
                        change: [-43.235% -0.1845% +81.995%] (p = 0.94 > 0.05)
                        No change in performance detected.
Found 20 outliers among 100 measurements (20.00%)
  20 (20.00%) high severe
Benchmarking lora_rank_scaling/forward/rank_64
Benchmarking lora_rank_scaling/forward/rank_64: Warming up for 3.0000 s
Benchmarking lora_rank_scaling/forward/rank_64: Collecting 100 samples in estimated 5.0441 s (1000 iterations)
Benchmarking lora_rank_scaling/forward/rank_64: Analyzing
lora_rank_scaling/forward/rank_64
                        time:   [5.9478 ms 9.6737 ms 13.728 ms]
                        change: [-46.022% -0.9237% +81.800%] (p = 0.97 > 0.05)
                        No change in performance detected.
Found 20 outliers among 100 measurements (20.00%)
  20 (20.00%) high severe

Benchmarking lazy_vs_eager_lora/eager_forward
Benchmarking lazy_vs_eager_lora/eager_forward: Warming up for 3.0000 s

Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.6s, enable flat sampling, or reduce sample count to 50.
Benchmarking lazy_vs_eager_lora/eager_forward: Collecting 100 samples in estimated 7.6151 s (5050 iterations)
Benchmarking lazy_vs_eager_lora/eager_forward: Analyzing
lazy_vs_eager_lora/eager_forward
                        time:   [2.2643 ms 2.4434 ms 2.6194 ms]
                        change: [-33.862% +6.8923% +65.336%] (p = 0.82 > 0.05)
                        No change in performance detected.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe
Benchmarking lazy_vs_eager_lora/lazy_forward_sync
Benchmarking lazy_vs_eager_lora/lazy_forward_sync: Warming up for 3.0000 s
Benchmarking lazy_vs_eager_lora/lazy_forward_sync: Collecting 100 samples in estimated 5.2410 s (1600 iterations)
Benchmarking lazy_vs_eager_lora/lazy_forward_sync: Analyzing
lazy_vs_eager_lora/lazy_forward_sync
                        time:   [1.9049 ms 2.7707 ms 3.7769 ms]
                        change: [-8.9969% +40.916% +120.93%] (p = 0.15 > 0.05)
                        No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

