## gpu-trace-perf

This is a rust rewrite of some tooling I built for comparing
performance between different graphics driver settings on graphics
traces.  The goal is for a driver developer to be able to quickly
experiment and find how their changes affect the performance of actual
rendering.

Right now apitrace, renderdoc, gfxretrace, and angle_trace_tests are supported.
You pass the tool a collection of GPU traces (or the path to the
angle_perf_traces binary, which will then enumerate tests), and it will run
through all of them, rerunning to get better stats over time, and give you an
estimate of the change in FPS from your driver change.

### Installing

```
apt-get install cargo
cargo install gpu-trace-perf
```

For apitrace traces (*.trace), you also need apitrace installed.  I
recommend having apitrace's waffle backend enabled, and
WAFFLE_PLATFORM=gbm set in the environment to not flicker windows on
the screen constantly.

For renderdoc traces (*.rdc), you need:

- python3
- renderdoc installed (`sudo apt-get install renderdoc`)
- renderdoc's python module findable from python3.

### Example usage

`gpu-trace-perf run --traces $HOME/src/traces-db --env baseline beforedriver --env new afterdriver`

This command will find all the traces in
[traces-db](https://gitlab.freedesktop.org/gfx-ci/tracie/traces-db/)
and run them in a loop printing stats until you feel ready to hit ^C.

Each `--env NAME WRAPPER` pair names a command wrapper script that prepares the
desired driver environment.  Provide one or more environments; when only a
single environment is given the tool continually reruns the traces and reports
basic FPS statistics for that setup.  When multiple environments are listed, the
first entry is treated as the baseline and every additional entry is compared
against it in the report.  The names you choose are what will be shown in the
printed statistics.  For example, to compare three different driver setups:

```
gpu-trace-perf run --traces $HOME/src/traces-db \
    --env baseline ./baseline.sh \
    --env upstream ./upstream.sh \
    --env downstream ./downstream.sh
```

The wrapper script itself can be as simple as:

```
#!/bin/sh

export LD_LIBRARY_PATH=$HOME/src/prefix/lib
"$@"
```

Since a traces db may be large and a change being tested may only affect a
subset of the traces, you can filter down which traces the replayed repeatedly
to only those whose stderr output is changed by some debug environment
variables:

```
    # Only re-run traces that had their shader compiler or command stream output changed on NVK.
    gpu-trace-perf run --traces $HOME/src/traces-db \
        --env baseline beforedriver --env new afterdriver \
        --debug-filter "NVK_DEBUG=push_dump" --debug-filter "NAK_DEBUG=print"
```

### Snapshot testing

When testing performance, you often want to also ensure that you haven't
regressed rendering, so the tool provides a snapshot (screenshot) capture and
diffing mode. This is a relatively new feature, and is currently supported only
for apitrace traces.  You can invoke it with the "snapshot" subcommand:

```
    # Run with the system driver.
    gpu-trace-perf snapshot --traces ~/src/traces-db --output baseline/
    # Run with your new GPU driver.
    meson devenv -C ~/src/mesa/build gpu-trace-perf snapshot --traces ~/src/traces-db --output test/ --baseline baseline/
```

The output directory will contain an index.html to summarize and visualize the
differences between the baseline run and your new one.  It uses javascript for
some visualization, so for full functionality you'll need to run an HTTP server
to avoid CORS errrors:

```
    sudo apt install python3-rangehttpserver
    cd test/ && python -m RangeHTTPServer
    firefox localhost:8080
```

### Running D3D traces

When its detects D3D traces (apitraces, or renderdoc traces with "dx9" or
similar in the name), gpu-trace-perf will run the appropriate replay tool under
wine. You will want to set your WINEPATH to include the directories for your
apitrace and renderdoc directories with the commands in them, e.g.:

```
export WINEPATH=$HOME/apitrace-latest-win64/bin/;$HOME/RenderDoc_1.39_64/
```

### Running ANGLE traces

To include ANGLE traces in your test list, include the angle_trace_tests binary
in the `-t` list (or the traces directory).  You can also select a specific
ANGLE trace with (for example):

```
gpu-trace-perf -t <path>/angle_trace_tests/TraceTest.minetest
```

### utrace timings

Normally, the trace tool-provided timings are used for the FPS result.  However,
sometimes you only care about a specific subset of your command buffers, and
watching just those can reduce the noise of the timings.  If the driver supports
utrace, you can pass "--utrace <comma_separated_utrace_events>" to use those
instead of the trace tool's timings.  This can be a particular help with
renderdoc traces, which have high CPU overhead for per-renderpass setup on
Vulkan.  Note that if your driver has a nonstandard prefix for start/end events,
you may need to add it to u_trace::Frame::event_times().  You can also specify
"drawcalls" as the event to expand to a list of common draw and compute events
across several drivers.

*NOTE* If you are testing apitraces, make sure you have apitrace newer than
13.0, that includes https://github.com/apitrace/apitrace/pull/961 to get correct
utrace results from GL traces.  D3D traces still don't tear down properly, so
the utrace results will be unreliable.

### Shader heuristic analysis

Sometimes as a developer, you want to select between two modes of compiling or
running a shader based on some heuristic.  This tool lets you generate the A/B
times per shader once, then iterate on your heuristic in the quick-to-run rust
code instead of doing lengthy trace replay runs for each idea you come up with.

The requirements are:

- The driver has utrace events for the start/end of draw call times
- The draw events include the hashes of the shaders for each stage involved in
  the draw.
- The change to the shader is represented in the shader hash.
- `check_debug_filter()` includes the shader dumping env vars for your driver
- your before/after scripts only append to any shader debug env vars also
  involved in shader dumping.
- `capture_draw_times()` has the utrace event names and the shader stage names
  for your driver.
- `shader_parser.rs` supports parsing your driver's shader outputs.

Currently, turnip is supported, and some code exists to support other drivers,
but is incomplete. If you meet all the requirements, running looks like:

````
    gpu-trace-perf run append \
        --env baseline beforedriver --env new afterdriver --output results/ \
        --capture-shaders  --traces $HOME/src/traces-db

    gpu-trace-perf shader-analyze output/
````

For any traces with shaders that changed modes, it will dump the per-trace
per-shader times between the before and after environments, and a table showing
the overall effect on trace times between the available heuristics (always
choose driver B, always choose optimally, and a dummy heuristic as an example).
Then, edit shader_analyze.rs to replace the dummy heuristic with your own,
potentially adding multiple to the table.

### Cross building for your embedded device

Add the following to ~/.cargo/config:

```
[target.armv7-unknown-linux-gnueabihf]
linker = "arm-linux-gnueabihf-gcc"

[target.aarch64-unknown-linux-gnu]
linker = "aarch64-linux-gnu-gcc"
```

And set up the new toolchain and build:

```
rustup target add aarch64-unknown-linux-gnu
cargo build --release --target aarch64-unknown-linux-gnu gpu-trace-perf
scp target/aarch64-unknown-linux-gnu/release/gpu-trace-perf device:bin/
```

### License

Licensed under the MIT license
   ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)
