ALL-SMI(1)

NAME
all-smi - Command-line utility for monitoring GPU hardware

SYNOPSIS
all-smi [local] [OPTIONS]
all-smi view --hosts URL... | --hostfile FILE [OPTIONS]
all-smi api [OPTIONS]
all-smi help [COMMAND]

DESCRIPTION
all-smi is a comprehensive hardware monitoring tool that provides real-time information about GPU/NPU utilization, memory usage, temperature, power consumption, chassis-level metrics, and other system information. It supports various hardware platforms including NVIDIA GPUs, AMD GPUs, Apple Silicon, NVIDIA Jetson, Intel Arc/Iris Xe client GPUs, Google Cloud TPUs, Tenstorrent NPUs, Rebellions NPUs, and Furiosa NPUs. Chassis monitoring provides node-level power consumption tracking (CPU+GPU+ANE), thermal pressure monitoring (Apple Silicon), and BMC sensor integration for inlet/outlet temperatures and fan speeds on server systems. The tool can run in three primary modes:

local mode
    An interactive TUI for monitoring local GPUs/NPUs (default when no command specified)

view mode
    An interactive TUI for monitoring remote nodes via API endpoints

api mode
    A Prometheus-compatible metrics endpoint for integration with monitoring systems

COMMANDS

local
    Run in local mode, monitoring local GPUs/NPUs with an interactive TUI. This is the default mode if no command is specified. No sudo required on macOS (uses native APIs).

view
    Run in remote view mode, monitoring remote nodes via API endpoints. Requires either --hosts or --hostfile option. Does not require sudo permissions.

api Run in API mode, exposing metrics in Prometheus format via HTTP endpoint. No sudo required on macOS (uses native APIs).

config SUBCOMMAND
    Manage the optional TOML configuration file. Subcommands:

    •   init — write a commented example config to the platform-canonical path.

    •   print — print the merged effective configuration (TOML or JSON).

    •   validate — parse a config file and report errors (exit 0 valid, 2 invalid).

    •   path — print the active config-file path and the candidate search order. Read-only; supports --json for scripts.

help
    Display help information about all-smi or a specific command.

OPTIONS

  Common Options

-h, --help
    Display help information

-V, --version
    Display version information

  Local Mode Options

-i, --interval SECONDS
    The interval in seconds at which to update the hardware information. Default: 1 second (Apple Silicon) or 2 seconds (others)

  View Mode Options (Remote Monitoring)

--hosts URL...
    (Required) A list of host addresses to connect to for remote monitoring. Multiple hosts can be specified. Format: http://hostname:port or https://hostname:port

--hostfile FILE
    (Required) A CSV file containing a list of host addresses to connect to for remote monitoring. The file should contain one URL per line. Either this or --hosts must be specified.

-i, --interval SECONDS
    The interval in seconds at which to update the hardware information. If not specified, uses adaptive interval based on node count:

    •   1-10 remote nodes: 3 seconds

    •   11-50 remote nodes: 4 seconds

    •   51-100 remote nodes: 5 seconds

    •   101+ remote nodes: 6 seconds

  API Mode Options

-p, --port PORT
    The port to listen on for the API server (default: 9090). Use 0 to disable TCP listener when using Unix socket.

-i, --interval SECONDS
    The interval in seconds at which to update the hardware information (default: 3)

--processes
    Include the process list in the API output

-s, --socket [PATH]
    Unix domain socket path for local IPC (Unix only). When specified without a path, uses platform default:

    •   Linux: /var/run/all-smi.sock (fallback to /tmp/all-smi.sock if no write permission)

    •   macOS: /tmp/all-smi.sock
    Socket permissions are set to 0600 (owner-only access) for security. Can be used together with --port for simultaneous TCP and Unix socket listeners.

INTERACTIVE CONTROLS (LOCAL/VIEW MODE)

Tab / Shift+Tab
    Navigate between different tabs (GPU, CPU, Memory, Chassis, Network, Disk)

Up/Down Arrow Keys, j/k
    Navigate through items in the current tab

Left/Right Arrow Keys, h/l
    Scroll horizontally when content exceeds screen width

Page Up/Page Down
    Scroll through items page by page

Home/End
    Jump to the first/last item

r   Refresh the display immediately

d   Sort by default (hostname+index)

f   Toggle GPU process filter (show only processes with GPU memory usage)

g   Sort by GPU memory usage

m   Sort by memory percentage

p   Sort by PID

u   Sort by utilization

h / 1
    Toggle help screen

q, Ctrl+C
    Quit the application

SUPPORTED PLATFORMS

NVIDIA GPUs
    Full support via nvidia-smi, including CUDA information, PCIe status, and performance states

AMD GPUs
    Support for Radeon and Instinct GPUs via ROCm/libamdgpu_top, including VRAM/GTT memory tracking and fdinfo-based process monitoring

Intel Arc / Iris Xe / Xe client GPUs
    Support for Arc A/B-series discrete GPUs and Iris Xe/Xe-LPG integrated GPUs via i915/xe sysfs (Linux) and WMI (Windows), including architecture classification, SYCL/oneAPI capability detection, engine-busy utilization computed from sysfs per-engine monotonic counters (Linux; max of render and compute engines reported as primary utilization), and per-process GPU memory tracking via /proc fdinfo (Linux; dedupes shared DRM file descriptors by drm-client-id). Optional Intel Level Zero (oneAPI) augmentation is available when the binary is built with --features level_zero (default: off). When enabled and libze_loader.so.1 (Linux) or ze_loader.dll (Windows) is present at runtime, additional per-engine activity metrics (including the XMX COMPUTE_SINGLE class not reachable via sysfs), energy-counter-derived power, temperature, dedicated/local memory state, frequency, and fan readings are surfaced when the corresponding Sysman domains are available and fresh. Seeded power and engine samples do not overwrite sysfs/WMI fallbacks. Missing loaders, missing optional symbols, unsupported domains, and permission failures degrade silently to the sysfs/WMI baseline, and per-field detail entries identify mixed metric sources.

Apple Silicon
    Native support for M1/M2/M3/M4 series with Metal API integration, ANE power monitoring, and thermal pressure tracking

NVIDIA Jetson
    Tegra-specific metrics with DLA (Deep Learning Accelerator) support

Google Cloud TPUs
    Support for TPU v2-v7/Ironwood architectures, HBM memory tracking, and integration with libtpu/JAX workflows

Tenstorrent NPUs
    Support for Grayskull and Wormhole architectures with multiple temperature sensors and clock domains

Rebellions NPUs
    Support for ATOM, ATOM+, and ATOM Max variants with PCIe Gen4 interface

Furiosa NPUs
    Support for RNGD architecture with 8-core NPUs and PE utilization tracking

x86 CPUs
    Intel and AMD CPU monitoring on Linux systems

API ENDPOINTS (API MODE)

/metrics
    Prometheus-compatible metrics endpoint with hardware statistics

/health
    Health check endpoint returning "OK"

EXAMPLES

Monitor local hardware in interactive TUI:
    all-smi
    or
    all-smi local

Start API server on default port 9090:
    all-smi api

Start API server on custom port with 5-second interval:
    all-smi api --port 8080 --interval 5

Start API server with Unix socket (default path):
    all-smi api --socket

Start API server with TCP and Unix socket simultaneously:
    all-smi api --port 9090 --socket /tmp/all-smi.sock

Start API server with Unix socket only (no TCP):
    all-smi api --port 0 --socket

Access metrics via Unix socket:
    curl --unix-socket /tmp/all-smi.sock http://localhost/metrics

Monitor local hardware with custom interval:
    all-smi local --interval 5

Monitor remote hosts (requires API endpoints):
    all-smi view --hosts http://node1:9090 http://node2:9090

Monitor hosts from file:
    all-smi view --hostfile cluster_hosts.csv

Monitor remote cluster with custom interval:
    all-smi view --hostfile hosts.csv --interval 10

FILES

hosts.csv
    Example hostfile format for remote monitoring. One URL per line:
    http://node001:9090
    http://node002:9090
    https://node003:9443

config.toml
    Optional TOML configuration file. Per-platform canonical paths:

    •   Linux: $XDG_CONFIG_HOME/all-smi/config.toml (fallback ~/.config/all-smi/config.toml)

    •   macOS: ~/Library/Application Support/all-smi/config.toml (also accepts ~/.config/all-smi/config.toml)

    •   Windows: %APPDATA%\all-smi\config.toml
    The active path for the current user is printed by all-smi --help (under "Configuration file") and by all-smi config path (read-only; supports --json for scripts). Override the discovery with --config PATH. A missing or malformed --config target is a hard error (exit 2); implicit discovery silently falls back to compiled defaults plus env vars when no candidate file exists.

energy-wal.bin
    Disk-backed write-ahead log persisting the cumulative joules counter across restarts. Default: under the platform cache directory as <cache>/all-smi/energy-wal.bin — Linux $XDG_CACHE_HOME/all-smi/energy-wal.bin (or ~/.cache/all-smi/energy-wal.bin when $XDG_CACHE_HOME is unset), macOS ~/Library/Caches/all-smi/energy-wal.bin, Windows %LOCALAPPDATA%\all-smi\energy-wal.bin. The record output (all-smi record) and the users-tab CSV export share the same <cache>/all-smi/ root for consistency. See ENERGY ACCOUNTING below for the ALL_SMI_ENERGY_WAL_PATH / ALL_SMI_ENERGY_NO_WAL overrides.

ENERGY ACCOUNTING
The Energy Session panel integrates instantaneous power draw over time to report cumulative energy consumption (kWh) and an optional cost estimate ($/kWh) for the current run. The session counter starts when the process launches and can be reset with the R key in the TUI; the underlying Prometheus all_smi_energy_joules_total counter is monotonic and is never reset.

Display behaviour is controlled via the [energy] section of the TOML config file (run all-smi config path to print the active location, or see FILES above) or the following environment variables, which take precedence over the file:

ALL_SMI_ENERGY_PRICE
    Electricity price per kWh as a decimal number (e.g. 0.12). A non-positive or non-finite value silently hides the cost column instead of surfacing $NaN. Default: 0.12.

ALL_SMI_ENERGY_CURRENCY
    Display-only currency code (e.g. USD, KRW, EUR). Default: USD.

ALL_SMI_ENERGY_NO_COST
    When set to any value, suppresses the monetary cost column. The kWh total and average power are still rendered.

ALL_SMI_ENERGY_WAL_PATH
    Path to the disk-backed write-ahead log used to persist the cumulative joules counter across restarts. Default: the platform cache directory's all-smi/energy-wal.bin — see FILES above for the per-platform path.

ALL_SMI_ENERGY_NO_WAL
    When set to any value, disables the disk-backed WAL — counters are in-memory only. Useful for ephemeral hosts.

ALL_SMI_ENERGY_GAP_SECONDS
    Sample-gap threshold in seconds (range [1, 3600]) above which the integrator switches from trapezoidal to hold-last integration. Default: 10.

ENVIRONMENT

RUST_LOG
    Set logging level (e.g., debug, info, warn, error)

NOTES

•   On macOS, no sudo permissions are required (uses native APIs: IOReport, SMC, NSProcessInfo)

•   Remote monitoring requires the target hosts to be running all-smi in API mode

•   The tool automatically detects available hardware and loads appropriate drivers

•   Connection limits are optimized for high-scale deployments (128+ nodes)

EXIT STATUS

0   Successful execution

1   General error or invalid arguments

130 Interrupted by Ctrl+C (SIGINT)

143 Terminated by SIGTERM

BUGS
Report bugs at: https://github.com/lablup/all-smi/issues

AUTHOR
Written by Jeongkyu Shin <inureyes@gmail.com>

COPYRIGHT
Copyright (C) 2025 Jeongkyu Shin.
License: MIT OR Apache-2.0
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

SEE ALSO
nvidia-smi(1) Project homepage: https://github.com/lablup/all-smi