ATOPGPUD(8)                  System Manager's Manual                 ATOPGPUD(8)

       atopgpud - GPU statistics daemon

       atopgpud [-v]

       The atopgpud daemon gathers statistical information from all Nvidia GPUs
       in the current system. With a sampling rate of one second, it maintains
       the statistics of every GPU, globally (system level) and per process.
       When atopgpud is active on the target system, atop connects to this
       daemon via a TCP socket and obtains all GPU statistics with every

       The approach to gather all GPU statistics in a separate daemon is
       required, because the Nvidia driver only offers the GPU busy percentage
       of the last second. Suppose that atop runs with a 10-minute interval and
       would fetch the GPU busy percentage directly from the Nvidia driver, it
       would reflect the busy percentage of the last second instead of the
       average busy percentage during 600 seconds.  Therefore, the atopgpud
       daemon fetches the GPU busy percentage every second and accumulates this
       into a counter that can be retrieved by atop regularly. The same approach
       applies to other GPU statistics.

       When the atopgpud daemon runs with root privileges, more process level
       counters (i.e.  GPU busy and GPU memory busy per process) are provided
       that are otherwise not applicable.

       Notice that certain GPU statistics are only delivered for specific GPU
       types.  For older or less sophisticated GPUs, the value -1 is returned
       for counters that are not maintained. In the output of atop these
       counters are shown as 'N/A'.

       When no (Nvidia) GPUs can be found in the target system, atopgpud
       immediately terminates with exit code 0.

       Log messages are written via the rsyslogd daemon with facility 'daemon'.
       With the -v flag (verbose), atopgpud also logs debug messages.

       The atopgpud daemon is written in Python, so a Python interpreter should
       be installed on the target system.  This can  either be Python version 2
       or Python version 3 (the code of atopgpud is written in a generic way).
       Take care that the first line of the atopgpud script contains the proper
       command name to activate a Python interpreter that is installed on the
       target system!

       The atopgpud daemon depends on the Python module pynvml to interface with
       the Nvidia driver.  This module can be installed by the pip or pip3
       command and is usually packaged under the name nvidia-ml-py
       Finally, the pynvml module is a Python wrapper around the libnvidia-ml
       shared library that needs to be installed as well.

       After installing the atop package, the atopgpud is not automatically
       started, nor will the service be enabed by default.  When you want to
       activate this service (permanently), enter the following commands (as

         systemctl enable atopgpu
         systemctl start atopgpu

       Client processes can connect to the atopgpud daemon on TCP port 59123.
       Subsequently, such client can send a request of two bytes, consisting of
       one byte request code followed by one byte integer being the API version
       The request code in the first byte can be 'T' to obtain information about
       the GPU types installed in this system (usually only requested once).
       The request code can be 'S' to obtain all statistical counter values
       (requested for every interval).

       The response of the daemon starts with a 4-byte integer. The first byte
       is the API version number that determines the response format while the
       subsequent three bytes indicate the length (big endian order) of the
       response string that follows.
       In the response strings the character '@' introduces system level
       information of one specific GPU and the character '#' introduces process
       level information related to that GPU.
       For further details about the meaning of the counters in a response
       string, please consult the source code.

       atop(1), atopsar(1), atoprc(5), netatop(4), netatopd(8), atopacctd(8)

       Gerlof Langeveld (

Linux                             November 2019                      ATOPGPUD(8)