Hardware Locality (hwloc): Environment Variables

Hardware Locality (hwloc) 2.14.0rc1

Environment Variables

The behavior of the hwloc library and tools may be tuned thanks to the following environment variables.

# Environment variables for changing the source of topology information

HWLOC_XMLFILE=/path/to/file.xml

  Enforce the discovery from the given XML file as if hwloc_topology_set_xml() had been called. This file may have been generated earlier with lstopo file.xml. For convenience, the XML backend provides empty binding hooks which just return success. To have hwloc still actually call OS-specific hooks, HWLOC_THISSYSTEM should be set 1 in the environment too, to assert that the loaded file is really the underlying system. See also Importing and exporting topologies from/to XML files.

HWLOC_SYNTHETIC=pack:3 [numa] L2:2 core:4 pu:2

  Enforce the discovery through a synthetic description string as if hwloc_topology_set_synthetic() had been called. For convenience, this backend provides empty binding hooks which just return success. See also Synthetic topologies.

HWLOC_FSROOT=/path/to/linux/filesystem-root/

  Switch to reading the topology from the specified Linux filesystem root instead of the main file-system root. This directory may have been saved previously from another machine with hwloc-gather-topology.

  One should likely also set HWLOC_COMPONENTS=linux,stop so that non-Linux backends are disabled (the -i option of command-line tools takes care of both).

  Not using the main file-system root causes hwloc_topology_is_thissystem() to return 0. For convenience, this backend provides empty binding hooks which just return success. To have hwloc still actually call OS-specific hooks, HWLOC_THISSYSTEM should be set 1 in the environment too, to assert that the loaded file is really the underlying system.

HWLOC_CPUID_PATH=/path/to/cpuid/

  Force the x86 backend to read dumped CPUIDs from the given directory instead of executing actual x86 CPUID instructions. This directory may have been saved previously from another machine with hwloc-gather-cpuid.

  One should likely also set HWLOC_COMPONENTS=x86,stop so that non-x86 backends are disabled (the -i option of command-line tools takes care of both).

  It causes hwloc_topology_is_thissystem() to return 0. For convenience, this backend provides empty binding hooks which just return success. To have hwloc still actually call OS-specific hooks, HWLOC_THISSYSTEM should be set 1 in the environment too, to assert that the loaded CPUID dump is really the underlying system.

HWLOC_DUMPED_HWDATA_DIR=/path/to/dumped/files/

  Loads file dumped by hwloc-dump-hwdata (on Linux) from the given directory. The default dump/load directory is configured during build based on --runstatedir, --localstatedir, and --prefix options. It usually points to /var/run/hwloc/ in Linux distribution packages, but it may also point to $prefix/var/run/hwloc/ when manually installing and only specifying --prefix.

HWLOC_ANNOTATE_GLOBAL_COMPONENTS=0

  Allow components to annotate the topology even if they are usually excluded by global components by default. For instance, setting this variable to 1 enables the addition of PCI vendor and model string info attributes to a XML topology that was generated without those names (if pciaccess was missing).

HWLOC_THISSYSTEM=1

  Assume that the selected backend provides the topology for the system on which we are running, even if it is not the OS-specific backend but the XML backend for instance. This means making the binding functions actually call the OS-specific system calls and really do binding, while the XML backend would otherwise provide empty hooks just returning success. This can be used for efficiency reasons to first detect the topology once, save it to a XML file, and quickly reload it later through the XML backend, but still having binding functions actually do bind.

  It enforces the return value of hwloc_topology_is_thissystem(), as if HWLOC_TOPOLOGY_FLAG_IS_THISSYSTEM was set with hwloc_topology_set_flags(). It also enables support for the variable HWLOC_THISSYSTEM_ALLOWED_RESOURCES.

# Environment variables for tweaking topology objects

HWLOC_PCI_LOCALITY=<domain/bus> <cpuset>;...
HWLOC_PCI_LOCALITY=/path/to/pci/locality/file

  Change the locality of I/O devices behind the specified PCI buses. If no I/O locality information is available or if the BIOS reports incorrect information, this variable allows to move a I/O device tree (OS and/or PCI devices with optional bridges) near a custom set of processors.

  Localities are given either inside the environment variable itself, or in the pointed file. They may be separated either by semi-colons or by line-breaks. Invalid localities are ignored, and it is possible to insert comments between actual localities by starting the line with # or //.

  Each locality contains a domain/bus specification (in hexadecimal numbers as usual) followed by a whitespace and a cpuset:

  - 0001 <cpuset> specifies the locality of all buses in PCI domain 0000.
  - 0000:0f <cpuset> specifies only PCI bus 0f in domain 0000.
  - 0002:04-0a <cpuset> specifies a range of buses (from 04 to 0a) within domain 0002.

  Domain/bus specifications should usually match entire hierarchies of buses behind a bridge (including primary, secondary and subordinate buses). For instance, if hostbridge 0000:00 is above other bridges/switches with buses 0000:01 to 0000:09, the variable should be HWLOC_PCI_LOCALITY="0000:00-09 <cpuset>", otherwise hwloc will try to extend the given locality to match the entire hierarchy. A single hierarchy of buses cannot be split between two localities.

  If the variable is defined to empty or invalid, no forced PCI locality is applied but hwloc's internal automatic locality quirks are disabled, which means the exact PCI locality reported by the platform is used.

HWLOC_X86_TOPOEXT_NUMANODES=0

  When using the x86 backend, setting this variable to 1 enables the building of NUMA nodes from AMD processor CPUID instructions. However this strategy does not always reflect BIOS configuration such as NUMA interleaving. And node indexes may be different from those of the operating system. Hence this should only be used when OS backends are wrong and the user is sure that CPUID returns correct NUMA information.

HWLOC_KEEP_NVIDIA_GPU_NUMA_NODES=0

  Show or hide NUMA nodes that correspond to NVIDIA GPU memory. By default they are ignored on POWER platforms to avoid interleaved memory being allocated on all CPUs and GPUs by mistake.

  Setting this environment variable to 0 hides the NUMA nodes (default on POWER). Setting to 1 exposes these NUMA nodes (default on non-POWER platforms such as NVIDIA Grace Hopper).

  These NUMA nodes may be recognized by the GPUMemory subtype. They also have a PCIBusID info attribute to identify the corresponding GPU.

HWLOC_KNL_MSCACHE_L3=0

  Expose the KNL MCDRAM in cache mode as a Memory-side Cache instead of a L3. hwloc releases prior to 2.1 exposed the MCDRAM cache as a CPU-side L3 cache. Now that Memory-side caches are supported by hwloc, it is still exposed as a L3 by default to avoid breaking existing applications. Setting this environment variable to 1 will expose it as a proper Memory-side cache.

HWLOC_WINDOWS_PROCESSOR_GROUP_OBJS=0

  Expose Windows processor groups as hwloc Group objects. By default (0), these groups are disabled because they may be incompatible with the hierarchy of resources that hwloc builds (leading to warnings). Setting this variable to 1 reenables the addition of these groups to the topology.

  This variable does not impact the querying of Windows processor groups using the dedicated API in hwloc/windows.h, this feature is always supported.

# Environment variables for tweaking hwloc heuristics

HWLOC_USE_NUMA_DISTANCES=7

  Enable or disable some use of NUMA distances and memory target/initiator information to improve the locality of NUMA nodes, especially CPU-less nodes.

  Bits in the value of this environment variable enable different features: Bit 0 enables the gathering of NUMA distances from the operating system. Bit 1 further enables the use of NUMA distances to improve the locality of CPU-less nodes. Bit 2 enables the use of target/initiator information. By default, all bits are set (7).

HWLOC_MEMTIERS_GUESS=none
HWLOC_MEMTIERS_GUESS=all

  Disable or enable all heuristics to guess memory subtypes and tiers. By default, hwloc only uses heuristics that are likely correct and disables those that are unlikely.

HWLOC_MEMTIERS=0x0f=HBM;0xf=DRAM

  Enforce the memory tiers from the given semi-colon separated list. Each entry specifies a bitmask (nodeset) of NUMA nodes and their subtype. Nodes not listed in any entry are not placed in any tier.

  If an empty value or none is given, tiers are entirely disabled.

HWLOC_MEMTIERS_REFRESH=1

  If set, this variable forces the rebuilding of memory tiers. This is mostly useful when importing a XML topology from an old hwloc version which was not able to guess memory subtypes and tiers.

HWLOC_GROUPING=1

  Enable or disable object grouping based on distances. By default (1), hwloc uses distance matrices between objects (either read from the OS or given by the user) to find groups of close objects. These groups are described by adding intermediate Group objects in the topology. Setting this environment variable to 0 will disable this grouping.

  See also HWLOC_GROUPING_VERBOSE for verbose messages about grouping.

HWLOC_GROUPING_ACCURACY=0.05

  Relax distance comparison during grouping. By default, objects may be grouped if their distances form a minimal distance graph. When setting this variable to 0.02, and when HWLOC_DISTANCES_ADD_FLAG_GROUP_INACCURATE is given, these distances do not have to be strictly equal anymore, they may just be equal with a 2% error.

  If set to try instead of a numerical value, hwloc will try to group with perfect accuracy (0, the default), then with 0.01, 0.02, 0.05 and finally 0.1.

  Numbers given in this environment variable should always use a dot as a decimal mark (for instance 0.01 instead of 0,01).

HWLOC_CPUKINDS_RANKING=default

  Change the ranking policy when guessing CPU kinds from frequencies and core types. hwloc tries to rank CPU kinds that are energy efficiency first, and then CPUs that are rather high-performance and power hungry.

  By default, if available, the OS-provided efficiency is used for ranking. Otherwise, the frequency and/or core types are used when available.

  This environment variable may be set to coretype+frequency, coretype+frequency_strict, coretype, frequency, frequency_base, frequency_max, forced_efficiency, no_forced_efficiency, default, or none.

HWLOC_CPUKINDS_MAXFREQ=adjust=10

  Change the use of the max frequency in the Linux backend. hwloc tries to read the base and max frequencies of each core on Linux. Some hardware features such as Intel Turbo Boost Max 3.0 make some cores report slightly higher max frequencies than others in the same CPU package. Despite having slightly different frequencies, these cores are considered identical instead of exposing an hybrid CPU. Hence, by default (adjust=10), hwloc uniformizes the max frequencies of cores that have the same base frequency (higher values are downgraded by up to 10%). When available, the CPU capacity value is also adjusted accordingly.

  If this environment variable is set to adjust=X, the 10% threshold is replaced with X. If set to 1, max frequencies are not adjusted anymore, some homogeneous processors may appear hybrid because of this. If set to 0, max frequencies are entirely ignored.

HWLOC_CPUKINDS_HOMOGENEOUS=0

  Uniformize max frequency, base frequency and Linux capacity to force a single homogeneous kind of CPUs. Setting to 1 enables this uniformization.

HWLOC_LINUX_CPUKINDS=-1

  Try to enable the discovery of CPU kinds on Linux. This is enabled by default except on old AMD CPUs known to be homogeneous. Setting to 1 always enables it, while 0 always disables it. Setting to cppc=0 enables it without using ACPI CPPC nominal frequency. Setting to midr=0 enables it without using the ARM MIDR register.

# Environment variables for changing allowed resources

HWLOC_THISSYSTEM_ALLOWED_RESOURCES=1

  Get the set of allowed resources from the native operating system even if the topology was loaded from XML or synthetic description, as if HWLOC_TOPOLOGY_FLAG_THISSYSTEM_ALLOWED_RESOURCES was set with hwloc_topology_set_flags(). This variable requires the topology to match the current system (see the variable HWLOC_THISSYSTEM).

  This is useful when the topology is not loaded directly from the local machine (e.g. for performance reason) and it comes with all resources, but the running process is restricted to only a part of the machine (for instance because of Linux Cgroup/Cpuset).

HWLOC_ALLOW=all

  Totally ignore administrative restrictions such as Linux Cgroups and consider all resources (PUs and NUMA nodes) as allowed. This is different from setting HWLOC_TOPOLOGY_FLAG_INCLUDE_DISALLOWED which gathers all resources but marks the unavailable ones as disallowed.

# Environment variables for controlling components and plugins

HWLOC_COMPONENTS=list,of,components

  Enable or disable the given comma-separated list of components (if they do not conflict with each other). Component names prefixed with - are disabled (a single phase may also be disabled).

  Once the end of the list is reached, hwloc falls back to enabling the remaining components (sorted by priority) that do not conflict with the already enabled ones, and unless explicitly disabled in the list. If stop is met, the enabling loop immediately stops, no more component is enabled.

  This is useful for understanding whether a topology issue comes from the native operating system backend or from the x86 backend: Setting to x86,stop or linux,stop will test one backend without the other.

  If xml or synthetic components are selected, the corresponding XML filename or synthetic description string should be passed in HWLOC_XMLFILE or HWLOC_SYNTHETIC respectively.

  Since this variable is the low-level and more generic way to select components, it takes precedence over environment variables for selecting components.

  If the variable is set to an empty string (or set to a single comma), no specific component is loaded first, all components are loaded in priority order.

  See Selecting which components to use for details.

HWLOC_GL=0

  The gl component is disabled by default since it may cause hwloc to hang when trying to connect to the X server. Setting this variable to 1 will let the component be enabled automatically as any other component.
  Users are advised to verify that lstopo runs fine with the HWLOC_GL=1 on a platform before enabling the variable more globally on that specific platform.

HWLOC_PLUGINS_PATH=/path/to/hwloc/plugins/:...

  Change the default search directory for plugins. By default, $libdir/hwloc is used. The variable may contain several colon-separated directories.

HWLOC_PLUGINS_BLACKLIST=filename1,filename2,...

  Prevent plugins from being loaded if their filename (without path) is listed. Plugin filenames may be found in verbose messages outputted when HWLOC_PLUGINS_VERBOSE=1.

HWLOC_RSMI_BACKEND=amd

  Use the new AMD SMI library for querying RSMI OS devices. If set to rocm instead, the old ROCm SMI library will be used. The default is AMD SMI is available, otherwise ROCm SMI.

# Environment variables for changing the verbosity

HWLOC_HIDE_ERRORS=1

  Enables or disable verbose reporting of errors. The hwloc library may issue warnings to the standard error stream when it detects a problem during topology discovery, for instance if the operating system (or user) gives contradictory topology information.

  By default (1), hwloc only shows critical errors such as invalid hardware topology information or invalid configuration. If set to 0 (default in lstopo), more errors are displayed, for instance a failure to initialize CUDA or NVML. If set to 2, no hwloc error messages are shown.

  Note that additional verbose messages may be enabled with other variables such as HWLOC_GROUPING_VERBOSE.

HWLOC_DEBUG_VERBOSE=0

  Disable all verbose messages that are enabled by default when –enable-debug is passed to configure. When set to more than 1, even more verbose messages are displayed. The default is 1.

HWLOC_XML_VERBOSE=1
HWLOC_SYNTHETIC_VERBOSE=1

  enables verbose messages in the XML or synthetic topology backends. hwloc XML backends (see Importing and exporting topologies from/to XML files) can emit some error messages to the error output stream. Enabling these verbose messages within hwloc can be useful for understanding failures to parse input XML topologies. Similarly, enabling verbose messages in the synthetic topology backend can help understand why the description string is invalid. See also Synthetic topologies.

HWLOC_GROUPING_VERBOSE=0

  Enable or disable some verbose messages during grouping. If this variable is set to 1, some debug messages will be displayed during distance-based grouping of objects even if debug was not specific at configure time. This is useful when trying to find an interesting distance grouping accuracy.

HWLOC_COMPONENTS_VERBOSE=1

  Display messages when components are registered or enabled. This is the recommended way to list the available components with their priority (all of them are registered at startup).

HWLOC_PLUGINS_VERBOSE=1

  Display verbose information about plugins: list which directories are scanned, which files are loaded, and which components are successfully loaded.