16 Gathering System-Level Information

Introduction

In this chapter, you’ll collect a practical cheat sheet for gathering system-level information with Python. Some functions are built into the standard library, and some come from third-party tools such as psutil.

For various reasons when writing applications, you’ll want to gather system-level information. It might be to fingerprint the environment, produce debugging output, or tell the user which system your application is having issues with.

Concepts Covered

Operating system and Python runtime details with platform
Hostname, service, and address helpers with socket
Filesystem and process helpers with os and pathlib
Cross-platform process inspection with psutil

Why This Matters

When you’re writing security tools, support scripts, or diagnostics, you’ll often need to understand where your code is running. Good system information helps you debug weird failures, produce useful reports, and avoid making assumptions that only work on your own machine.

Objective

Use Python to retrieve operating system, path, network, process, and environment information in a way that’s readable enough to reuse in your own tools.

Operating System and Version

platform.machine()

Returns the machine type, e.g. i386. You’ll get an empty string if Python can’t determine the value.

>>> import platform
>>> platform.machine()
'x86_64'

platform.node()

Returns the computer’s network name. You’ll get an empty string if Python can’t determine the value.

Warning

The network name may not be fully qualified!

>>> import platform
>>> platform.node()
'salem'

platform.processor()

Returns the processor name, e.g. 'amdk6'. You’ll get an empty string if Python can’t determine the value.

>>> import platform
>>> platform.processor()
''

Empty strings aren’t automatically errors here. Sometimes the operating system simply doesn’t expose the detail Python asked for, so treat these helpers as useful clues rather than sworn testimony.

platform.python_build()

Returns a tuple (buildno, builddate) stating the Python build number and date as strings.

>>> import platform
>>> platform.python_build()
('default', 'Jan 22 2019 21:57:15')

platform.python_compiler()

Returns a string identifying the compiler used for compiling Python.

>>> import platform
>>> platform.python_compiler()
'GCC 6.3.0 20170516'

platform.python_version()

Returns the Python version as a 'major.minor.patchlevel' string.

>>> import platform
>>> platform.python_version()
'3.7.1'

platform.system()

Returns the system/OS name, e.g. 'Linux', 'Windows', or 'Java'. You’ll get an empty string if Python can’t determine the value.

>>> import platform
>>> platform.system()
'Linux'

platform.uname()

Gives you a fairly portable uname interface. It returns a namedtuple() containing six attributes: system, node, release, version, machine, and processor.

>>> import platform
>>> platform.uname()
uname_result(system='Linux', node='salem', release='4.9.0-8-amd64', version='#1 SMP Debian 4.9.144-3.1 (2019-02-19)', machine='x86_64', processor='')

Networking

Caution

Some behaviour may be platform-dependent, because these calls use the operating system’s socket APIs.

Networking helpers are especially good at reminding us that Python is often politely asking the operating system for an answer. If the OS resolver, host file, or service database differs, Python’s answer may differ too.

socket.getfqdn([name])

Returns a fully qualified domain name for name. If name is omitted or empty, Python treats it as the local host.

>>> import socket
>>> socket.getfqdn("localhost")
'localhost'

socket.gethostbyname(hostname)

Translates a host name to IPv4 address format. The IPv4 address comes back as a string, such as '100.50.200.5'. If the host name is already an IPv4 address, Python returns it unchanged.

>>> import socket
>>> socket.gethostbyname("localhost")
'127.0.0.1'

socket.gethostname()

Returns a string containing the hostname of the machine where the Python interpreter is currently running.

Warning

gethostname() doesn’t always return the FQDN (Fully Qualified Domain Name). If you need an FQDN, use socket.getfqdn().

>>> import socket
>>> socket.gethostname()
'salem'

socket.gethostbyaddr(ip_address)

Returns a tuple (hostname, aliaslist, ipaddrlist). hostname is the primary host name responding to the given IP (Internet Protocol) address, aliaslist contains alternative host names for the same address, and ipaddrlist contains IP v4/v6 addresses for the same interface on the same host.

>>> import socket
>>> socket.gethostbyaddr("127.0.0.1")
('localhost', [], ['127.0.0.1'])

socket.getservbyname(servicename[, protocolname])

Translate an Internet service name and protocol name to a port number for that service. Python gets these mappings from the operating system, which usually tracks the IANA Service Name and Transport Protocol Port Number Registry.

If you provide the protocol name, use tcp or udp; otherwise Python can match any protocol.

>>> import socket
>>> socket.getservbyname("gopher", 'tcp')
70
>>> socket.getservbyname("pop3")
110

socket.getservbyport(port[, protocolname])

Translate an Internet port number and protocol name to a service name for that service.

If you provide the protocol name, use tcp or udp; otherwise Python can match any protocol.

>>> import socket
>>> socket.getservbyport(70)
'gopher'
>>> socket.getservbyport(70, 'udp')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
OSError: port/proto not found

Operating System Interfaces

The os module gets us closer to the process Python is running inside: real and effective IDs, login names, groups, and other operating-system details. These are the kinds of calls that are boring right up until a permissions bug makes them fascinating.

os.getegid()

Returns the effective group id of the current process. This corresponds to the “set id” bit on the file being executed in the current process.

Note

Only available on UNIX (and macOS).

>>> import os
>>> os.getegid()
1000

os.getuid()

Returns the current process’s real user id.

Note

Only available on UNIX (and macOS).

>>> import os
>>> os.getuid()
1000

os.getgid()

Returns the real group id of the current process.

Note

Only available on UNIX (and macOS).

>>> import os
>>> os.getgid()
1000

os.getgrouplist()

Returns the list of group ids that a user belongs to. If group isn’t already in the list, Python includes it; typically, group is the group ID field from the password record for user.

Note

Only available on UNIX (and macOS).

>>> import os
>>> os.getgrouplist('alzion', 1)
[1, 24, 25, 27, 29, 30, 44, 46, 108, 113, 114]

os.getlogin()

Returns the name of the user logged in on the controlling terminal of the process.

Note

Only available on UNIX (and macOS) and Windows.

>>> import os
>>> os.getlogin()
'alzion'

Using `pathlib`

The built-in pathlib module gives you a readable, object-oriented way to manipulate paths and find pathname information. Instead of treating paths as plain strings, you create a Path object and ask it questions.

Tip

Path doesn’t expand ~ automatically. Use .expanduser() when you want a user-relative path to become a real home-directory path.

This is another place where being explicit saves pain. A path that works from your terminal may not work from a daemon, notebook, cron job, service file, or test runner if the working directory is different.

Path.cwd() and Path(path).resolve()

Return absolute path information. Path.cwd() gives you the current working directory. Path(path).resolve() returns an absolute version of path and resolves symlinks it can follow.

>>> from pathlib import Path
>>> Path.cwd()
PosixPath('/current/project')
>>> Path('imgs').resolve()
PosixPath('/current/project/imgs')

Path(path).exists()

Returns True if the path refers to an existing path or an open file. Returns False for broken symbolic links. On some platforms, if you don’t have permission to inspect the path, it may return False even if the path physically exists.

>>> from pathlib import Path
>>> Path('missing-report.txt').exists()
False
>>> Path('imgs').exists()
True

Path(path).expanduser()

Expands an initial ~ to the user’s home directory.

Python uses the platform’s usual home-directory rules: on UNIX-like systems, it starts with the HOME environment variable when available; on Windows, it checks the usual user-profile environment variables.

>>> from pathlib import Path
>>> Path('~').expanduser()
PosixPath('/home/alzion')
>>> Path('~/tmp/learning-python-for-offensive-security').expanduser()
PosixPath('/home/alzion/tmp/learning-python-for-offensive-security')

Path(path).stat().st_atime

Returns the last access time for path. The return value is the number of seconds since EPOCH, e.g. 1551832874.

Tip

EPOCH is also known as POSIX time or UNIX Epoch time. It describes a point in time as the number of seconds that have elapsed since 00:00:00 Thursday, 1 January 1970 (UTC), minus leap seconds.

Raises an OSError if the file doesn’t exist or is inaccessible.

>>> from pathlib import Path
>>> Path('.').stat().st_atime
1551834369.9661243
>>> Path('requirements.txt').stat().st_atime
1551832462.8749728

Path(path).stat().st_mtime

Returns the last modified time for path. The return value is the number of seconds since EPOCH, e.g. 1551832874.

Raises an OSError if the file doesn’t exist or is inaccessible.

>>> from pathlib import Path
>>> Path('.').stat().st_mtime
1551834369.9661243
>>> Path('requirements.txt').stat().st_mtime
1551832462.8749728

Path(path).stat().st_size

Returns the size of a path in bytes. Raises an OSError if the file doesn’t exist or is inaccessible.

>>> from pathlib import Path
>>> Path('.').stat().st_size
320
>>> Path('requirements.txt').stat().st_size
73

Path(path).is_absolute()

Returns True if path is an absolute pathname. On Unix, that means it begins with a slash. On Windows, it begins with a slash or backslash after Python accounts for a possible drive letter.

>>> from pathlib import Path
>>> Path('.').is_absolute()
False
>>> Path('/home/alzion/sauce/learning-python-for-offensive-security').is_absolute()
True

Path(path).is_file()

Returns True if path is an existing regular file.

>>> from pathlib import Path
>>> Path('.').is_file()
False
>>> Path('/home/alzion/sauce/learning-python-for-offensive-security/requirements.txt').is_file()
True

Path(path).is_dir()

Returns True if path is an existing directory.

>>> from pathlib import Path
>>> Path('.').is_dir()
True
>>> Path('/home/alzion/sauce/learning-python-for-offensive-security/requirements.txt').is_dir()
False

Path(path).is_mount()

Returns True if path is a mount point: a place in a file system where a different file system has been mounted.

Warning

Mount-point detection has platform-specific edge cases, especially with bind mounts and unusual filesystem layouts. Treat Path.is_mount() as a useful helper, not a complete storage-inventory tool.

`psutil`

psutil (process and system utilities) is a third-party cross-platform library for retrieving information about running processes and system utilisation (CPU, memory, disks, network, and sensors) in Python. It implements a number of UNIX-based command-line tools and works on Linux, Windows, macOS, FreeBSD, OpenBSD, NetBSD, Sun Solaris, and AIX.

You can install it with python -m pip install psutil in the environment where you plan to run the examples. It’s really handy when you need to inspect what’s running, what resources exist, and what your process can see.

psutil.pids()

Returns a sorted list of currently running PIDs. To iterate over all processes and avoid race conditions, prefer process_iter().

>>> import psutil
>>> psutil.pids()
[0, 1, 40, 41, 44, 45, 46, 48, 51, 52, 54, 55, 58, 59, 65, 66, 67, 70, 71, 75, 76, 77, 78, 79, 80, 81, 82, 84, 86, 87, 89, 91, 93, 94, 95, 96, 98, 100, 101, 102, 103, 104, 107, 109, 114, 115, 125, 148, 162, 164, 173, 175, 176, 177, 178, 179, 186, 192, 193, 197, 199, 200, 201, 202, 203, 204, 206, 207, 208, 209, 210, 211, 212, 213, 215, 216, 217, 218, 219, 224, 228, 229, 230, 231, 232, 233, 236, 238, 239, 240, 241, 242, 250, 251, 253, 254, 259, 263, 264, 265, 268, 271, 272, 274, 276, 277, 278, 279, 280, 281, 282, 284, 286, 287, 288, 289, 290, 291, 292, 293, 296, 297, 298, 303, 304, 305, 306, 307, 308, 311, 314, 315, 316, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 359, 361, 363, 364, 365, 366, 367, 370, 372, 373, 374, 375, 377, 378, 379, 380, 381, 382, 384, 385, 389, 392, 393, 394, 397, 398, 400, 401, 405, 406, 407, 409, 411, 412, 415, 416, 417, 421, 423, 425, 426, 427, 428, 429, 430, 431, 432, 434, 435, 448, 450, 451, 462, 463, 464, 465, 466, 469, 479, 480, 481, 483, 484, 485, 489, 495, 497, 555, 557, 562, 565, 567, 568, 641, 642, 653, 656, 662, 665, 666, 673, 674, 675, 676, 689, 690, 691, 694, 695, 697, 699, 700, 701, 705, 708, 709, 711, 712, 719, 727, 729, 730, 731, 732, 733, 735, 737, 738, 739, 740, 744, 745, 746, 747, 750, 751, 752, 755, 756, 757, 758, 762, 763, 835, 838, 839, 842, 844, 872, 918, 1074, 1080, 1081, 1082, 1083, 1113, 1126, 1128, 1135, 1146, 1149, 1150, 1161, 1163, 1164, 1165, 1237, 1281, 1283, 1284, 1286, 1289, 1290, 1293, 1350, 1351, 1396, 1416, 1439, 1461, 1467, 1471, 1472, 1473, 1474, 1475, 1477, 1481, 1506, 1511, 1512, 1514, 1584, 1593, 1662, 1664, 1666, 1667, 1671, 1673, 1690, 1710, 1775, 1928, 2112, 2117, 2118, 2119, 2125, 2126, 2127, 2128, 2163, 2164, 2165, 2166, 2167, 2168, 2171, 2176, 2183, 2184, 2185, 2188, 2190, 2218, 2219]

psutil.pid_exists(pid)

Check whether the given PID exists in the current process list.

>>> import psutil
>>> psutil.pid_exists(415)
True
>>> psutil.pid_exists(20)
False

psutil.Process(pid=None)

Represents an OS process with the given pid. If pid is omitted, Python uses the current process pid (os.getpid).

>>> process = psutil.Process(740)

psutil.Process(pid=None).name()

Returns the process name. On Windows, the return value is cached after the first call. On POSIX, it isn’t cached because the process name may change.

>>> process = psutil.Process(740)
>>> process.name()
'Brave Browser Helper'

psutil.Process().environ()

Returns the process environment variables as a dict.

Caution

May not reflect changes made after the process started.

>>> import psutil
>>> env = psutil.Process().environ()
>>> sorted(env)[:4]
['HOME', 'LC_CTYPE', 'PATH', 'SHELL']

psutil.Process().environ()

Returns the process current working directory as an absolute path.

>>> psutil.Process().cwd()
'/Volumes/0X00/python/heartbleed'

psutil.cpu_count()

Returns the number of CPUs available on the machine.

>>> import psutil
>>> psutil.cpu_count()
4

psutil.disk_partitions()

Returns all mounted disk partitions as a list of named tuples including device, mount point, and filesystem type. The optional all argument tries to distinguish physical devices such as hard drives, CD-ROMs, and USB devices from memory partitions when set to False.

>>> import psutil
>>> psutil.disk_partitions(all=False)
[sdiskpart(device='/dev/disk1s1', mountpoint='/', fstype='apfs', opts='rw,local,rootfs,dovolfs,journaled,multilabel'), sdiskpart(device='/dev/disk1s4', mountpoint='/private/var/vm', fstype='apfs', opts='rw,noexec,local,dovolfs,dontbrowse,journaled,multilabel,noatime'), sdiskpart(device='/dev/disk2s1', mountpoint='/Volumes/0X00', fstype='exfat', opts='rw,nosuid,local,ignore-ownership')]
>>> psutil.disk_partitions(all=False)
[sdiskpart(device='/dev/disk1s1', mountpoint='/', fstype='apfs', opts='rw,local,rootfs,dovolfs,journaled,multilabel'), sdiskpart(device='/dev/disk1s4', mountpoint='/private/var/vm', fstype='apfs', opts='rw,noexec,local,dovolfs,dontbrowse,journaled,multilabel,noatime'), sdiskpart(device='/dev/disk2s1', mountpoint='/Volumes/0X00', fstype='exfat', opts='rw,nosuid,local,ignore-ownership')]

psutil.Popen(*args, **kwargs)

A more convenient interface to the built-in subprocess.Popen.

>>> import psutil
>>> from subprocess import PIPE
>>> p = psutil.Popen(["/usr/bin/python", "-c", "print('hello')"], stdout=PIPE)
>>> p.name()
'Python'
>>> p.username()
'rebecca'
>>> p.communicate()
(b'hello\n', None)

psutil.Popen also works as a context manager (with is supported). On exit, file descriptors are closed and the process is waited for, which gives you a cleaner shutdown:

>>> import psutil
>>> import subprocess
>>> with psutil.Popen(["ifconfig"], stdout=subprocess.PIPE) as proc:
>>>     log.write(proc.stdout.read())

Find a Process by Name

This tiny helper is the start of a useful pattern: gather a list of processes, filter it by something meaningful, and report the result. In a real tool, remember that process lists can change while you’re looking at them, so expect the occasional race condition.

import psutil

def find_procs_by_name(name: str):
    ls = []
    for process in psutil.process_iter(attrs=['name']):
        if process.info['name'] == name:
            ls.append(process)
    return ls

Identify an IP Address Location

This example shows how a command-line tool can combine argparse, an HTTP client, and JSON parsing. Use documentation addresses such as 203.0.113.42 from RFC 5737 while practising so the example stays harmless and repeatable.

The json part of the URL is the API endpoint path that asks the service to return JSON. It isn’t an HTTP Content-Type, and the IP address isn’t being sent as JSON.

If you’re the decorating and/or customisation type, this is also another chance to use a sweet argparse description. Just keep the target address safe while practising.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
__version__ = "0.0.0"

import json
import urllib3
import argparse


def build_argument_parser():
    parser = argparse.ArgumentParser(description=f"""

 ███▄ ▄███▓▄▄▄    ▄▄▄█████▓▄▄▄          ██░ ██ ▄▄▄      ██▀███  ██▓
▓██▒▀█▀ ██▒████▄  ▓  ██▒ ▓▒████▄       ▓██░ ██▒████▄   ▓██ ▒ ██▓██▒
▓██    ▓██▒██  ▀█▄▒ ▓██░ ▒▒██  ▀█▄     ▒██▀▀██▒██  ▀█▄ ▓██ ░▄█ ▒██▒
▒██    ▒██░██▄▄▄▄█░ ▓██▓ ░░██▄▄▄▄██    ░▓█ ░██░██▄▄▄▄██▒██▀▀█▄ ░██░
▒██▒   ░██▒▓█   ▓██▒▒██▒ ░ ▓█   ▓██▒   ░▓█▒░██▓▓█   ▓██░██▓ ▒██░██░
░ ▒░   ░  ░▒▒   ▓▒█░▒ ░░   ▒▒   ▓▒█░    ▒ ░░▒░▒▒▒   ▓▒█░ ▒▓ ░▒▓░▓  
░  ░      ░ ▒   ▒▒ ░  ░     ▒   ▒▒ ░    ▒ ░▒░ ░ ▒   ▒▒ ░ ░▒ ░ ▒░▒ ░
░      ░    ░   ▒   ░       ░   ▒       ░  ░░ ░ ░   ▒    ░░   ░ ▒ ░
       ░        ░  ░            ░  ░    ░  ░  ░     ░  ░  ░     ░  

Version: {__version__}
""", formatter_class=argparse.RawTextHelpFormatter)
    parser.add_argument(dest="ip_address",
                        help="the IP address you wish to find the location of")
    return parser.parse_args()


def main(arguments: argparse.Namespace):
    http = urllib3.PoolManager()
    url = f"http://ip-api.com/json/{arguments.ip_address}"
    response = http.request("GET", url)

    data = json.loads(response.data.decode("utf-8"))
    print(f"The host {arguments.ip_address} resides "
          f"in {data['country']} ({data['lat']}, {data['lon']}).")


if __name__ == "__main__":
    args = build_argument_parser()
    main(args)

Checkpoint

Write down three facts your system can report without making a network request: one from platform, one from os or pathlib, and one from socket. Which of those facts would be useful in a bug report?

Extension Activity

Turn the find_procs_by_name() example into a tiny report that prints each matching process PID and name. If you try it on your own machine, choose a harmless process name and handle the case where no matches are found.

Introduction

Concepts Covered

Why This Matters

Objective

Operating System and Version

Networking

Operating System Interfaces

Using pathlib

psutil

Find a Process by Name

Identify an IP Address Location

Checkpoint

Extension Activity

Further Reading

Using `pathlib`

`psutil`