8 Load and Clean a Wordlist

Introduction

When it comes to brute-forcing, wordlists make the world go ’round. Some people spend days, weeks, months, and even years refining their lists: most common names, company-specific guesses, service names, and all sorts of community-built maybes. Crafting an efficient wordlist is a loveless labour, but handling one well is a very learnable Python task.

In this lesson, you’ll teach the bruteforcer how to load candidate subdomain labels from disk. The code stays small, but it introduces an important tool-building habit: treat input as something that can be missing, blank, or messy.

Concepts Covered

Path handling with pathlib
Opening files with context managers
Stripping newline characters
Skipping blank input

Why This Matters

A scanner is only as helpful as the data you feed it. Wordlist handling is where beginner tools often fall over, so this lesson makes that path predictable before the project adds network behaviour.

Objective

Load a wordlist from disk and return a clean list of candidate labels.

Path Building in Python

pathlib.Path lets Python work with file paths without hand-building strings for each operating system. The course fixture gives us the repository root, and Path joins the pieces from there.

This is the modern version of “where did I put that file?” Instead of gluing strings together and hoping the slashes are right, we let Path do the operating-system-aware work for us.

from pathlib import Path
from fixtures.local_targets import course_root

wordlist_path = course_root() / "data" / "wordlist.txt"
print(wordlist_path.relative_to(course_root()))
print(wordlist_path.is_file())

1: Join path pieces with / instead of manually typing path separators.
2: Print the path relative to the course root so the output is stable across machines.
3: Check that the path points to a real file.

data/wordlist.txt
True

Here’s what’s happening: wordlist_path is a Path object, not just text. That means it can answer useful questions like “are you a file?” before the rest of the tool tries to read from it.

Reading and Cleaning Lines

When handling files it’s good practice to use a context manager. The with keyword ensures the file is closed properly even if an exception is raised, which is one less small cleanup problem for your application.

readlines() and normal file iteration keep newline characters. For a wordlist, those newlines are noise, so the loader should strip each line before returning it. strip() returns a copy of the string with whitespace removed from the beginning and end; it doesn’t modify the original string in place.

Python’s motto is often described as “ask for forgiveness, not permission”. Here, we open the file and let FileNotFoundError happen naturally for now. In the next pieces of the project, we’ll decide which failures should be handled and which should stop the tool.

def load_wordlist(path: Path) -> list[str]:
    with path.open(encoding="utf-8") as file:
        return [line.strip() for line in file if line.strip()]

candidates = load_wordlist(wordlist_path)
print(candidates[:4])
print(f"{len(candidates)} candidates loaded")

1: Use a context manager so Python closes the file automatically.
2: Strip whitespace and skip blank lines before the rest of the tool sees the input.

['www', 'mail', 'ftp', 'localhost']
6 candidates loaded

Here’s what’s happening: by the time the wordlist leaves this function, the rest of the program receives useful labels, not a jumble of trailing \n characters and empty lines.

Checkpoint

Add load_wordlist(path) to your project script. It should:

accept a Path
open the file with UTF-8 encoding
return stripped, non-empty lines
let FileNotFoundError happen naturally for now

When you load data/wordlist.txt, the first four entries should be:

['www', 'mail', 'ftp', 'localhost']

If your output includes extra blank strings or \n characters, don’t despair; Python is showing you exactly where the cleaning step still needs attention.

from pathlib import Path

def load_wordlist(path: Path) -> list[str]:
    # Open the file and return cleaned lines.
    pass

Extension Activity

Update load_wordlist() so it also skips lines that begin with #. Keep the original order of the remaining entries.

For more advanced users: try writing the same function once with a normal for loop and once with a list comprehension. List comprehensions are more concise, but sometimes being more verbose is easier on the programmer, and that’s a perfectly respectable choice.

Answer