The Python Trick I Wish I Knew Sooner: Mastering list dirs python os.listdir(), pathlib, os.walk()

I remember when I first started exploring list dirs Python. I was building a small project that needed to read all the files in a specific folder. I knew the files were there, but I had no idea how to get my Python script to “see” them. It felt like my program was blind. I spent a good amount of time searching for the solution, and that’s when I first came across os.listdir().

It was a total game changer for me. It was this simple, elegant piece of code that opened up a whole new world of file and directory manipulation. For anyone who has ever faced a similar challenge, I want to share my comprehensive guide to this essential function, including the ahas and the gotchas I encountered along the way.

Getting Started With List Dirs Python: What Exactly is `os.listdir()`?

At its core, os.listdir() is a function that lives in Python’s built in os module. Its job is incredibly straightforward: it takes a path to a directory and returns a list containing the names of all the entries in that directory.

When I first used it, I was working with a folder named my_documents that had a few files and another folder inside it. I ran a simple script:

				
					import os

# My directory path
my_path = 'my_documents'

# Get the list of contents
contents = os.listdir(my_path)

print(contents)

The output was exactly what I needed:

				
					['report.docx', 'images', 'data.xlsx']

The function was doing exactly what I wanted it to do. It was giving me a list of all the things inside that folder. It does not matter if the item is a file or another directory; it lists them all.

This was a huge victory for me. I had a clear, simple list I could loop through to process each file individually.

The First Pitfall I Ran Into: Relative vs. Absolute Paths

My initial success was a bit misleading. I quickly learned a crucial lesson: os.listdir() returns only the file or folder names, not their full paths. This became a problem when I tried to open one of the files I had found. My code looked something like this:

				
					import os

my_path = 'my_documents'
contents = os.listdir(my_path)

# This code will fail!
for item in contents:
    print(f"Trying to open: {item}")
    # Python doesn't know where 'report.docx' is
    # It will look in the current working directory, not 'my_documents'
    # with open(item, 'r') as file:
    # ...

The script gave me a FileNotFoundError. Why? Because my program was looking for report.docx in the same directory where my Python script was, not in the my_documents folder. The function gave me a list of names, but it was my job to put the full path together.

I had to learn to join the directory path and the item name to create a complete, usable path. I started using os.path.join() because it handles the different slash styles for Windows and Linux automatically.

				
					import os

my_path = 'my_documents'
contents = os.listdir(my_path)

for item in contents:
    # This creates the full path
    full_path = os.path.join(my_path, item)
    print(f"Successfully created full path: {full_path}")
    # Now I can open the file!
    # with open(full_path, 'r') as file:
    # ...

This simple fix was a huge learning moment for me and a critical step toward writing more robust Python code.

When `os.listdir()` is Not Enough: Dealing With Subdirectories

Another challenge I faced was when I needed to get a list of every file and folder within a directory and all its subdirectories. os.listdir() is not designed for this. It only gives you the contents of one single directory.

For example, if my my_documents folder looked like this:

				
					my_documents/
├── report.docx
├── images/
│   ├── photo1.jpg
│   └── photo2.png
└── data.xlsx

Running os.listdir('my_documents') would still only give me ['report.docx', 'images', 'data.xlsx']. It would completely ignore the images inside the images subdirectory.

This limitation led me to discover another powerful function in the os module: os.walk().

os.walk() is much more powerful and designed for exactly this purpose. It generates a list of files and subdirectories by “walking” a directory tree from top to bottom.

For each directory it finds, it returns three things in a tuple: the path to the current directory, a list of its subdirectories, and a list of its files.

Here is how I used it to solve my problem:

				
					import os

for dirpath, dirnames, filenames in os.walk('my_documents'):
    print(f"Current Path: {dirpath}")
    print(f"Subdirectories: {dirnames}")
    print(f"Files: {filenames}")
    print("-" * 20)

The output was exactly what I needed:

				
					Current Path: my_documents
Subdirectories: ['images']
Files: ['report.docx', 'data.xlsx']
--------------------
Current Path: my_documents/images
Subdirectories: []
Files: ['photo1.jpg', 'photo2.png']
--------------------

Using os.walk() made it easy to get a complete list of every single item, regardless of how deep it was in the directory structure.

My Final Recommendation: Using `pathlib`

While os.listdir() is a classic and still very useful, my final piece of advice is to consider using a more modern, object oriented approach with the pathlib module. It was introduced in Python 3.4 and, in my opinion, makes working with file systems much cleaner and easier to read.

Instead of returning a list of strings like os.listdir(), pathlib gives you a list of Path objects. These objects have methods built right in, so you can do things like check if an item is a file or a directory, and you can join paths with a simple slash operator.

Here is how I would solve my original problem today using pathlib:

				
					from pathlib import Path

my_path = Path('my_documents')

# Use a generator to get all items in the directory
contents = my_path.iterdir()

for item in contents:
    # Check if the item is a file
    if item.is_file():
        print(f"Found a file: {item}")

    # Check if the item is a directory
    if item.is_dir():
        print(f"Found a directory: {item}")

And to get every file and folder in the entire directory tree, pathlib has a fantastic method called glob().

				
					from pathlib import Path

my_path = Path('my_documents')

# Use ** to find items in subdirectories
for file_path in my_path.glob('**/*'):
    print(f"Found something: {file_path}")

This code is much cleaner, more intuitive, and less prone to the kind of path joining errors I ran into early on.

My Summary of Key Learnings

os.listdir() is a great, simple function for getting the contents of a single directory. Just remember that it returns only names, not full paths.
os.path.join() is your best friend when working with os.listdir() to combine directory names and file names into a full path.
os.walk() is the correct tool for the job if you need to go through an entire directory tree, including all its subdirectories.
pathlib is the modern, more powerful, and cleaner way to handle all your file system needs in Python. I now prefer it for all my projects because of its intuitive object oriented approach.

My exploration with these functions taught me that Python often has more than one way to do things, and choosing the right tool for the job is a key part of becoming an effective programmer. I hope sharing my experience helps you avoid some of the early mistakes I made and gives you the confidence to tackle your next file system task.

Explore further into the fascinating world of python by reading my main pillar post: Python For AI – My 5 Years Experience To Coding Intelligence

To Run, Generate, Explain, and Troubleshot Python code use our free AI Python Coding Assistant.