Introduction to Python Programming

BPLCK105B-205B

Module 4
Organizing Files: The shutil Module, Walking a Directory Tree, Compressing Files with the
zipfile Module, Project: Renaming Files with American-Style Dates to European-Style
Dates,Project: Backing Up a Folder into a ZIP File,
Debugging: Raising Exceptions, Getting the Traceback as a String, Assertions, Logging,
IDLE‟s Debugger.
Textbook 1: Chapters 9-10

Organizing Files
Programs can organize pre-existing files on the hard drive. Maybe we’ve had the experience of
going through a folder full of dozens, hundreds, or even thousands of files and copying,
renaming, moving, or compressing them all by hand. By programming our computer to do these
tasks, you can transform it into a quick-working file clerk who never makes mistakes.

The shutil Module

The shutil (or shell utilities) module has functions to let us copy, move, rename, and delete
files in your Python programs. To use the shutil functions, you will first need to use import
shutil.

Copying Files and Folders

 The shutil module provides functions for copying files, as well as entire folders. Calling
shutil.copy(source, destination) will copy the file at the path source to the folder at the path
destination.
 If destination is a filename, it will be used as the new name of the copied file. This function
returns a string of the path of the copied file.

Entering the following into the interactive shell to see how shutil.copy() works:

>>> import shutil, os

>>> os.chdir('C:\\')
>>> shutil.copy('C:\\bird.txt', 'C:\\colourful')
'C:\\colourful\\bird.txt'
>>> shutil.copy('eggs.txt', 'C:\\delicious\\food.txt')
'C:\\delicious\\food.txt'
 While shutil.copy() will copy a single file, shutil.copytree() will copy an entire folder and
every folder and file contained in it.
 Introduction to Python Programming
BPLCK105B-205B

 Calling shutil.copytree(source, destination) will copy the folder at the path source, along
with all of its files and subfolders, to the folder at the path destination.
>>> import shutil, os
>>> os.chdir('C:\\')
>>> shutil.copytree('C:\\hidden', 'C:\\Truth')
'C:\\Truth'

Moving and Renaming Files and Folders

Calling shutil.move(source, destination) will move the file or folder at the path source to the
path destination and will return a string of the absolute path of the new location.

>>> import shutil

>>> shutil.move('C:\\facts.txt', 'C:\\Interesting')
'C:\\Interesting\\facts.txt'
>>> shutil.move('C:\\bag.txt', 'C:\\school\\new_bag.txt')
'C:\\school\\new_bag.txt'

Assuming a folder named “Interesting” already exists in the C:\ directory, this shutil.move() calls
says, “Move C:\facts.txt into the folder C:\ Interesting.”. The line says, “Move C:\bag.txt into the
folder C:\school, and rename that bag.txt file to new_bag.txt.”

Permanently Deleting Files and Folders

We can delete a single file or a single empty folder with functions in the os module, whereas to
delete a folder and all of its contents, we use the shutil module.

• Calling os.unlink (path) will delete the file at path.

• Calling os.rmdir(path) will delete the folder at path. This folder must be empty of any files or
folders.
• Calling shutil.rmtree(path) will remove the folder at path, and all files and folders it contains
will also be deleted.

import os
for filename in os.listdir():
if filename.endswith('.rxt'):
os.unlink(filename)

Safe Deletes with the send2trash Module

 A much better way to delete files and folders is with the third-party send2trash module. We
can install this module by running pip install send2trash from a Terminal window.
 Using send2trash is much safer than Python’s regular delete functions, because it will send
folders and files to computer’s trash or recycle bin instead of permanently deleting them.
 Introduction to Python Programming
BPLCK105B-205B

 If a bug in the program deletes something with send2trash we didn’t intend to delete, you can
later restore it from the recycle bin.

Entering the following into the interactive shell:

>>> import send2trash

>>> baconFile = open('bacon.txt', 'a') # creates the file
>>> baconFile.write('Bacon is not a vegetable.')
25
>>> baconFile.close()
>>> send2trash.send2trash('bacon.txt')

Walking a Directory Tree

If we want to rename every file in some folder and also every file in every subfolder of that
folder, Python provides a function to handle this process. Let’s look at the C:\delicious folder
with its contents, shown in Figure.

Here is an example program that uses the os.walk() function on

the directory tree from Figure:

import os
for folderName, subfolders, filenames in os.walk('C:\\delicious'):
print('The current folder is ' + folderName)
for subfolder in subfolders:
print('SUBFOLDER OF ' + folderName + ': ' + subfolder)
for filename in filenames:
print('FILE INSIDE ' + folderName + ': '+ filename)
print('')

The os.walk() function is passed a single string value: the path of

a folder. We can use os.walk() in a for loop statement to walk a
directory tree. The os.walk() function will return three values on
each iteration through the loop:

1. A string of the current folder’s name

2. A list of strings of the folders in the current folder
3. A list of strings of the files in the current folder

Just like we can choose the variable name i in the code for i in range(10):, we can also choose the
variable names for the three values listed earlier. I usually use the names folder name, subfolders,
and filenames.

After we run this program, it will output the following:

The current folder is C:\delicious
 Introduction to Python Programming
BPLCK105B-205B

SUBFOLDER OF C:\delicious: cats

SUBFOLDER OF C:\delicious: walnut
FILE INSIDE C:\delicious: spam.txt
The current folder is C:\delicious\cats
FILE INSIDE C:\delicious\cats: catnames.txt
FILE INSIDE C:\delicious\cats: zophie.jpg
The current folder is C:\delicious\walnut
SUBFOLDER OF C:\delicious\walnut: waffles
The current folder is C:\delicious\walnut\waffles
FILE INSIDE C:\delicious\walnut\waffles: butter.txt.

Since os.walk() returns lists of strings for the subfolder and filename variables, we can use these
lists in their own for loops. Replace the print() function calls with our own custom code.

Compressing Files with the zipfile Module

ZIP files, which can hold the compressed contents of many other files.
Compressing a file reduces its size, which is useful when transferring it
over the Internet. And since a ZIP file can also contain multiple files
and subfolders, it’s a handy way to package several files into one. This
single file, called an archive file, can then be, say, attached to an email.

Our Python programs can both create and open (or extract) ZIP files
using functions in the zipfile module. A ZIP file named example.zip
that has the contents shown in Figure.

Reading ZIP Files

To read the contents of a ZIP file, first we must create a ZipFile object. They are values through
which the program interacts with the file. To create a ZipFile object, call the zipfile.ZipFile()
function, passing it a string of the .zip file’s filename. Note that zipfile is the name of the Python
module, and ZipFile() is the name of the function.

For example, enter the following into the interactive shell:

>>> import zipfile, os
>>> os.chdir('C:\\') # move to the folder with example.zip
>>> exampleZip = zipfile.ZipFile('example.zip')
>>> exampleZip.namelist()
['spam.txt', 'cats/', 'cats/catnames.txt', 'cats/zophie.jpg']
>>> spamInfo = exampleZip.getinfo('spam.txt')
>>> spamInfo.file_size
13908
>>> spamInfo.compress_size
3828
>>> 'Compressed file is %sx smaller!'% (round(spamInfo.file_size/spamInfo .compress_size, 2))
'Compressed file is 3.63x smaller!'
 Introduction to Python Programming
BPLCK105B-205B

>>> exampleZip.close()

 A ZipFile object has a namelist() method that returns a list of strings for all the files and
folders contained in the ZIP file.
 These strings can be passed to the getinfo() ZipFile method to return a ZipInfo object about
that particular file.
 ZipInfo objects have their own attributes, such as file_size and compress_size in bytes, which
hold integers of the original file size and compressed file size, respectively.
 While a ZipFile object represents an entire archive file, a ZipInfo object holds useful
information about a single file in the archive.
 The command at u calculates how efficiently example.zip is compressed by dividing the
original file size by the compressed file size and prints this information using a string
formatted with %s.

Extracting from ZIP Files

The extractall() method for ZipFile objects extracts all the files and folders from a ZIP file into
the current working directory.

>>> import zipfile, os

>>> os.chdir('C:\\') # move to the folder with example.zip
>>> exampleZip = zipfile.ZipFile('example.zip')
>>> exampleZip.extractall()
>>> exampleZip.close()

After running this code, the contents of example.zip will be extracted to C:\. Optionally, you can
pass a folder name to extractall() to have it extract the files into a folder other than the current
working directory.
If the folder passed to the extractall() method does not exist, it will be created.For instance, if we
replaced the call at u with exampleZip.extractall('C:\\ delicious'), the code would extract the files
from example.zip into a newly created C:\delicious folder.

The extract() method for ZipFile objects will extract a single file from the ZIP file. Continuing
the interactive shell example:

>>> exampleZip.extract('spam.txt')
'C:\\spam.txt'
>>> exampleZip.extract('spam.txt', 'C:\\some\\new\\folders')
'C:\\some\\new\\folders\\spam.txt'
>>> exampleZip.close()

The string you pass to extract() must match one of the strings in the list returned by namelist().
Optionally, we can pass a second argument to extract() to extract the file into a folder other than
the current working directory. If this second argument is a folder that doesn’t yet exist, Python
 Introduction to Python Programming
BPLCK105B-205B

will create the folder. The value that extract() returns is the absolute path to which the file was
extracted.

Creating and Adding to ZIP Files

 To create our own compressed ZIP files, we must open the ZipFile object in write mode by
passing 'w' as the second argument.
 When we pass a path to the write() method of a ZipFile object, Python will compress the file
at that path and add it into the ZIP file.
 The write() method’s first argument is a string of the filename to add. The second argument
is the compression type parameter, which tells the computer what algorithm it should use to
compress the files; we can always just set this value to zipfile.ZIP_DEFLATED.

Entering the following into the interactive shell:

>>> import zipfile

>>> newZip = zipfile.ZipFile('new.zip', 'w')
>>> newZip.write('spam.txt', compress_type=zipfile.ZIP_DEFLATED)
>>> newZip.close()

This code will create a new ZIP file named new.zip that has the compressed contents of spam.txt.
Just as with writing to files, write mode will erase all existing contents of a ZIP file. If you want
to simply add files to an existing ZIP file, pass 'a' as the second argument to zipfile.ZipFile() to
open the ZIP file in append mode.
 Introduction to Python Programming
BPLCK105B-205B

DEBUGGING
This chapter covers some tools and techniques for finding the root cause of bugs in the program
to fix bugs faster and with less effort. There are a few tools and techniques to identify what
exactly the code is doing and where it’s going wrong.

Logging and assertions are two features that can detect bugs early. In general, the earlier catch
bugs, the easier they will be to fix. The debugger is a feature of IDLE that executes a program
one instruction at a time, giving a chance to inspect the values in variables while your code runs,
and track how the values change over the course of program.

 RAISING EXCEPTIONS

Raising an exception is a way of saying, “Stop running the code in this function and move the
program execution to the except statement.” If there are no try and except statements covering
the raise statement that raised the exception, the program simply crashes and displays the
exception’s error message. Exceptions are raised with a raise statement consists of the following:

• The raise keyword

• A call to the Exception() function
• A string with a helpful error message passed to the Exception () function

For example boxprint program using exception

def boxPrint(symbol, width, height):

if len(symbol) != 1:
raise Exception('Symbol must be a single character string.')
if width <= 2:
raise Exception('Width must be greater than 2.')
if height <= 2:
raise Exception('Height must be greater than 2.')
print(symbol * width)
for i in range(height - 2):
print(symbol + (' ' * (width - 2)) + symbol)
print(symbol * width)
for sym, w, h in (('*', 4, 4), ('O', 20, 5), ('x', 1, 3), ('ZZ', 3, 3)):
 Introduction to Python Programming
BPLCK105B-205B

try:
boxPrint(sym, w, h)
except Exception as err:
print('An exception happened: ' + str(err))

Output:

****
* *
* *
****
OOOOOOOOOOOOOOOOOOOO
O O
O O
O O
OOOOOOOOOOOOOOOOOOOO

An exception happened: Width must be greater than 2.

An exception happened: Symbol must be a single character string.

Getting the Traceback As A String

1. def spam():
2. bacon()
3.
4. def bacon():
5. raise Exception('This is the error message.')
6.
7. spam()

Traceback (most recent call last): File "errorExample.py", line 7, in spam ()

File "errorExample.py", line 2, in spam bacon()
File "errorExample.py", line 5, in bacon raise
Exception ('This is the error message.')
Exception: This is the error message.

From the traceback, you can see that the error happened on line 5, in the bacon () function. This
particular call to bacon () came from line 2, in the spam () function, which in turn was called on
 Introduction to Python Programming
BPLCK105B-205B

line 7. In programs where functions can be called from multiple places, the call stack can help
you determine which call led to the error.

Assertions
An assertion is a sanity check to make sure the code isn’t doing something obviously wrong.
These sanity checks are performed by assert statements. If the sanity check fails, then an
Assertion Error exception is raised. In code, an assert statement consists of the following:

• The assert keyword

• A condition (that is, an expression that evaluates to True or False)
• A comma
• A string to display when the condition is False For example, enter the following into the
interactive shell

An assert statement says, “I assert that this condition holds true, and if not, there is a bug
somewhere in the program.” if an assert fails, our program should crash. This will reduce the
amount of code we will have to check before finding the code that’s causing the bug. Assertions
are for programmer errors, not user errors. For errors that can be recovered from (such as a file
not being found or the user entering invalid data), raise an exception instead of detecting it with
an assert statement.

Using an Assertion in a Traffic Light Simulation

Building a traffic light simulation program, the data structure representing the stoplights at an
intersection is a dictionary with keys 'ns' and 'ew', for the stoplights facing north-south and east-
west, respectively. The values at these keys will be one of the strings 'green', 'yellow', or 'red'.
The code would look something like this

market_2nd = {'ns': 'green', 'ew': 'red'}

mission_16th = {'ns': 'red', 'ew': 'green'}

def switchLights(stoplight):
for key in stoplight.keys():
if stoplight[key] == 'green':
stoplight[key] = 'yellow'
elif stoplight[key] == 'yellow':
stoplight[key] = 'red'
elif stoplight[key] == 'red':
 Introduction to Python Programming
BPLCK105B-205B

stoplight[key] = 'green'
switchLights(market_2nd)

But if while writing switchLights() you had added an assertion to check that at least one of the
lights is always red, you might have included the following at the bottom of the function:
assert 'red' in stoplight.values(), 'Neither light is red! ' + str(stoplight) . Neither direction of traffic
has a red light, meaning that traffic could be going both ways. By failing fast early in the
program’s execution, we can save ourself a lot of future debugging effort.
Disabling Assertions
Assertions are for development, not the final product. By the time you hand off your program to
someone else to run, it should be free of bugs and not require the sanity checks. Assertions can
be disabled by passing the -O option when running Python.

Logging
Logging levels provide a way to categorize log messages by importance. There are five logging
levels, described in Table from least to most important. Messages can be logged at each level
using a different logging function. Level Logging Function Description DEBUG logging.
debug() The lowest level. Used for small details.

Level Logging Description

Function
DEBUG logging.debug() The lowest level. Used for small details. Usually you care
about these messages only when diagnosing problems.
>>> import logging
>>> logging.basicConfig(level=logging.DEBUG, format=' %(asctime)s - %(levelname)s -
%(message)s')
>>> logging.debug('Some debugging details.')
2015-05-18 19:04:26,901 - DEBUG - Some debugging details
INFO logging.info() Used to record information on general events in your
program or confirm that things are working at their point in
the program.
>>> logging.info('The logging module is working.')
2015-05-18 19:04:35,569 - INFO - The logging module is working.
WARNING logging. Used to indicate a potential problem that doesn’t prevent the
warning() program from working but might do so in the future.
>>> logging.warning('An error message is about to be logged.')
2015-05-18 19:04:56,843 - WARNING - An error message is about to be logged.
ERROR logging.error() Used to record an error that caused the program to fail to do
something.
>>> logging.error('An error has occurred.')
 Introduction to Python Programming
BPLCK105B-205B

2015-05-18 19:05:07,737 - ERROR - An error has occurred.

CRITICAL logging.critical() The highest level. Used to indicate a fatal error that has
caused or is about to cause the program to stop running
entirely.
>>> logging.critical('The program is unable to recover!') 2015-05-18 19:05:45,794 - CRITICAL
- The program is unable to recover
Disabling Logging
The logging.disable() function disables these so that you don’t have to go into your program and
remove all the logging calls by hand. You simply pass logging.disable() a logging level, and it
will suppress all log messages at that level or lower. So if you want to disable logging entirely,
just add logging.disable(logging.CRITICAL) to your program. For example,

>>> import logging

>>> logging.basicConfig(level=logging.INFO, format=' %(asctime)s -
%(levelname)s - %(message)s')
Debugging 225
>>> logging.critical('Critical error! Critical error!')
2015-05-22 11:10:48,054 - CRITICAL - Critical error! Critical error!
>>> logging.disable(logging.CRITICAL)
>>> logging.critical('Critical error! Critical error!')
>>> logging.error('Error! Error!'

Since logging.disable() will disable all messages after it, we will probably want to add it near the
import logging line of code in your program. This way,we can easily find it to comment out or
uncomment that call to enable or disable logging messages as needed

IDLE’s Debugger
The debugger is a feature of IDLE that allows you to execute your program one line at a time.
The debugger will run a single line of code and then wait for you to tell it to continue. The
program will stay paused until you press one of the five buttons in the Debug Control window:
Go, Step, Over, Out, or Quit.

Go
 Clicking the Go button will cause the program to execute normally until it terminates or
reaches a breakpoint.
 If we have done debugging and want the program to continue normally, need to click the
Go button.
Step
 Clicking the Step button will cause the debugger to execute the next line of code and then
pause again.
 The Debug Control window’s list of global and local variables will be updated if their
values change.
 Introduction to Python Programming
BPLCK105B-205B

 If the next line of code is a function call, the debugger will “step into” that function and
jump to the first line of code of that function.
Over
 Clicking the Over button will execute the next line of code, similar to the Step button.
However, if the next line of code is a function call, the Over button will “step over” the
code in the function.
 The function’s code will be executed at full speed, and the debugger will pause as soon as
the function call returns

Out
 Clicking the Out button will cause the debugger to execute lines of code at full speed
until it returns from the current function.
 If we have stepped into a function call with the Step button and now simply want to keep
executing instructions until we get back out, need to click the Out button to “step out” of
the current function call.

Quit
 If we want to stop debugging entirely and not bother to continue executing the rest of the
program, click the Quit button.
 The Quit button will immediately terminate the program.
 If we want to run your program normally again, need to select Debug>Debugger again to
disable the debugger.

Breakpoints

 A breakpoint can be set on a specific line of code and forces the debugger to pause
whenever the program execution reaches that line.
 To set a breakpoint, right-click the line in the file editor and select Set Breakpoint.
 To remove a breakpoint, right-click the line in the file editor and select Clear Breakpoint
from the menu.

The following program simulates flipping a coin 1,000 times

import random
heads = 0
for i in range(1, 1001):
if random.randint(0, 1) == 1:
heads = heads + 1 Break Point
if i == 500:
print('Halfway done!')
print('Heads came up ' + str(heads) + ' times.')

When we run this program without the debugger, it quickly outputs something like the
following:
 Introduction to Python Programming
BPLCK105B-205B

Halfway done!
Heads came up 490 times.
If we were interested in the value of heads at the halfway point of the program’s execution, when
500 of 1000 coin flips have been completed, we could instead just set a breakpoint on the line
print('Halfway done!') .

***********************