Introduction to Python Programming (BPLCK105B) Module 4

CHAPTER 1-ORGANISING FILES

1. The shutil Module

 The shutil (or shell utilities) module has functions to let you copy, move, rename,
and delete files in your Python programs
1.1 Copying Files and Folders
 The shutil module provides functions for copying files, as well as entire folders.
Calling shutil.copy(source, destination) will copy the file at the path source to the
folder at the path destination.
 If destination is a filename, it will be used as the new name of the copied file. This

[Type the document title]

function returns a string of the path of the copied file.
Program:
>>> import shutil, os
>>> os.chdir('C:\\')
u>>> shutil.copy('C:\\spam.txt', 'C:\\delicious')
'C:\\delicious\\spam.txt'
>>> shutil.copy('eggs.txt', 'C:\\delicious\\eggs2.txt')
'C:\\delicious\\eggs2.txt'

 The first shutil.copy() call copies the file at C:\spam.txt to the folder C:\delicious. The
return value is the path of the newly copied file. Note that since a folder was specified
as the destination
 The original spam.txt filename is used for the new, copied file’s filename. The second
shutil.copy()call also copies the file at C:\eggs.txt to the folder C:\delicious but gives

[Type
the copied file the name the
eggs2.txt. document title]
Program:

>>> import shutil, os

>>> os.chdir('C:\\')
>>> shutil.copytree('C:\\bacon', 'C:\\bacon_backup')
'C:\\bacon_backup'

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 1

Introduction to Python Programming (BPLCK105B) Module 4

 The shutil.copytree() call creates a new folder named bacon_backup with the same
content as the original bacon folder

1.2 Moving and Renaming Files and Folders

 Calling shutil.move(source, destination) will move the file or folder at the path source
to the path destination and will return a string of the absolute path of the new location.
 If destination points to a folder, the source file gets moved into destination and keeps
its current filename [Type the document title]

>>> import shutil

>>> shutil.move('C:\\bacon.txt', 'C:\\eggs')
'C:\\eggs\\bacon.txt'
 Assuming a folder named eggs already exists in the C:\ directory, this shutil.move()
calls says, “Move C:\bacon.txt into the folder C:\eggs.”
 The destination path can also specify a filename
[Type the document title]

>>> shutil.move('C:\\bacon.txt', 'C:\\eggs\\new_bacon.txt')

'C:\\eggs\\new_bacon.txt'

 Both of the previous examples worked under the assumption that there was a folder
eggs in the C:\ directory. But if there is no eggs folder, then move() will rename
bacon.txt to a file named eggs. [Type the document title]

>>> shutil.move('C:\\bacon.txt', 'C:\\eggs')

'C:\\eggs'
 The folders that make up the destination must already exist, or else Python will throw
an exception

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 2

Introduction to Python [Type the document
Programming title]
(BPLCK105B) Module 4

>>> shutil.move('spam.txt', 'c:\\does_not_exist\\eggs\\ham')

Traceback (most recent call last):
File "C:\Python34\lib\shutil.py", line 521, in move
os.rename(src, real_dst)
FileNotFoundError: [WinError 3] The system cannot find the path specified:
'spam.txt' -> 'c:\\does_not_exist\\eggs\\ham'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "<pyshell#29>", line 1, in <module>
shutil.move('spam.txt', 'c:\\does_not_exist\\eggs\\ham')
File "C:\Python34\lib\shutil.py", line 533, in move
copy2(src, real_dst)
File "C:\Python34\lib\shutil.py", line 244, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "C:\Python34\lib\shutil.py", line 108, in copyfile
with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory:
'c:\\does_not_exist\\eggs\\ham'

1.3 Permanently Deleting Files and Folders

 You can delete a single file or a single empty folder with functions in the os module,
whereas to delete a folder and all of its contents, you use the shutil module.
 Calling os.unlink(path) will delete the file at path.
 Calling os.rmdir(path) will delete the folder at path. This folder must be empty of
any files or folders
 Calling shutil.rmtree(path) will remove the folder at path, and all files and folders it
contains will also be deleted.

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 3

Introduction to Python Programming (BPLCK105B) Module 4
[Type the document title]

import os
for filename in os.listdir():
if filename.endswith('.rxt'):
os.unlink(filename)
 If you had any important files ending with .rxt, they would have been accidentally,
permanently deleted [Type the document title]

import os
for filename in os.listdir():
if filename.endswith('.rxt'):
#os.unlink(filename)
print(filename)
 Now the os.unlink() call is commented, so Python ignores it. Instead, you will print
the filename of the file that would have been deleted.

1.4 Safe Deletes with the send2trash Module

 Since Python’s built-in shutil.rmtree() function irreversibly deletes files and folders, it
can be dangerous to use
 A much better way to delete files and folders is with the third-party send2trash
module
 You can install this module by running pip install send2trash from a Terminal
window
 Using send2trash is much safer than Python’s regular delete functions, because it will
send folders and files to your computer’s trash or recycle bin instead of permanently
deleting them.

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 4

Introduction to Python Programming (BPLCK105B) Module 4
[Type the document title]

>>> import send2trash

>>> baconFile = open('bacon.txt', 'a') # creates the file
>>> baconFile.write('Bacon is not a vegetable.')
25
>>> baconFile.close()
>>> send2trash.send2trash('bacon.txt')

2. Walking a Directory Tree

 you want to rename every file in some folder and also every file in every subfolder of
that folder.
 That is, you want to walk through the directory tree, touching each file as you go.
 Writing a program to do this could get tricky; fortunately, Python provides a function
to handle this process for you.

Figure: An example folder that contains three folders and four files

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 5

Introduction to Python [Type the document
Programming title]
(BPLCK105B) Module 4

import os
for folderName, subfolders, filenames in os.walk('C:\\delicious'):
print('The current folder is ' + folderName)
for subfolder in subfolders:
print('SUBFOLDER OF ' + folderName + ': ' + subfolder)
for filename in filenames:
print('FILE INSIDE ' + folderName + ': '+ filename)
print('')

 The os.walk() function is passed a single string value: the path of a folder. You can
use os.walk() in a for loop statement to walk a directory tree, much like how you can
use the range() function to walk over a range of numbers.
 Unlike range(), the os.walk() function will return three values on each iteration
through the loop:
1. A string of the current folder’s name
2. A list of strings of the folders in the current folder
3. A list of strings of the files in the current folder
[Type the document title]

The current folder is C:\delicious

SUBFOLDER OF C:\delicious: cats
SUBFOLDER OF C:\delicious: walnut
FILE INSIDE C:\delicious: spam.txt

The current folder is C:\delicious\cats

FILE INSIDE C:\delicious\cats: catnames.txt
FILE INSIDE C:\delicious\cats: zophie.jpg

The current folder is C:\delicious\walnut

SUBFOLDER OF C:\delicious\walnut: waffles

The current folder is C:\delicious\walnut\waffles

FILE INSIDE C:\delicious\walnut\waffles: butter.txt.

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 6

Introduction to Python Programming (BPLCK105B) Module 4

3. Compressing Files with the zipfile Module

 Compressing a file reduces its size, which is useful when transferring it over
the Internet.
 since a ZIP file can also contain multiple files and subfolders, it’s a handy way
to package several files into one.
 This single file, called an archive file, can then be, say, attached to an email.

Figure : The contents of example.zip

3.1 Reading ZIP Files

 To read the contents of a ZIP file, first you must create a ZipFile object (note the
capital letters Z and F).
 ZipFile objects are conceptually similar to the File objects you saw returned by the
open() function
 They are values through which the program interacts with the file. To create a ZipFile
object, call the zipfile.ZipFile() function
For example

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 7

Introduction to Python [Type the document
Programming title]
(BPLCK105B) Module 4

>>> import zipfile, os

>>> os.chdir('C:\\') # move to the folder with example.zip
>>> exampleZip = zipfile.ZipFile('example.zip')
>>> exampleZip.namelist()
['spam.txt', 'cats/', 'cats/catnames.txt', 'cats/zophie.jpg']
>>> spamInfo = exampleZip.getinfo('spam.txt')
>>> spamInfo.file_size
13908
>>> spamInfo.compress_size
3828
>>> 'Compressed file is %sx smaller!' % (round(spamInfo.file_size / spamInfo
.compress_size, 2))
'Compressed file is 3.63x smaller!'
>>> exampleZip.close()

 A ZipFile object has a namelist() method that returns a list of strings for all the files
and folders contained in the ZIP file.
 These strings can be passed to the getinfo() ZipFile method to return a ZipInfo object
about that particular file. ZipInfo objects have their own attributes, such as file_size
and compress_size in bytes, which hold integers of the original file size and
compressed file size, respectively.

3.2 Extracting from ZIP Files

 The extractall() method for ZipFile objects extracts all the files and folders
from a ZIP file into the current working directory.

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 8

Introduction to Python [Type the document
Programming title]
(BPLCK105B) Module 4

>>> import zipfile, os

>>> os.chdir('C:\\') # move to the folder with example.zip
>>> exampleZip = zipfile.ZipFile('example.zip')
>>> exampleZip.extractall()
>>> exampleZip.close()

 The extract() method for ZipFile objects will extract a single file from the ZIP file.
Continue the interactive shell example [Type the document title]

>>> exampleZip.extract('spam.txt')
'C:\\spam.txt'
>>> exampleZip.extract('spam.txt', 'C:\\some\\new\\folders')
'C:\\some\\new\\folders\\spam.txt'
>>> exampleZip.close()

3.3 Creating and Adding to ZIP Files

 To create your own compressed ZIP files, you must open the ZipFile object in write
mode by passing 'w' as the second argument.
 When you pass a path to the write() method of a ZipFile object, Python will compress
the file at that path and add it into the ZIP file.
 The write() method’s first argument is a string of the filename to add.
 The second argument is the compression type parameter, which tells the computer
what algorithm it should use to compress the files;

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 9

 [Type the document title]
Introduction to Python Programming (BPLCK105B) Module 4

>>> import zipfile

>>> newZip = zipfile.ZipFile('new.zip', 'w')
>>> newZip.write('spam.txt', compress_type=zipfile.ZIP_DEFLATED)
>>> newZip.close()

4. Project: Renaming Files with American-Style Dates to European-Style

Dates
The program does:
 It searches all the filenames in the current working directory for
American-style dates.
 When one is found, it renames the file with the month and day
swapped to make it European-style.
 This means the code will need to do the following:
 Create a regex that can identify the text pattern of American-style
dates.
 Call os.listdir() to find all the files in the working directory.
 Loop over each filename, using the regex to check whether it has a
date.
 If it has a date, rename the file with shutil.move().

Step 1: Create a Regex for American-Style Dates

 The first part of the program will need to import the necessary modules and
create a regex that can identify MM-DD-YYYY dates.

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 10

Introduction to Python[Type the document
Programming title]
(BPLCK105B) Module 4

#! python3
# renameDates.py - Renames filenames with American MM-DD-YYYY date
format
# to European DD-MM-YYYY.
import shutil, os, re

# Create a regex that matches files with the American date format.
datePattern = re.compile(r"""^(.*?) # all text before the date
((0|1)?\d)- # one or two digits for the month
((0|1|2|3)?\d)- # one or two digits for the day
((19|20)\d\d) # four digits for the year
(.*?)$ # all text after the date
""", re.VERBOSE)

# TODO: Loop over the files in the working directory.

# TODO: Skip files without a date.
# TODO: Get the different parts of the filename.
# TODO: Form the European-style filename.
# TODO: Get the full, absolute file paths.
# TODO: Rename the files.

 After importing the re module at the top, call re.compile() to create a Regex
object
 Passing re.VERBOSE for the second argument w will allow whitespace and
comments in the regex string to make it more readable.

Step 2: Identify the Date Parts from the Filenames

 Next, the program will have to loop over the list of filename strings returned from
os.listdir() and match them against the regex.

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 11

Introduction to Python Programming (BPLCK105B) Module 4

 The matched text will be stored in several variables

[Type the document title]

#! python3
# renameDates.py - Renames filenames with American MM-DD-YYYY date
format
# to European DD-MM-YYYY.

--snip—

# Loop over the files in the working directory.

for amerFilename in os.listdir('.'):
mo = datePattern.search(amerFilename)
# Skip files without a date.
if mo == None:
continue

# Get the different parts of the filename.

beforePart = mo.group(1)
monthPart = mo.group(2)
dayPart = mo.group(4)
yearPart = mo.group(6)
afterPart = mo.group(8)
--snip--

 To keep the group numbers straight, try reading the regex from the beginning and
count up each time you encounter an opening parenthesis.
 Without thinking about the code, just write an outline of the regular expression.

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 12

Introduction to Python Programming (BPLCK105B) Module 4

datePattern = re.compile(r"""^(1) # all text before the date

(2 (3) )- # one or two digits for the month
(4 (5) )- # one or two digits for the day
(6 (7) ) # four digits for the year
(8)$ # all text after the date
""", re.VERBOSE)

Step 3: Form the New Filename and Rename the Files

 As the final step, concatenate the strings in the variables made in the previous step
[Type
with the European-style date:the document title]

#! python3
# renameDates.py - Renames filenames with American MM-DD-YYYY date
format
# to European DD-MM-YYYY.

--snip--
# Form the European-style filename.
euroFilename = beforePart + dayPart + '-' + monthPart + '-' + yearPart +
afterPart
# Get the full, absolute file paths.
absWorkingDir = os.path.abspath('.')
amerFilename = os.path.join(absWorkingDir, amerFilename)
euroFilename = os.path.join(absWorkingDir, euroFilename)

# Rename the files.

print('Renaming "%s" to "%s"...' % (amerFilename, euroFilename))
#shutil.move(amerFilename, euroFilename) # uncomment after testing

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 13

Introduction to Python Programming (BPLCK105B) Module 4

Project: Backing Up a Folder into a ZI P File

Step 1: Figure Out the ZIP File’s Name

 The code for this program will be placed into a function named backupToZip(). This
will make it easy to copy and paste the function into other Python programs that need
this functionality.
[Type the document title]

#! python3
# backupToZip.py - Copies an entire folder and its contents into
# a ZIP file whose filename increments.

import zipfile, os
def backupToZip(folder):
# Backup the entire contents of "folder" into a ZIP file.
folder = os.path.abspath(folder) # make sure folder is absolute
# Figure out the filename this code should use based on
# what files already exist.
number = 1
while True:
zipFilename = os.path.basename(folder) + '_' + str(number) + '.zip'
if not os.path.exists(zipFilename):
break
number = number + 1
# TODO: Create the ZIP file.
# TODO: Walk the entire folder tree and compress the files in each folder.
print('Done.')
backupToZip('C:\\delicious')

 Add the shebang (#!) line, describe what the program does, and import the zipfile and
os modules
 Define a backupToZip() function that takes just one parameter, folder. This parameter
is a string path to the folder whose contents should be backed up.

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 14

Introduction to Python Programming (BPLCK105B) Module 4

 The function will determine what filename to use for the ZIP file it will
create; then the function will create the file, walk the folder folder, and each of the
subfolders and files to the ZIP file.

Step 2: Create the New ZIP File

#! python3
# backupToZip.py - Copies an entire folder and its contents into
# a ZIP file whose filename increments.

--snip--
while True:
zipFilename = os.path.basename(folder) + '_' + str(number) + '.zip'
if not os.path.exists(zipFilename):
break
number = number + 1
# Create the ZIP file.
print('Creating %s...' % (zipFilename))
backupZip = zipfile.ZipFile(zipFilename, 'w')

# TODO: Walk the entire folder tree and compress the files in each folder.
print('Done.')

backupToZip('C:\\delicious')

Step 3: Walk the Directory Tree and Add to the ZIP File

 Now you need to use the os.walk() function to do the work of listing every file in the
folder and its subfolders

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 15

Introduction to Python Programming
[Type (BPLCK105B)
the document title] Module 4

#! python3
# backupToZip.py - Copies an entire folder and its contents into
# a ZIP file whose filename increments.

--snip—

# Walk the entire folder tree and compress the files in each folder.
for foldername, subfolders, filenames in os.walk(folder):
print('Adding files in %s...' % (foldername))
# Add the current folder to the ZIP file.
backupZip.write(foldername)
# Add all the files in this folder to the ZIP file.
for filename in filenames:
newBase / os.path.basename(folder) + '_'
if filename.startswith(newBase) and filename.endswith('.zip')
continue # don't backup the backup ZIP files
backupZip.write(os.path.join(foldername, filename))
backupZip.close()
print('Done.')

backupToZip('C:\\delicious')
[Type the document title]

Creating delicious_1.zip...
Adding files in C:\delicious...
Adding files in C:\delicious\cats...
Adding files in C:\delicious\waffles...
Adding files in C:\delicious\walnut...
Adding files in C:\delicious\walnut\waffles...
Done.

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 16

Introduction to Python Programming (BPLCK105B) Module 4

CHAPTER -2 DEBUGGING

1. Raising Exceptions
 Python raises an exception whenever it tries to execute invalid code.
 Raising an exception is a way of saying, “Stop running the code in this function and
move the program execution to the except statement.”
 Exceptions are raised with a raise statement. In code, a raise statement consists of the
following:
 The raise keyword
 A call to the Exception() function
 A string with a helpful error message passed to the Exception() function
 For example

>>> raise Exception('This is the error message.')

Traceback (most recent call last):
File "<pyshell#191>", line 1, in <module>
raise Exception('This is the error message.')
Exception: This is the error message

 If there are no try and except statements covering the raise statement that raised the
exception, the program simply crashes and displays the exception’s error message..

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 17

Introduction to Python Programming (BPLCK105B)
[Type the document title] Module 4

def boxPrint(symbol, width, height):

if len(symbol) != 1:
raise Exception('Symbol must be a single character string.')
if width <= 2:
raise Exception('Width must be greater than 2.')
if height <= 2:
raise Exception('Height must be greater than 2.')
print(symbol * width)
for i in range(height - 2):
print(symbol + (' ' * (width - 2)) + symbol)
print(symbol * width)

for sym, w, h in (('*', 4, 4), ('O', 20, 5), ('x', 1, 3), ('ZZ', 3, 3)):
try:
boxPrint(sym, w, h)
except Exception as err:
print('An exception happened: ' + str(err))

 This program uses the except Exception as err form of the except statement If an
Exception object is returned from boxPrint()
 This except statement will store it in a variable named err. The Exception object can
then be converted to a string by passing it to str() to produce a userfriendly error
message

Output [Type the document title]

****
**
**
****
OOOOOOOOOOOOOOOOOOOO
O O
O O
O O
OOOOOOOOOOOOOOOOOOOO
An exception happened: Width must be greater than 2.
An exception happened: Symbol must be a single character string.

2. Getting the Traceback as a String

 When Python encounters an error, it produces a treasure trove of error
information called the traceback.

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 18

Introduction to Python Programming (BPLCK105B) Module 4

 The traceback includes the error message, the line number of the line that
[Type the document title]
caused the error, and the sequence of the function calls that led to the error.

def spam():
bacon()
def bacon():
raise Exception('This is the error message.')
spam()
Output:
[Type the document title]

Traceback (most recent call last):

File "errorExample.py", line 7, in <module>
spam()
File "errorExample.py", line 2, in spam
bacon()
File "errorExample.py", line 5, in bacon
raise Exception('This is the error message.')
Exception: This is the error message.

 The traceback is displayed by Python whenever a raised exception goes unhandled

 But you can also obtain it as a string by calling traceback.format_exc(). This function
is useful if you want the information from an exception’s traceback but also want an
except statement to gracefully handle the exception.
 For example,

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 19

Introduction to Python Programming (BPLCK105B)
[Type the document title] Module 4

>>> import traceback

>>> try:
raise Exception('This is the error message.')
except:
errorFile = open('errorInfo.txt', 'w')
errorFile.write(traceback.format_exc())
errorFile.close()
print('The traceback info was written to errorInfo.txt.')
116
The traceback info was written to errorInfo.txt.

[Type the document title]

 The 116 is the return value from the write() method, since 116 characters were written
to the file.

Traceback (most recent call last):

File "<pyshell#28>", line 2, in <module>
Exception: This is the error message.
3. Assertions
 An assertion is a sanity check to make sure your code isn’t doing something
obviously wrong.
 These sanity checks are performed by assert statements. If the sanity check fails, then
an AssertionError exception is raised.
 assert statement consists of the following:
 The assert keyword
 A condition (that is, an expression that evaluates to True or False)
 A comma
 A string to display when the condition is False
 For example,

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 20

Introduction to Python Programming (BPLCK105B) Module 4
[Type the document title]

>>> podBayDoorStatus = 'open'

>>> assert podBayDoorStatus == 'open', 'The pod bay doors need to be "open".'
>>> podBayDoorStatus = 'I\'m sorry, Dave. I\'m afraid I can't do that.''
>>> assert podBayDoorStatus == 'open', 'The pod bay doors need to be "open".'
Traceback (most recent call last):
File "<pyshell#10>", line 1, in <module>
assert podBayDoorStatus == 'open', 'The pod bay doors need to be "open".'
AssertionError: The pod bay doors need to be "open".

 set podBayDoorStatus to 'open', so from now on, we fully expect the value of this
variable to be 'open'
 In a program that uses this variable, we might have written a lot of code under the
assumption that the value is 'open'—code that depends on its being 'open' in order to
work as we expect. So we add an assertion to make sure we’re right to assume
podBayDoorStatus is 'open'
 Here, we include the message 'The pod bay doors need to be "open".' so it’ll be easy
to see what’s wrong if the assertion fails.

Using an Assertion in a Traffic Light Simulation

 The data structure representing the stoplights at an intersection is a dictionary with
keys 'ns' and 'ew', for the stoplights facing north-south and east-west, respectively.
 The values at these keys will be one of the strings 'green', 'yellow', or 'red'. The code
would look something like this: [Type the document title]

market_2nd = {'ns': 'green', 'ew': 'red'}

mission_16th = {'ns': 'red', 'ew': 'green'}
 To start the project, you want to write a switchLights() function, which will take an
intersection dictionary as an argument and switch the lights.

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 21

Introduction to Python Programming (BPLCK105B) Module 4

 At first, you might think that switchLights() should simply switch each light to the
next color in the sequence: Any 'green' values should change to 'yellow', 'yellow'
values should change to 'red', and 'red' values should change to 'green'.
[Type the document title]
Program:
def switchLights(stoplight):
for key in stoplight.keys():
if stoplight[key] == 'green':
stoplight[key] = 'yellow'
elif stoplight[key] == 'yellow':
stoplight[key] = 'red'
elif stoplight[key] == 'red':
stoplight[key] = 'green'
switchLights(market_2nd)

 while writing switchLights() you had added an assertion to check that at least one of
the lights is always red,
[Type the document title]
 include the following at the bottom of the function:

assert 'red' in stoplight.values(), 'Neither light is red! ' + str(stoplight)

[Type the document title]
 With this assertion in place, your program would crash with this error message:

Traceback (most recent call last):

File "carSim.py", line 14, in <module>
switchLights(market_2nd)
File "carSim.py", line 13, in switchLights
assert 'red' in stoplight.values(), 'Neither light is red! ' + str(stoplight)
u AssertionError: Neither light is red! {'ns': 'yellow', 'ew': 'green'}
 The important line here is the AssertionError
 Neither direction of traffic has a red light, meaning that traffic could be going both
ways. By failing fast early in the program’s execution

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 22

Introduction to Python Programming (BPLCK105B) Module 4

4. Logging
 Logging is a great way to understand what’s happening in your program and
in what order its happening.
 Python’s logging module makes it easy to create a record of custom messages
that you write.
 These log messages will describe when the program execution has reached the
logging function call and list any variables you have specified at that point in
time.
 On the other hand, a missing log message indicates a part of the code was
skipped and never executed.
Using the logging Module
 To enable the logging module to display log messages on your screen as your
program runs,
[Type the document title]

import logging
logging.basicConfig(level=logging.DEBUG, format=' %(asctime)s -
%(levelname)s
- %(message)s')
 when Python logs an event, it creates a LogRecord object that holds
information about that event
 The logging module’s basicConfig() function lets you specify what details
about the LogRecord object you want to see and how you want those details
displayed

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 23

Introduction to Python Programming (BPLCK105B) Module 4
[Type the document title]
Program:

import logging
logging.basicConfig(level=logging.DEBUG, format=' %(asctime)s - %(levelname)s
- %(message)s')
logging.debug('Start of program')
def factorial(n):
logging.debug('Start of factorial(%s%%)' % (n))
total = 1
for i in range(n + 1):
total *= i
logging.debug('i is ' + str(i) + ', total is ' + str(total))
logging.debug('End of factorial(%s%%)' % (n))
return total
print(factorial(5))
logging.debug('End of program')

 debug() function will call basicConfig(), and a line of information will be printed.
 This information will be in the format we specified in basicConfig() and will include
the messages we passed to debug().
Output:
[Type the document title]

2015-05-23 16:20:12,664 - DEBUG - Start of program

2015-05-23 16:20:12,664 - DEBUG - Start of factorial(5)
2015-05-23 16:20:12,665 - DEBUG - i is 0, total is 0
2015-05-23 16:20:12,668 - DEBUG - i is 1, total is 0
2015-05-23 16:20:12,670 - DEBUG - i is 2, total is 0
2015-05-23 16:20:12,673 - DEBUG - i is 3, total is 0
2015-05-23 16:20:12,675 - DEBUG - i is 4, total is 0
2015-05-23 16:20:12,678 - DEBUG - i is 5, total is 0
2015-05-23 16:20:12,680 - DEBUG - End of factorial(5)
0
2015-05-23 16:20:12,684 - DEBUG - End of program

 The factorial() function is returning 0 as the factorial of 5, which isn’t right.

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 24

Introduction to Python Programming (BPLCK105B) Module 4

 The for loop should be multiplying the value in total by the numbers from 1 to 5. But
the log messages displayed by logging.debug() show that the i variable is starting at 0
instead of 1.
 Since zero times anything is zero, the rest of the iterations also have the wrong value
for total
 Logging messages provide a trail of breadcrumbs that can help you figure out when
things started to go wrong.
 Change the for i in range(n + 1): line to for i in range(1, n + 1):, and run the program
again

Output
[Type the document title]

2015-05-23 17:13:40,650 - DEBUG - Start of program

2015-05-23 17:13:40,651 - DEBUG - Start of factorial(5)
2015-05-23 17:13:40,651 - DEBUG - i is 1, total is 1
2015-05-23 17:13:40,654 - DEBUG - i is 2, total is 2
2015-05-23 17:13:40,656 - DEBUG - i is 3, total is 6
2015-05-23 17:13:40,659 - DEBUG - i is 4, total is 24
2015-05-23 17:13:40,661 - DEBUG - i is 5, total is 120
2015-05-23 17:13:40,661 - DEBUG - End of factorial(5)
120
2015-05-23 17:13:40,666 - DEBUG - End of program

Logging Levels
 Logging levels provide a way to categorize your log messages by importance. There
are five logging levels
 Messages can be logged at each level using a different logging function.

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 25

Introduction to Python [Type the document
Programming title]
(BPLCK105B) Module 4

Level Logging Function Description

DEBUG logging.debug() The lowest level. Used for small details.
Usually you care about these messages
only when diagnosing problems.
INFO logging.info() Used to record information on general
events in your program or confirm that
things are working at their point in the
program.
WARNING l logging.warning() Used to indicate a potential problem that
doesn’t prevent the program from
working but might do so in the future.

ERROR logging.error() Used to record an error that caused the

program to fail to do something.
CRITICAL logging.critical() The highest level. Used to indicate a fatal
error that has caused or is about to cause
the program to stop running entirely.
Table 10-1: Logging Levels in Python

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 26

Introduction to Python [Type the document
Programming title]
(BPLCK105B) Module 4

>>> import logging

>>> logging.basicConfig(level=logging.DEBUG, format=' %(asctime)s -
%(levelname)s - %(message)s')
>>> logging.debug('Some debugging details.')
2015-05-18 19:04:26,901 - DEBUG - Some debugging details.
>>> logging.info('The logging module is working.')
2015-05-18 19:04:35,569 - INFO - The logging module is working.
>>> logging.warning('An error message is about to be logged.')
2015-05-18 19:04:56,843 - WARNING - An error message is about to be
logged.
>>> logging.error('An error has occurred.')
2015-05-18 19:05:07,737 - ERROR - An error has occurred.
>>> logging.critical('The program is unable to recover!')
2015-05-18 19:05:45,794 - CRITICAL - The program is unable to recover!

Disabling Logging

 The logging.disable() function disables these so that you don’t have to go into your
program and remove all the logging calls by hand.
 pass logging.disable() a logging level, and it will suppress all log messages at that
level or lower
[Type the document title]

>>> import logging

>>> logging.basicConfig(level=logging.INFO, format=' %(asctime)s -%(levelname)s -

%(message)s')

>>> logging.critical('Critical error! Critical error!')

2015-05-22 11:10:48,054 - CRITICAL - Critical error! Critical error!
>>> logging.disable(logging.CRITICAL)
>>> logging.critical('Critical error! Critical error!')
>>> logging.error('Error! Error!')

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 27

Introduction to Python Programming (BPLCK105B) Module 4

 Since logging.disable() will disable all messages after it, you will probably want to
add it near the import logging line of code in your program

Logging to a File
 Instead of displaying the log messages to the screen, you can write them to a text file.
The logging.basicConfig() function takes a filename keyword argument,
[Type the document title]

import logging
logging.basicConfig(filename='myProgramLog.txt',level=logging.DEBUG,
format=' %(asctime)s - %(levelname)s - %(message)s')

5. IDLE ’s Debugger
 The debugger is a feature of IDLE that allows you to execute your program one line
at a time.
 The debugger will run a single line of code and then wait for you to tell it to continue
 To enable IDLE’s debugger, click Debug4Debugger in the interactive shell window.
 When the Debug Control window appears, select all four of the Stack, Locals, Source,
and Globals checkboxes so that the window shows the full set of debug information
 While the Debug Control window is displayed, any time you run a program from the
file editor
 debugger will pause execution before the first instruction and display the following:
 The line of code that is about to be executed
 A list of all local variables and their values
 A list of all global variables and their values

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 28

Introduction to Python Programming (BPLCK105B) Module 4

Figure: The Debug Control window

 You’ll notice that in the list of global variables there are several variables you haven’t
defined, such as __builtins__, __doc__, __file__, and so on. These are variables that
Python automatically sets whenever it runs a program.
 The program will stay paused until you press one of the five buttons in the Debug
Control window: Go, Step, Over, Out, or Quit.
Go
 Clicking the Go button will cause the program to execute normally until it terminates
or reaches a breakpoint
 If you are done debugging and want the program to continue normally, click the Go
button.
Step
 Clicking the Step button will cause the debugger to execute the next line of code and
then pause again
 The Debug Control window’s list of global and local variables will be updated if their
values change.
 If the next line of code is a function call, the debugger will “step into” that function
and jump to the first line of code of that function.

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 29

Introduction to Python Programming (BPLCK105B) Module 4

Over
 Clicking the Over button will execute the next line of code, similar to the Step button.
 The Over button will “step over” the code in the function. The function’s code will be
executed at full speed, and the debugger will pause as soon as the function call
returns.
 For example, if the next line of code is a print() call, you don’t really care about code
inside the built-in print() function; you just want the string you pass it printed to the
screen.
Quit
 If you want to stop debugging entirely and not bother to continue executing the rest of
the program, click the Quit button
 The Quit button will immediately terminate the program. If you want to run your
program normally again, select Debug4Debugger again to disable the debugger.

Debugging a Number Adding Program

[Type the document title]

print('Enter the first number to add:')

first = input()
print('Enter the second number to add:')
second = input()
print('Enter the third number to add:')
third = input()
print('The sum is ' + first + second + third)

[Type the document title]

Save it as buggyAddingProgram.py and run it first without the debugger enabled.

Enter the first number to add:

5
Enter the second number to add:
3
Enter the third number to add:
42
The sum is 5342

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 30

Introduction to Python Programming (BPLCK105B) Module 4

 The program hasn’t crashed, but the sum is obviously wrong. Let’s enable the Debug
Control window and run it again, this time under the debugger
 When you press F5 or select Run4Run Module (with Debug4Debugger enabled and
all four checkboxes on the Debug Control window checked), the program starts in a
paused state on line 1.
 The debugger will always pause on the line of code it is about to execute.

Figure The Debug Control window when the program first starts under the debugger

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 31

Introduction to Python Programming (BPLCK105B) Module 4

 Click the Over button once to execute the first print() call. You should use Over
instead of Step here, since you don’t want to step into the code for the print() function.
 The Debug Control window will update to line 2, and line 2 in the file editor window
will be highlighted.

Figure: The Debug Control window after clicking Over

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 32

Introduction to Python Programming (BPLCK105B) Module 4

 Click Over again to execute the input() function call, and the buttons in the Debug
Control window will disable themselves while IDLE waits for you to type something
for the input() call into the interactive shell window.
 Enter 5 and press Return. The Debug Control window buttons will be reenabled.
 Keep clicking Over, entering 3 and 42 as the next two numbers, until the debugger is
on line 7, the final print() call in the program
 Globals section that the first, second, and third variables are set to string values '5', '3',
and '42' instead of integer values 5, 3, and 42.
 When the last line is executed, these strings are concatenated instead of added
together, causing the bug.

Figure The Debug Control window on the last line. The variables are set to strings, causing
the bug.

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 33

Introduction to Python Programming (BPLCK105B) Module 4

Breakpoints

 A breakpoint can be set on a specific line of code and forces the debugger to mpause
whenever the program execution reaches that line.
 Open a new file editor window and enter the following program, which simulates flipping a
coin 1,000 times. [Type the document title]

import random
heads = 0
for i in range(1, 1001):
if random.randint(0, 1) == 1:
heads = heads + 1
if i == 500:
print('Halfway done!')
print('Heads came up ' + str(heads) + ' times.')

 The random.randint(0, 1) call u will return 0 half of the time and 1 the other half of
the time.
 This can be used to simulate a 50/50 coin flip where 1 represents heads.

Output: [Type the document title]

Halfway done!
Heads came up 490 times.
 If you ran this program under the debugger, you would have to click the Over button
thousands of times before the program terminated.
 If you were interested in the value of heads at the halfway point of the program’s
execution, when 500 of 1000 coin flips have been completed, you could instead just
set a breakpoint on the line print('Halfway done!')
 To set a breakpoint, right-click the line in the file editor and select Set Breakpoint,

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 34

Introduction to Python Programming (BPLCK105B) Module 4

Figure : Setting a breakpoint

Manjushree T L, Asst Prof., Dept of ISE, SVIT Page 35