Advance Praise for Head First Python

“Head First Python is a great introduction to not just the Python language, but Python as it’s used in the
real world. The book goes beyond the syntax to teach you how to create applications for Android phones,
Google’s App Engine, and more.”
— David Griffiths, author and Agile coach

“Where other books start with theory and progress to examples, Head First Python jumps right in with code
and explains the theory as you read along. This is a much more effective learning environment, because
it engages the reader to do from the very beginning. It was also just a joy to read.  It was fun without
being flippant and informative without being condescending. The breadth of examples and explanation
covered the majority of what you’ll use in your job every day. I’ll recommend this book to anyone
starting out on Python.”
— Jeremy Jones, coauthor of Python for Unix and Linux System Administration

“Head First Python is a terrific book for getting a grounding in a language that is increasing in relevance
day by day.”
— Phil Hartley, University of Advancing Technology
Praise for other Head First books
“Kathy and Bert’s Head First Java transforms the printed page into the closest thing to a GUI you’ve ever
seen. In a wry, hip manner, the authors make learning Java an engaging ‘what’re they gonna do next?’
experience.”
— Warren Keuffel, Software Development Magazine

“Beyond the engaging style that drags you forward from know-nothing into exalted Java warrior status, Head
First Java covers a huge amount of practical matters that other texts leave as the dreaded ‘exercise for the
reader.…’ It’s clever, wry, hip and practical—there aren’t a lot of textbooks that can make that claim and live
up to it while also teaching you about object serialization and network launch protocols.”
— Dr. Dan Russell, Director of User Sciences and Experience Research
IBM Almaden Research Center (and teaches Artificial Intelligence at
Stanford University)

“It’s fast, irreverent, fun, and engaging. Be careful—you might actually learn something!”
— Ken Arnold, former Senior Engineer at Sun Microsystems
Coauthor (with James Gosling, creator of Java), The Java Programming
Language

“I feel like a thousand pounds of books have just been lifted off of my head.”
— Ward Cunningham, inventor of the Wiki and founder of the Hillside Group

“Just the right tone for the geeked-out, casual-cool guru coder in all of us. The right reference for practi-
cal development strategies—gets my brain going without having to slog through a bunch of tired, stale
professor­-speak.”
— Travis Kalanick, founder of Scour and Red Swoosh
Member of the MIT TR100

“There are books you buy, books you keep, books you keep on your desk, and thanks to O’Reilly and the
Head First crew, there is the penultimate category, Head First books. They’re the ones that are dog-eared,
mangled, and carried everywhere. Head First SQL is at the top of my stack. Heck, even the PDF I have
for review is tattered and torn.”
— Bill Sawyer, ATG Curriculum Manager, Oracle

“This book’s admirable clarity, humor and substantial doses of clever make it the sort of book that helps
even non-programmers think well about problem-solving.”
— Cory Doctorow, co-editor of Boing Boing
Author, Down and Out in the Magic Kingdom
and Someone Comes to Town, Someone Leaves Town
 Praise for other Head First books
“I received the book yesterday and started to read it…and I couldn’t stop. This is definitely très ‘cool.’ It
is fun, but they cover a lot of ground and they are right to the point. I’m really impressed.”
— Erich Gamma, IBM Distinguished Engineer, and coauthor of Design Patterns

“One of the funniest and smartest books on software design I’ve ever read.”
— Aaron LaBerge, VP Technology, ESPN.com

“What used to be a long trial and error learning process has now been reduced neatly into an engaging
paperback.”
— Mike Davidson, CEO, Newsvine, Inc.

“Elegant design is at the core of every chapter here, each concept conveyed with equal doses of
pragmatism and wit.”
— Ken Goldstein, Executive Vice President, Disney Online

“I ♥ Head First HTML with CSS & XHTML—it teaches you everything you need to learn in a ‘fun-coated’
format.”
— Sally Applin, UI Designer and Artist

“Usually when reading through a book or article on design patterns, I’d have to occasionally stick myself
in the eye with something just to make sure I was paying attention. Not with this book. Odd as it may
sound, this book makes learning about design patterns fun.

“While other books on design patterns are saying ‘Bueller…Bueller…Bueller…’ this book is on the float
belting out ‘Shake it up, baby!’”
— Eric Wuehler

“I literally love this book. In fact, I kissed this book in front of my wife.”
— Satish Kumar
Other related books from O’Reilly
Learning Python
Programming Python
Python in a Nutshell
Python Cookbook
Python for Unix and Linux System Administration

Other books in O’Reilly’s Head First series

Head First Algebra
Head First Ajax
Head First C#, Second Edition
Head First Design Patterns
Head First EJB
Head First Excel
Head First 2D Geometry
Head First HTML with CSS & XHTML
Head First iPhone Development
Head First Java
Head First JavaScript
Head First Object-Oriented Analysis & Design (OOA&D)
Head First PHP & MySQL
Head First Physics
Head First PMP, Second Edition
Head First Programming
Head First Rails
Head First Servlets & JSP, Second Edition
Head First Software Development
Head First SQL
Head First Statistics
Head First Web Design
Head First WordPress
Head First Python

Wouldn’t it be dreamy if there

were a Python book that didn’t
make you wish you were anywhere
other than stuck in front of your
computer writing code? I guess it’s
just a fantasy...

Paul Barry

Beijing • Cambridge • Farnham • Kln • Sebastopol • Tokyo

Head First Python
by Paul Barry

Copyright © 2011 Paul Barry. All rights reserved.

Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly Media books may be purchased for educational, business, or sales promotional use. Online editions
are also available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/
institutional sales department: (800) 998-9938 or corporate@oreilly.com.

Series Creators: Kathy Sierra, Bert Bates Deirdre

Editor: Brian Sawyer
Cover Designer: Karen Montgomery
Production Editor: Rachel Monaghan
Proofreader: Nancy Reinhardt
Indexer: Angela Howard
Page Viewers: Deirdre, Joseph, Aaron, and Aideen
Aideen
Printing History:
November 2010: First Edition.
Joseph

Aaron
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. The Head First series designations,
Head First Python, and related trade dress are trademarks of O’Reilly Media, Inc.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as
trademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of a trademark
claim, the designations have been printed in caps or initial caps.

While every precaution has been taken in the preparation of this book, the publisher and the author assume no
responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.

No athletes were pushed too hard in the making of this book.

TM

This book uses RepKover™,  a durable and flexible lay-flat binding.

ISBN: 978-1-449-38267-4
[M]
I dedicate this book to all those generous people in the Python
community who have helped to make this great little language the
first-rate programming technology it is.
And to those that made learning Python and its technologies just
complex enough that people need a book like this to learn it.
the author

Author of Head First Python

Paul Barry recently worked out that he has been

programming for close to a quarter century, a fact that came
as a bit of a shock. In that time, Paul has programmed in
lots of different programming languages, lived and worked
Paul in two countries on two continents, got married, had three
kids (well…his wife Deirdre actually had them, but Paul was
there), completed a B.Sc. and M.Sc. in Computing, written or
cowritten three other books, as well as a bunch of technical
articles for Linux Journal (where he’s a Contributing Editor).
When Paul first saw Head First HTML with CSS & XHTML,
he loved it so much he knew immediately that the Head First
approach would be a great way to teach programming. He
was only too delighted then, together with David Griffiths, to
create Head First Programming in an attempt to prove his hunch
correct.
Paul’s day job is working as a lecturer at The Institute of
Technology, Carlow, in Ireland. As part of the Department
of Computing and Networking, Paul gets to spend his day
exploring, learning, and teaching programming technologies
to his students, including Python.
Paul recently completed a post-graduate certificate in
Learning and Teaching and was more than a bit relieved to
discover that most of what he does conforms to current third-
level best practice.

viii
 table of contents

Table of Contents (Summary)

Intro xxiii
1 Meet Python: Everyone Loves Lists 1
2 Sharing Your Code: Modules of Functions 33
3 Files and Exceptions: Dealing with Errors 73
4 Persistence: Saving Data to Files 105
5 Comprehending Data: Work That Data! 139
6 Custom Data Objects: Bundling Code with Data 173
7 Web Development: Putting It All Together 213
8 Mobile App Development: Small Devices 255
9 Manage Your Data: Handling Input 293
10 Scaling Your Webapp: Getting Real 351
11 Dealing with Complexity: Data Wrangling 397
i Leftovers: The Top Ten Things (We Didn’t Cover) 435

Table of Contents (the real thing)

Intro
Your brain on Python.  Here you are trying to learn something, while
here your brain is doing you a favor by making sure the learning doesn’t stick.
Your brain’s thinking, “Better leave room for more important things, like which
wild animals to avoid and whether naked snowboarding is a bad idea.” So how
do you trick your brain into thinking that your life depends on knowing Python?

Who is this book for? xxiv

We know what you’re thinking xxv
Metacognition xxvii
Bend your brain into submission xxix
Read me xxx
The technical review team xxxii
Acknowledgments xxxiii

ix
table of contents

meet python

1
Everyone loves lists
You’re asking one question: “What makes Python different?”
The short answer is: lots of things. The longer answers starts by stating that there’s
lots that’s familiar, too. Python is a lot like any other general-purpose programming
language, with statements, expressions, operators, functions, modules, methods,
and classes. All the usual stuff, really. And then there’s the other stuff Python provides
that makes the programmer’s life—your life—that little bit easier. You’ll start your tour
of Python by learning about lists. But, before getting to that, there’s another important
question that needs answering…

What’s to like about Python? 2

Install Python 3 3
Use IDLE to help learn Python 4
Work effectively with IDLE 5
Deal with complex data 6
Create simple Python lists 7
Lists are like arrays 9
Add more data to your list 11
Work with your list data 15
For loops work with lists of any size 16
Store lists within lists 18
Check a list for a list 20
Complex data is hard to process 23
Handle many levels of nested lists 24
Don’t repeat code; create a function 28
Create a function in Python 29
Recursion to the rescue! 31
Your Python Toolbox 32

& Terry Gilliam, 91 mins

The Holy Grail, 1975, Terry Jones
Graham Chapman Terry Jones
Mi cha el Pal in, Joh n Cleese, Terry Gilliam, Eric Idle &

x
 table of contents

sharing your code

2
Modules of functions
Reusable code is great, but a shareable module is better. 
By sharing your code as a Python module, you open up your code to the entire Python
community…and it’s always good to share, isn’t it? In this chapter, you’ll learn how to
create, install, and distribute your own shareable modules. You’ll then load your module
onto Python’s software sharing site on the Web, so that everyone can benefit from your
work. Along the way, you’ll pick up a few new tricks relating to Python’s functions, too.

It’s too good not to share 34

Turn your function into a module 35
Modules are everywhere 36
Comment your code 37
Prepare your distribution 40
Build your distribution 41
A quick review of your distribution 42
Import a module to use it 43
Python’s modules implement namespaces 45
Register with the PyPI website 47
Upload your code to PyPI 48
Welcome to the PyPI community 49
Control behavior with an extra argument 52
Before your write new code, think BIF 53
Python tries its best to run your code 57
Trace your code 58
Work out what’s wrong 59
Update PyPI with your new code 60
You’ve changed your API 62
nester Use optional arguments 63
Your module supports both APIs 65
Your API is still not right 66
Your module’s reputation is restored 70
nester.py
Your Python Toolbox 71
setup.py

xi
table of contents

files and exceptions

3
Dealing with errors
It’s simply not enough to process your list data in your code.
You need to be able to get your data into your programs with ease, too. It’s no surprise
then that Python makes reading data from files easy. Which is great, until you
consider what can go wrong when interacting with data external to your programs…
and there are lots of things waiting to trip you up! When bad stuff happens, you need a
strategy for getting out of trouble, and one such strategy is to deal with any exceptional
situations using Python’s exception handling mechanism showcased in this chapter.

Data is external to your program 74

It’s all lines of text 75
Take a closer look at the data 77
Know your data 79
Know your methods and ask for help 80
Know your data (better) 82
Two very different approaches 83
Add extra logic 84
Handle exceptions 88
Try first, then recover 89
Identify the code to protect 91
Take a pass on the error 93
What about other errors? 96
Add more error-checking code… 97
…Or add another level of exception handling 98
So, which approach is best? 99
You’re done…except for one small thing 101
Be specific with your exceptions 102
split(beans) Your Python Toolbox 103

xii
 table of contents

persistence

4
Saving data to files
It is truly great to be able to process your file-based data.
But what happens to your data when you’re done? Of course, it’s best to save your
data to a disk file, which allows you to use it again at some later date and time. Taking
your memory-based data and storing it to disk is what persistence is all about. Python
supports all the usual tools for writing to files and also provides some cool facilities for
efficiently storing Python data.

Programs produce data 106

Open your file in write mode 110
Files are left open after an exception! 114
Extend try with finally 115
Knowing the type of error is not enough 117
Use with to work with files 120
Default formats are unsuitable for files 124
Why not modify print_lol()? 126
Pickle your data 132
Save with dump and restore with load 133
Generic file I/O with pickle is the way to go! 137
Your Python Toolbox 138

['Is this the right room for an

argument?', "No you haven't!",
'When?', "No you didn't!", "You
didn't!", 'You did not!', 'Ah!
(taking out his wallet and paying)
[‘Is this the right room
for an argument?’, “No
Just the five minutes.', 'You most
certainly did not!', "Oh no you
you haven’t!”, ‘When?’,
“No you didn’t!”, “You
didn’t!”, ‘You did not!’,
‘Ah! (taking out his wallet
and paying) Just the five

didn't!", "Oh no you didn't!", "Oh

minutes.’, ‘You most
certainly did not!’, “Oh
no you didn’t!”, “Oh no
you didn’t!”, “Oh look,
this isn’t an argument!”,

look, this isn't an argument!",

“No it isn’t!”, “It’s
just contradiction!”,
‘It IS!’, ‘You just
contradicted me!’, ‘You

"No it isn't!", "It's just

DID!’, ‘You did just
then!’, ‘(exasperated)
Oh, this is futile!!’,
‘Yes it is!’]

contradiction!", 'It IS!', 'You

just contradicted me!', 'You DID!',
'You did just then!', '(exasperated)
Oh, this is futile!!', 'Yes it
is!']

xiii
table of contents

comprehending data

5
Work that data!
Data comes in all shapes and sizes, formats and encodings.
To work effectively with your data, you often have to manipulate and transform it into a
common format to allow for efficient processing, sorting, and storage. In this chapter,
you’ll explore Python goodies that help you work your data up into a sweat, allowing
you to achieve data-munging greatness.

Coach Kelly needs your help 140

Sort in one of two ways 144
The trouble with time 148
Comprehending lists 155
Iterate to remove duplicates 161
Remove duplicates with sets 166
Your Python Toolbox 172

This chapter’s
guaranteed to give you
a workout!

xiv
 table of contents

custom data objects

6
Bundling code with data
It’s important to match your data structure choice to your data. 
And that choice can make a big difference to the complexity of your code. In Python,
although really useful, lists and sets aren’t the only game in town. The Python dictionary
lets you organize your data for speedy lookup by associating your data with names, not
numbers. And when Python’s built-in data structures don’t quite cut it, the Python class
statement lets you define your own. This chapter shows you how.

Coach Kelly is back (with a new file format) 174

Use a dictionary to associate data 178
Bundle your code and its data in a class 189
Define a class 190
Use class to define classes 191
The importance of self 192
Every method’s first argument is self 193
Inherit from Python’s built-in list 204
Coach Kelly is impressed 211
Your Python Toolbox 212

The Object
Factory

xv
table of contents

web development

7
Putting it all together
Sooner or later, you’ll want to share your app with lots of people. 
You have many options for doing this. Pop your code on PyPI, send out lots of emails, put
your code on a CD or USB, or simply install your app manually on the computers of those
people who need it. Sounds like a lot of work…not to mention boring. Also, what happens
when you produce the next best version of your code? What happens then? How do
you manage the update? Let’s face it: it’s such a pain that you’ll think up really creative
excuses not to. Luckily, you don’t have to do any of this: just create a webapp instead. And,
as this chapter demonstrates, using Python for web development is a breeze.

It’s good to share 214

You can put your program on the Web 215
What does your webapp need to do? 218
Design your webapp with MVC 221
Model your data 222
View your interface 226
Control your code 234
CGI lets your web server run programs 235
Display the list of athletes 236
The dreaded 404 error! 242
Create another CGI script 244
Enable CGI tracking to help with errors 248
A small change can make all the difference 250
Your webapp’s a hit! 252
Your Python Toolbox 253

xvi
 table of contents

mobile app development

8
Small devices
Putting your data on the Web opens up all types of possibilities. 
Not only can anyone from anywhere interact with your webapp, but they are increasingly
doing so from a collection of diverse computing devices: PCs, laptops, tablets, palmtops,
and even mobile phones. And it’s not just humans interacting with your webapp that
you have to support and worry about: bots are small programs that can automate web
interactions and typically want your data, not your human-friendly HTML. In this chapter,
you exploit Python on Coach Kelly’s mobile phone to write an app that interacts with your
webapp’s data.

The world is getting smaller 256

Coach Kelly is on Android 257
Don’t worry about Python 2 259
Set up your development environment 260
Configure the SDK and emulator 261
Install and configure Android Scripting 262
Add Python to your SL4A installation 263
Test Python on Android 264
Define your app’s requirements 266
The SL4A Android API 274
Select from a list on Android 278
The athlete’s data CGI script 281
The data appears to have changed type 284
JSON can’t handle your custom datatypes 285
Run your app on a real phone 288
Configure AndFTP 289
The coach is thrilled with his app 290
Your Python Toolbox 291

xvii
table of contents

manage your data

9
Handling input
The Web and your phone are not just great ways to display data. 
They are also great tools to for accepting input from your users. Of course, once your
webapp accepts data, it needs to put it somewhere, and the choices you make when
deciding what and where this “somewhere” is are often the difference between a webapp
that’s easy to grow and extend and one that isn’t. In this chapter, you’ll extend your
webapp to accept data from the Web (via a browser or from an Android phone), as well
as look at and enhance your back-end data-management services.

Your athlete times app has gone national 294

Use a form or dialog to accept input 295
Create an HTML form template 296
The data is delivered to your CGI script 300
Ask for input on your Android phone 304
It’s time to update your server data 308
Avoid race conditions 309
You need a better data storage mechanism 310
Use a database management system 312
Python includes SQLite 313
Exploit Python’s database API 314
The database API as Python code 315
A little database design goes a long way 316
Define your database schema 317
What does the data look like? 318
Transfer the data from your pickle to SQLite 321
What ID is assigned to which athlete? 322
Insert your timing data 323
SQLite data management tools 326
Integrate SQLite with your existing webapp 327
You still need the list of names 332
Get an athlete’s details based on ID 333
You need to amend your Android app, too 342
Update your SQLite-based athlete data 348
The NUAC is over the moon! 349
Your Python Toolbox 350

xviii
 table of contents

scaling your webapp

10
Getting real
The Web is a great place to host your app…until things get real. 
Sooner or later, you’ll hit the jackpot and your webapp will be wildly successful. When
that happens, your webapp goes from a handful of hits a day to thousands, possibly ten
of thousands, or even more. Will you be ready? Will your web server handle the load?
How will you know? What will it cost? Who will pay? Can your data model scale to
millions upon millions of data items without slowing to a crawl? Getting a webapp up and
running is easy with Python and now, thanks to Google App Engine, scaling a Python
webapp is achievable, too.

There are whale sightings everywhere 352

The HFWWG needs to automate 353
Build your webapp with Google App Engine 354
Download and install App Engine 355
Make sure App Engine is working 356
App Engine uses the MVC pattern 359
Model your data with App Engine 360
What good is a model without a view? 363
Use templates in App Engine 364
Django’s form validation framework 368
Check your form 369
Controlling your App Engine webapp 370
Restrict input by providing options 376
Meet the “blank screen of death” 378
Process the POST within your webapp 379
Put your data in the datastore 380
Don’t break the “robustness principle” 384
Accept almost any date and time 385
It looks like you’re not quite done yet 388
Sometimes, the tiniest change can make all the difference… 389
Capture your user’s Google ID, too 390
Deploy your webapp to Google’s cloud 391
Your HFWWG webapp is deployed! 394
Your Python Toolbox 395

xix
table of contents

dealing with complexity

11
Data wrangling
It’s great when you can apply Python to a specific domain area. 
Whether it’s web development, database management, or mobile apps, Python helps
you get the job done by not getting in the way of you coding your solution. And then
there’s the other types of problems: the ones you can’t categorize or attach to a domain.
Problems that are in themselves so unique you have to look at them in a different, highly
specific way. Creating bespoke software solutions to these type of problems is an area
where Python excels. In this, your final chapter, you’ll stretch your Python skills to the
limit and solve problems along the way.
What’s a good time goal for the next race? 398
So…what’s the problem? 400
Start with the data 401
Store each time as a dictionary 407
Dissect the prediction code 409
Get input from your user 413
Getting input raises an issue… 414
Search for the closest match 416
The trouble is with time 418
The time-to-seconds-to-time module 419
The trouble is still with time… 422
Port to Android 424
Your Android app is a bunch of dialogs 425
Put your app together… 429
Your app’s a wrap! 431
Your Python Toolbox 432

xx
 table of contents

leftovers

i
The Top Ten Things (we didn’t cover)
You’ve come a long way.
But learning about Python is an activity that never stops. The more Python you code,
the more you’ll need to learn new ways to do certain things. You’ll need to master new
tools and new techniques, too. There’s just not enough room in this book to show you
everything you might possibly need to know about Python. So, here’s our list of the top
ten things we didn’t cover that you might want to learn more about next.

#1: Using a “professional” IDE 436

#2: Coping with scoping 437
#3: Testing 438
#4: Advanced language features 439
#5: Regular expressions 440
#6: More on web frameworks 441
#7: Object relational mappers and NoSQL 442
#8: Programming GUIs 443
#9: Stuff to avoid 444
#10: Other books 445

xxi
how to use this book
Intro
I can’t believe
they put that in a
Python book.

ning question:
In this section, we answer thein bur
Python book?”
“So why DID they put that a

xxiii
how to use this book

Who is this book for?

If you can answer “yes” to all of these:

1 Do you already know how to program in another

programming language?

2 Do you wish you had the know-how to program Python,

add it to your list of tools, and make it do new things?

3 Do you prefer actually doing things and applying the stuff

you learn over listening to someone in a lecture rattle on
for hours on end?

this book is for you.

Who should probably back away from this book?

If you can answer “yes” to any of these:

1 Do you already know most of what you need to know to

program with Python?

2 Are you looking for a reference book to Python, one that

covers all the details in excruciating detail?

3 Would you rather have your toenails pulled out by 15

screaming monkeys than learn something new? Do you
believe a Python book should cover everything and if it
bores the reader to tears in the process then so much the
better?

this book is not for you.

[Note from marketing: this book

is for anyone with a credit card…
we’ll accept a check, too.]

xxiv   intro
 the intro

We know what you’re thinking

“How can this be a serious Python book?”
“What’s with all the graphics?”
“Can I actually learn it this way?”
Your bra
THIS is imin thinks
portant.
We know what your brain is thinking
Your brain craves novelty. It’s always searching, scanning, waiting for something
unusual. It was built that way, and it helps you stay alive.
So what does your brain do with all the routine, ordinary, normal things
you encounter? Everything it can to stop them from interfering with the
brain’s real job—recording things that matter. It doesn’t bother saving the
boring things; they never make it past the “this is obviously not important”
filter.
How does your brain know what’s important? Suppose you’re out for a day
hike and a tiger jumps in front of you, what happens inside your head and
body?
Neurons fire. Emotions crank up. Chemicals surge. Great. Only 450
more dull, dry,
And that’s how your brain knows… boring pages.
This must be important! Don’t forget it!
hinks
But imagine you’re at home, or in a library. It’s a safe, warm, tiger‑free zone. Your brain t th
You’re studying. Getting ready for an exam. Or trying to learn some tough THIS isn’t w
or
technical topic your boss thinks will take a week, ten days at the most. saving.
Just one problem. Your brain’s trying to do you a big favor. It’s trying to
make sure that this obviously non-important content doesn’t clutter up scarce
resources. Resources that are better spent storing the really big things.
Like tigers. Like the danger of fire. Like how you should never have
posted those “party” photos on your Facebook page. And there’s no
simple way to tell your brain, “Hey brain, thank you very much, but
no matter how dull this book is, and how little I’m registering on the
emotional Richter scale right now, I really do want you to keep this
stuff around.”

you are here 4   xxv

how to use this book

t” reader as a learner.
We think of a “Head Firs
then make sure
me th ing ? First, you have to get it,
So what does it take
to lea rn so on the latest
t pu sh ing fac ts int o your head. Based
It’s not abou onal psychology, lea
rning
you don’t forget it. urob iology, and educati
e sc ien ce, ne
research in cognitiv turns your brain on
.
re th an text on a page. We know what
takes a lot mo
ciples:
First lear ning prin
Some of the Head
ch
, and make learning mu
are far mo re me mo rable than words alone ng s mo re
ages also makes thi
Make it visual. Im in rec all an d transfer studies). It
89% impro vem en t r tha n on
more effective (up to relate to, rathe
th in or ne ar the gr aphics they the
understandable. Put
the word s wi ve problems related to
lea rne rs wil l be up to twice as likely to sol
er page, and
the bottom or on anoth
content. rformed up
t studies, students pe
d pe rs on al ized style. In recen erson,
Use a conver satio
na l an reader, using a first-p
ts if the con ten t spoke directly to the language.
to 40% better on post-
learning tes lec turing. Use cas l
ua
a for ma l ton e. Tell stories instead of
conversational style rat
her than tak ing ulating dinner rty
pa
wo uld you pa y mo re attention to: a stim
seriously. Which
Don’t take yourself too
e?
companion or a lec tur neurons,
s you actively flex your
k mo re de ep ly. In other words, unles d ins pired to
Get the lear ner to
thin ged, curious, an
. A rea de r ha s to be motivated, enga
nothing much happen
s in your he ad t, you need challenges,
d ge ne rat e ne w knowledge. And for tha
conclusions, an of the brain and
solve problems, draw s, an d act ivit ies that involve both sides
provoking quest ion
exercises, and thought-
multiple senses. s but I can’t
really want to learn thi
ad er ’s at te nt io n. We’ve all had the “I the ordinary,
Get—and keep—t
he re that are out of
ur bra in pa ys attention to things be
stay awake past page
one” experi en ce. Yo l topic doesn have to
’t
exp ect ed . Lea rni ng a new, tough, technica
-catching, un
interesting, strange, eye if it’s not.
l lea rn much more quick ly
boring. Your brain wil gely dependent
t yo ur ab ilit y to rem ember something is lar
tha something.
ions. We now know ember when you feel
Touch their emot r wh at you care about. You rem
em be
on its emotional conten
t. Yo u rem ’re talking emoti s like
on
sto rie s ab ou t a boy and his dog. We e,
No, we’re not talking
heart‑wren ch ing when you solve a puzzl
” , an d the fee lin g of “I Rule!” that comes hn ica l
, “what the…? t “I’m more tec
surprise, curiosity, fun ha rd, or rea lize yo u know something tha
ody else thinks is
learn something everyb
en eering doesn’t.
gin
than thou” Bob from

xxvi   intro
 the intro

Metacognition: thinking about thinking

If you really want to learn, and you want to learn more quickly and more I wonder how
deeply, pay attention to how you pay attention. Think about how you think. I can trick my brain
Learn how you learn. into remembering
this stuff...
Most of us did not take courses on metacognition or learning theory when we
were growing up. We were expected to learn, but rarely taught to learn.
But we assume that if you’re holding this book, you really want to learn how
to design user-friendly websites. And you probably don’t want to spend a lot
of time. If you want to use what you read in this book, you need to remember
what you read. And for that, you’ve got to understand it. To get the most from
this book, or any book or learning experience, take responsibility for your brain.
Your brain on this content.
The trick is to get your brain to see the new material you’re learning as
Really Important. Crucial to your well‑being. As important as a tiger.
Otherwise, you’re in for a constant battle, with your brain doing its best to
keep the new content from sticking.

So just how DO you get your brain to treat

programming like it was a hungry tiger?
There’s the slow, tedious way, or the faster, more effective way. The
slow way is about sheer repetition. You obviously know that you are able to learn
and remember even the dullest of topics if you keep pounding the same thing into your
brain. With enough repetition, your brain says, “This doesn’t feel important to him, but he
keeps looking at the same thing over and over and over, so I suppose it must be.”
The faster way is to do anything that increases brain activity, especially different
types of brain activity. The things on the previous page are a big part of the solution,
and they’re all things that have been proven to help your brain work in your favor. For
example, studies show that putting words within the pictures they describe (as opposed to
somewhere else in the page, like a caption or in the body text) causes your brain to try to
makes sense of how the words and picture relate, and this causes more neurons to fire.
More neurons firing = more chances for your brain to get that this is something worth
paying attention to, and possibly recording.
A conversational style helps because people tend to pay more attention when they
perceive that they’re in a conversation, since they’re expected to follow along and hold up
their end. The amazing thing is, your brain doesn’t necessarily care that the “conversation”
is between you and a book! On the other hand, if the writing style is formal and dry, your
brain perceives it the same way you experience being lectured to while sitting in a roomful
of passive attendees. No need to stay awake.
But pictures and conversational style are just the beginning…

you are here 4   xxvii

how to use this book

Here’s what WE did:

We used pictures, because your brain is tuned for visuals, not text. As far as your brain’s
concerned, a picture really is worth a thousand words. And when text and pictures work
together, we embedded the text in the pictures because your brain works more effectively
when the text is within the thing the text refers to, as opposed to in a caption or buried in the
text somewhere.
We used redundancy, saying the same thing in different ways and with different media types,
and multiple senses, to increase the chance that the content gets coded into more than one area
of your brain.
We used concepts and pictures in unexpected ways because your brain is tuned for novelty,
and we used pictures and ideas with at least some emotional content, because your brain
is tuned to pay attention to the biochemistry of emotions. That which causes you to feel
something is more likely to be remembered, even if that feeling is nothing more than a little
humor, surprise, or interest.
We used a personalized, conversational style, because your brain is tuned to pay more
attention when it believes you’re in a conversation than if it thinks you’re passively listening
to a presentation. Your brain does this even when you’re reading.
We included more than 80 activities, because your brain is tuned to learn and remember
more when you do things than when you read about things. And we made the exercises
challenging-yet-do-able, because that’s what most people prefer.
We used multiple learning styles, because you might prefer step-by-step procedures, while
someone else wants to understand the big picture first, and someone else just wants to see
an example. But regardless of your own learning preference, everyone benefits from seeing the
same content represented in multiple ways.
We include content for both sides of your brain, because the more of your brain you
engage, the more likely you are to learn and remember, and the longer you can stay focused.
Since working one side of the brain often means giving the other side a chance to rest, you
can be more productive at learning for a longer period of time.
And we included stories and exercises that present more than one point of view,
because your brain is tuned to learn more deeply when it’s forced to make evaluations and
judgments.
We included challenges, with exercises, and by asking questions that don’t always have
a straight answer, because your brain is tuned to learn and remember when it has to work at
something. Think about it—you can’t get your body in shape just by watching people at the
gym. But we did our best to make sure that when you’re working hard, it’s on the right things.
That you’re not spending one extra dendrite processing a hard-to-understand example,
or parsing difficult, jargon-laden, or overly terse text.
We used people. In stories, examples, pictures, etc., because, well, because you’re a person.
And your brain pays more attention to people than it does to things.

xxviii   intro
 the intro

Here’s what YOU can do to bend

your brain into submission
So, we did our part. The rest is up to you. These tips are a
starting point; listen to your brain and figure out what works
for you and what doesn’t. Try new things.
Cut this out and sti
on your refrigerator.ck it

1 Slow down. The more you understand, the 6 Drink water. Lots of it.
less you have to memorize. Your brain works best in a nice bath of fluid.
Don’t just read. Stop and think. When the book asks Dehydration (which can happen before you ever
you a question, don’t just skip to the answer. Imagine feel thirsty) decreases cognitive function.
that someone really is asking the question. The
more deeply you force your brain to think, the better 7 Listen to your brain.
chance you have of learning and remembering. Pay attention to whether your brain is getting
overloaded. If you find yourself starting to skim
2 Do the exercises. Write your own notes. the surface or forget what you just read, it’s time
We put them in, but if we did them for you, that for a break. Once you go past a certain point, you
would be like having someone else do your workouts won’t learn faster by trying to shove more in, and
for you. And don’t just look at the exercises. Use a you might even hurt the process.
pencil. There’s plenty of evidence that physical
activity while learning can increase the learning.
8 Feel something.
Your brain needs to know that this matters. Get
3 Read the “There are No Dumb Questions.”
involved with the stories. Make up your own
That means all of them. They’re not optional captions for the photos. Groaning over a bad joke
sidebars, they’re part of the core content! is still better than feeling nothing at all.
Don’t skip them.

4 Make this the last thing you read before bed. 9 Write a lot of code!
Or at least the last challenging thing. There’s only one way to learn to program: writing
Part of the learning (especially the transfer to a lot of code. And that’s what you’re going to
long-term memory) happens after you put the book do throughout this book. Coding is a skill, and the
down. Your brain needs time on its own, to do more only way to get good at it is to practice. We’re going
processing. If you put in something new during that to give you a lot of practice: every chapter has
processing time, some of what you just learned will exercises that pose a problem for you to solve. Don’t
be lost. just skip over them—a lot of the learning happens
when you solve the exercises. We included a solution
5 Talk about it. Out loud. to each exercise—don’t be afraid to peek at the
Speaking activates a different part of the brain. If solution if you get stuck! (It’s easy to get snagged
you’re trying to understand something, or increase on something small.) But try to solve the problem
your chance of remembering it later, say it out loud. before you look at the solution. And definitely get it
Better still, try to explain it out loud to someone else. working before you move on to the next part of the
You’ll learn more quickly, and you might uncover book.
ideas you hadn’t known were there when you were
reading about it.

you are here 4   xxix

how to use this book

Read Me
This is a learning experience, not a reference book. We deliberately stripped out everything
that might get in the way of learning whatever it is we’re working on at that point in the
book. And the first time through, you need to begin at the beginning, because the book
makes assumptions about what you’ve already seen and learned.

This book is designed to get you up to speed with Python as

quickly as possible.
As you need to know stuff, we teach it. So you won’t find long lists of technical material, no
tables of Python’s operators, not its operator precedence rules. We don’t cover everything,
but we’ve worked really hard to cover the essential material as well as we can, so that you
can get Python into your brain quickly and have it stay there. The only assumption we make
is that you already know how to program in some other programming language.

This book targets Python 3

We use Release 3 of the Python programming language in this book, and we cover how to
get and install Python 3 in the first chapter. That said, we don’t completely ignore Release
2, as you’ll discover in Chapters 8 through 11. But trust us, by then you’ll be so happy using
Python, you won’t notice that the technologies you’re programming are running Python 2.

We put Python to work for you right away.

We get you doing useful stuff in Chapter 1 and build from there. There’s no hanging
around, because we want you to be productive with Python right away.

The activities are NOT optional.

The exercises and activities are not add-ons; they’re part of the core content of the book.
Some of them are to help with memory, some are for understanding, and some will help
you apply what you’ve learned. Don’t skip the exercises.

The redundancy is intentional and important.

One distinct difference in a Head First book is that we want you to really get it. And we
want you to finish the book remembering what you’ve learned. Most reference books don’t
have retention and recall as a goal, but this book is about learning, so you’ll see some of the
same concepts come up more than once.

xxx   intro
 the intro

The examples are as lean as possible.

Our readers tell us that it’s frustrating to wade through 200 lines of an example looking
for the two lines they need to understand. Most examples in this book are shown within
the smallest possible context, so that the part you’re trying to learn is clear and simple.
Don’t expect all of the examples to be robust, or even complete—they are written
specifically for learning, and aren’t always fully functional.
We’ve placed a lot of the code examples on the Web so you can copy and paste them as
needed. You’ll find them at two locations:

http://www.headfirstlabs.com/books/hfpython/
http://python.itcarlow.ie

The Brain Power exercises don’t have answers.

For some of them, there is no right answer, and for others, part of the learning
experience of the Brain Power activities is for you to decide if and when your answers
are right. In some of the Brain Power exercises, you will find hints to point you in the
right direction.

you are here 4   xxxi

the review team

The technical review team

David Griffiths Jeremy Jones

Phil Hartley

Technical Reviewers:

David Griffiths is the author of Head First Rails Phil Hartley has a degree in Computer Science
and the coauthor of Head First Programming. He began from Edinburgh, Scotland. Having spent more than
programming at age 12, when he saw a documentary 30 years in the IT industry with specific expertise in
on the work of Seymour Papert. At age 15, he wrote OOP, he is now teaching full time at the University
an implementation of Papert’s computer language of Advancing Technology in Tempe, AZ. In his spare
LOGO. After studying Pure Mathematics at University, time, Phil is a raving NFL fanatic
he began writing code for computers and magazine
Jeremy Jones is coauthor of Python for Unix and
articles for humans. He’s worked as an agile coach,
Linux System Administration. He has been actively using
a developer, and a garage attendant, but not in that
Python since 2001. He has been a developer, system
order. He can write code in over 10 languages and
administrator, quality assurance engineer, and tech
prose in just one, and when not writing, coding, or
support analyst. They all have their rewards and
coaching, he spends much of his spare time traveling
challenges, but his most challenging and rewarding job
with his lovely wife—and fellow Head First author—
has been husband and father.
Dawn.

xxxii   intro
 the intro

Acknowledgments
My editor:
Brian Sawyer was Head First Python’s editor. When not editing
books, Brian likes to run marathons in his spare time. This turns out
to be the perfect training for working on another book with me (our
second together). O’Reilly and Head First are lucky to have someone
of Brian’s caliber working to make this and other books the best they
can be. Brian Sawyer
The O’Reilly team:
Karen Shaner provided administrative support and very capably coordinated the techical review process, responding
quickly to my many queries and requests for help. There’s also the back-room gang to thank—the O’Reilly Production
Team—who guided this book through its final stages and turned my InDesign files into the beautiful thing you’re
holding in your hands right now (or maybe you’re on an iPad, Android tablet, or reading on your PC—that’s cool, too).
And thanks to the other Head First authors who, via Twitter, offered cheers, suggestions, and encouragement
throughout the entire writing process. You might not think 140 characters make a big difference, but they really do.
I am also grateful to Bert Bates who, together with Kathy Sierra, created this series of books with their wonderful
Head First Java. At the start of this book, Bert took the time to set the tone with a marathon 90-minute phone call,
which stretched my thinking on what I wanted to do to the limit and pushed me to write a better book. Now, some nine
months after the phone call, I’m pretty sure I’ve recovered from the mind-bending Bert put me through.
Friends and colleagues:
My thanks again to Nigel Whyte, Head of Department, Computing and Networking at The Institute of Technology,
Carlow, for supporting my involvement in yet another book (especially so soon after the last one).
My students (those enrolled on 3rd Year Games Development and 4th Year Software Engineering) have been exposed
to this material in various forms over the last 18 months. Their positive reaction to Python and the approach I take with
my classes helped inform the structure and eventual content of this book. (And yes, folks, some of this is on your final).
Family:
My family, Deirdre, Joseph, Aaron, and Aideen had to, once more, bear the grunts and groans, huffs and puffs,
and more than a few roars on more than one occasion (although, to be honest, not as often they did with Head First
Programming). After the last book, I promised I wouldn’t start another one “for a while.” It turned out “a while” was no
more than a few weeks, and I’ll be forever grateful that they didn’t gang up and throw me out of the house for breaking
my promise. Without their support, and especially the ongoing love and support of my wife, Deirdre, this book would
not have seen the light of day.
The without-whom list:
My technical review team did an excellent job of keeping me straight and making sure what I covered was spot on.
They confirmed when my material was working, challenged me when it wasn’t and not only pointed out when stuff
was wrong, but provided suggestions on how to fix it. This is especially true of David Griffiths, my co-conspirator on
Head First Programming, whose technical review comments went above and beyond the call of duty. David’s name might
not be on the cover of this book, but a lot of his ideas and suggestions grace its pages, and I was thrilled and will forever
remain grateful that he approached his role as tech reviewer on Head First Python with such gusto.

you are here 4   xxxiii

safari books online

Safari® Books Online

Safari Books Online is an on-demand digital library that lets
you easily search over 7,500 technology and creative reference
books and videos to find the answers you need quickly.
With a subscription, you can read any page and watch any video from our library
online. Read books on your cell phone and mobile devices. Access new titles before
they are available for print, and get exclusive access to manuscripts in development
and post feedback for the authors. Copy and paste code samples, organize your
favorites, download chapters, bookmark key sections, create notes, print out pages,
and benefit from tons of other time-saving features.
O’Reilly Media has uploaded this book to the Safari Books Online service. To have
full digital access to this book and others on similar topics from O’Reilly and other
publishers, sign up for free at http://my.safaribooksonline.com/?portal=oreilly.

xxxiv   intro
1 meet python

Everyone loves lists

Yes, yes...we have lots
of Pythons in stock... I’ll
just make a quick list.

You’re asking one question: “What makes Python different?”

The short answer is: lots of things. The longer answers starts by stating that there’s lots
that’s familiar, too. Python is a lot like any other general-purpose programming language,
with statements, expressions, operators, functions, modules, methods, and classes.
All the usual stuff, really. And then there’s the other stuff Python provides that makes
the programmer’s life—your life—that little bit easier. You’ll start your tour of Python by
learning about lists. But, before getting to that, there’s another important question that
needs answering…

this is a new chapter   1

python greatness

What’s to like about Python?

Lots. Rather than tell you, this book’s goal is to show you the greatness that is
Python.

Yeah... I need something that I can deploy

on PCs, Macs, handhelds, phones,the Web,
on big servers and small clients...and it has
to let me build GUIs quickly and painlessly...
OK, yes, yeah, I’m listening... What?!?
You’re kidding! Python can do all that?

Before diving head first into Python, let’s get a bit of housekeeping out of
the way.
To work with and execute the Python code in this book, you need a copy of
the Python 3 interpreter on your computer. Like a lot of things to do with
Python, it’s not difficult to install the interpreter. Assuming, of course, it’s not
already there…

2   Chapter 1
 meet python

Install Python 3
Before you write and run Python code, you need to make sure the Python
interpreter is on your computer. In this book, you’ll start out with Release 3 of
Python, the very latest (and best) version of the language.
A release of Python might already be on your computer. Mac OS X comes
with Python 2 preinstalled, as do most versions of Linux (which can also ship
with Release 3). Windows, in contrast, doesn’t include any release of Python.
Let’s check your computer for Python 3. Open up a command-line prompt
and, if you are using Mac OS X or Linux, type:

python3 -V
That’s an UP
Do this!
On Windows, use this command:
“v”, by the wayP.ERCASE

c:\Python31\python.exe -V If Python 3 is missing from

your computer, download
a copy for your favorite OS
from the www.python.org
Using the ” website.
UPPERCASE “vPython
the
results in ing on
version appear
File Edit Window Help WhichPython?
$ python3 -V
screen. Python 3.1.2
$
$ python3
Without the Python 3.1.2 (r312:79360M, Mar 24 2010, 01:33:18)
UPPERCASE “v”, [GCC 4.0.1 (Apple Inc. build 5493)] on darwin
you are taken Type "help", "copyright", "credits" or "license" for more info.
into the Python >>>
>>> quit()
interpreter. $
Use the quit()
command to exit
the interpreter and
return to your OS
prompt.

When you install Python 3, you also get IDLE, Python’s simple—yet
surprisingly useful— integrated development environment. IDLE includes a
color syntax-highlighting editor, a debugger, the Python Shell, and a complete
copy of Python 3’s online documentation set.
Let’s take a quick look at IDLE.

you are here 4   3

 idle hands

Use IDLE to help learn Python

IDLE lets you write code in its full-featured code editor as well as experiment
with code at the Python Shell. You’ll use the code editor later in this book
but, when learning Python, IDLE’s shell really rocks, because it lets you try
out new Python code as you go.
When you first start IDLE, you are presented with the “triple chevron”
prompt (>>>) at which you enter code. The shell takes your code statement
and immediately executes it for you, displaying any results produced on screen.
IDLE knows all about Python syntax and offers “completion hints” that pop
up when you use a built-in function like print(). Python programmers
generally refer to built-in functions as BIFs. The print() BIF displays
messages to standard output (usually the screen).
Unlike other C-based
languages, which use { and }
to delimit blocks, Python uses
Enter your codome pt. indentation instead.
at the >>> pr

See results
immediately.

IDLE uses colored syntax to highlight your code. By default, built-in IDLE knows
functions are purple, strings are green, and language keywords (like if) are
orange. Any results produced are in blue. If you hate these color choices, Python’s syntax
don’t worry; you can easily change them by adjusting IDLE’s preferences.
IDLE also knows all about Python’s indentation syntax, which requires code
and helps you
blocks be indented. When you start with Python, this can be hard to get conform to
used to, but IDLE keeps you straight by automatically indenting as needed.
the Python
indentation rules.
4   Chapter 1
 meet python

Work effectively with IDLE

This is how IDLE looks on
IDLE has lots of features, but you need to know about only
my computer. It might look a
a few of them to get going.
little different on yours, but
TAB completion not by much. (And, yes, it’s
meant to look this ugly.)
Start to type in some code, and then press the TAB key.
IDLE will offer suggestions to help you complete your
statement.

Type “pr” and then

TAB at the >>> prompt
to see IDLE’s list of
command completion
suggestions.

Recall code statements

Press Alt-P to recall the previous code statement entered into Alt-P for Previous
IDLE or press Alt-N to move to the next code statement
(assuming there is one). Both key combinations can be used
to cycle rapidly through all of the code you’ve entered into Alt-N for Next
IDLE, re-executing any code statements as needed.
Unless you’re on
Edit recalled code a Mac, in which
case it’s Ctrl-P
Once you recall your code statement, you can edit it and and Ctrl-N.
move around the statement using the arrow keys. It’s
possible to edit any statement that you’ve previously
entered, even code statements that span multiple lines.

Adjust IDLE’s preferences

IDLE’s preferences dialog lets you adjust its default Tweak IDLE
behavior to your tastes. There are four tabs of settings to to your heart’s
tweak. You can control font and tab behavior, the colors content.
used to syntax highlight, the behavior of certain key-
combinations, and IDLE’s start-up settings. So, if shocking
pink strings is really your thing, IDLE gives you the power
to change how your code looks on screen.

you are here 4   5

dealing with data

Deal with complex data

Any program of any worth that you create has to work with data. Sometimes,
the data is simple and straightforward—easy to work with. Other times, the
data you have to work with is complex in its structure and meaning, forcing you
to work hard to make sense of it all, let alone write code to process it.
To tame complexity, you can often arrange your data as a list: there’s the list
of customers, your friend’s list, the shopping list, and your to-do list (to name
a few). Arranging data in lists is so common that Python makes it easy for you
to create and process lists in code.
Let’s look at some complex data before learning how to create and process list
data with Python.

I’ve been making lists of

movie data for years and
would love to be able to
A highly process it on my laptop... There sure is
organized a lot of data
movie buff listed here.

The Holy Grail, 1975, Terry Jones & Terry Gilliam, 91 mins
Graham Chapman

The Holy Grail, 1975, Terry Jones & Terry Gilliam, 91 mins
Michael Palin, John Cleese, Terry Gilliam, Eric Idle & Terry Jones
The Life of Brian, 1979, Terry Jones, 94 mins
Graham Chapman

Graham Chapman
Michael Palin, John Cleese, Terry Gilliam, Eric Idle & Terry Jones
The Meaning of Life, 1983, Terry Jones, 107 mins
The six Monty Python cast members
The Holy Grail, 1975, Terry Jones & Terry Gilliam, 91 mins

Michael Palin, John Cleese, Terry Gilliam, Eric Idle & Terry Jones
Graham Chapman

Michael Palin, John Cleese, Terry Gilliam, Eric Idle & Terry Jones
The Life of Brian, 1979, Terry Jones, 94 mins

The Life of Brian, 1979, Terry Jones, 94 mins

Graham Chapman

Michael Palin, John Cleese, Terry Gilliam, Eric Idle & Terry Jones
The Meaning of Life, 1983, Terry Jones, 107 mins

Graham Chapman
The six Monty Python cast members

The Holy Grail, 1975, Terry Jones & Terry Gilliam, 91 mins
Graham Chapman

Michael Palin, John Cleese, Terry Gilliam, Eric Idle & Terry Jones

Michael Palin, John Cleese, Terry Gilliam, Eric Idle & Terry Jones
The Life of Brian, 1979, Terry Jones, 94 mins
Graham Chapman

The Meaning of Life, 1983, Terry Jones, 107 mins

Michael Palin, John Cleese, Terry Gilliam, Eric Idle & Terry Jones
The Meaning of Life, 1983, Terry Jones, 107 mins
The six Monty Python cast members

The six Monty Python cast members

Graham Chapman, Michael Palin, John Cleese, Terry Gilliam, Eric Idle & Terry Jones

This data is complex, too.

On first glance, this collection of data does indeed look quite complex.
However, the data appears to conform to some sort of structure: there’s a line
for a list of basic movie facts, then another line for the lead actor(s), followed
by a third line listing the movie’s supporting actors.

This looks like a structure you can work with…

6   Chapter 1
 meet python

Create simple Python lists

Let’s start with the following simple list of movie titles and work up from
there:
The Holy Grail
The Life of Brian
The Meaning of Life A short list of some
Monty Python movies
Here’s the same list written in a way that Python understands:

movies = ["The Holy Grail",

"The Life of Brian",
"The Meaning of Life"]

To turn the human-friendly list into a Python-friendly one, follow this four-
step process:

1 Convert each of the names into strings by surrounding the data with quotes.

2 Separate each of the list items from the next with a comma.

3 Surround the list of items with opening and closing square brackets.

4 Assign the list to an identifier (movies in the preceding code) using the
assignment operator (=).

It’s perfectly OK to put your list creation code all on one line, assuming, of
course, that you have room:

movies = ["The Holy Grail", "The Life of Brian", "The Meaning of Life"]

This works, too.

you are here 4   7

not my type

Hang on a second! Aren’t you

forgetting something? Don’t you need to
declare type information for your list?

No, because Python’s variable identifiers

don’t have a type.
Many other programming languages insist that every
identifier used in code has type information declared for
it. Not so with Python: identifiers are simply names that
refer to a data object of some type.
Think of Python’s list as a high-level collection. The
type of the data items is not important to the list. It’s
OK to state that your movies list is a “collection of
strings,” but Python doesn’t need to be told this. All
Python needs to know is that you need a list, you’ve
given it a name, and the list has some data items in it.

8   Chapter 1
 meet python

Lists are like arrays

When you create a list in Python, the interpreter creates an array-like data
structure in memory to hold your data, with your data items stacked from
the bottom up. Like array technology in other programming languages, the
first slot in the stack is numbered 0, the second is numbered 1, the third is
numbered 2, and so on:

This is your Item #0 Item #2

“movies” list in Item #1
code.
movies = ["The Holy Grail", "The Life of Brian", "The Meaning of Life"]

This is your “movies”

list in memory.
"The Meaning of Life"
2
Each data item
in the list has a
"The Life of Brian" 1 numeric OFFSET
associated with it.

"The Holy Grail" 0

Python starts counting
from zero.
Access list data using the square bracket notation
As with arrays, you can access the data item in a list slot using the standard
square bracket offset notation:

print(movies[1]) The Life of Brian

a
Use the “print()” BIF to display No surprise here, really…the
data item on screen. requested data appears on screen
.

Let’s use IDLE to learn a bit about how lists work.

you are here 4   9

idle session

Lists in Python might look like arrays, but they are much more than that: they are full-blown Python collection
objects. This means that lists come with ready-to-use functionality in the form of list methods.

Let’s get to know some of Python’s list methods. Open up IDLE and follow along with the code entered at the >>>
prompt. You should see exactly the same output as shown here.

Start by defining a list of names, which you then display on screen using the print() BIF. Then, use the len()
BIF to work out how many data items are in the list, before accessing and displaying the value of the second data
item:
>>> cast = ["Cleese", 'Palin', 'Jones', "Idle"]
>>> print(cast)
['Cleese', 'Palin', 'Jones', 'Idle']

It’s OK to invoke a BIF on .

>>> print(len(cast))

the results of another BIF

4
>>> print(cast[1])
Palin

With your list created, you can use list methods to add a single data item to the end of your list (using the
append() method), remove data from the end of your list (with the pop() method), and add a collection of
data items to the end of your list (thanks to the extend() method):

>>> cast.append("Gilliam")
Methods are invoked using the
>>> print(cast)
common “.” dot notation.
['Cleese', 'Palin', 'Jones', 'Idle', 'Gilliam']
>>> cast.pop()
'Gilliam'
ted by commas,
It’s another list: items separats.
surrounded by square bracke
>>> print(cast)
['Cleese', 'Palin', 'Jones', 'Idle']
>>> cast.extend(["Gilliam", "Chapman"])
>>> print(cast)
['Cleese', 'Palin', 'Jones', 'Idle', 'Gilliam', 'Chapman']

Finally, find and remove a specific data item from your list (with the remove() method) and then add a data item
before a specific slot location (using the insert() method):

>>> cast.remove("Chapman")
>>> print(cast)
['Cleese', 'Palin', 'Jones', 'Idle', 'Gilliam'] After all that, we end up with
>>> cast.insert(0, "Chapman") the cast of Monty Python’s
>>> print(cast) Flying Circus!
['Chapman', 'Cleese', 'Palin', 'Jones', 'Idle', 'Gilliam']

10   Chapter 1
 meet python

Add more data to your list

With your list of movie names created, now you need to add more of the
movie buff ’s complex data to it. You have a choice here:

I think I’ll use the

appropriate list With something this small,
methods to add the extra I’m gonna simply re-create
data I need. my list from scratch.

Either strategy works. Which works best for you depends on what you are
trying to do. Let’s recall what the movie buff ’s data looks like:

A number
representing The Holy Grail, 1975, Terry Jones & Terry Gilliam, 91 mins
the year is Graham Chapman
next. Michael Palin, John Cleese, Terry Gilliam, Eric Idle & Terry Jones
The Life of Brian, 1979, Terry Jones, 94 mins
Graham Chapman
Michael Palin, John Cleese, Terry Gilliam, Eric Idle & Terry Jones
The Meaning of Life, 1983, Terry Jones, 107 mins
The six Monty Python cast members
Graham Chapman, Michael Palin, John Cleese, Terry Gilliam, Eric Idle & Terry Jones

The next piece of data you need to add to your list is a number (which
represents the year the movie was released), and it must be inserted after each
movie name. Let’s do that and see what happens.

you are here 4   11

mixed type

What?!? There’s no way you can mix

data of different types in lists, is
there? Surely this is madness?

No, not madness, just the way Python works.

Python lists can contain data of mixed type. It’s perfectly OK
to mix strings with numbers within the same Python list. In fact,
you can mix more than just strings and numbers; you can store
data of any type in a single list, if you like.
Recall that a Python list is a high-level collection, designed from
the get-go to store a collection of “related things.” What type
those things have is of little interest to the list, because the list
exists merely to provide the mechanism to store data in list form.
So, if you really need to store data of mixed type in a list,
Python won’t stop you.

12   Chapter 1
 meet python

Let’s take a bit of time to try to work out which strategy to use when adding data to your list in
this case.
Given the following list-creation code:

movies = ["The Holy Grail", "The Life of Brian", "The Meaning of Life"]

1 Work out the Python code required to insert the numeric year data into the preceding list,
changing the list so that it ends up looking like this:

["The Holy Grail", 1975, "The Life of Brian", 1979, "The Meaning of Life", 1983]

Write your
insertion
code here.

2 Now write the Python code required to re-create the list with the data you need all in one go:

Write your
re-creation
code here.

In this case, which of these two methods do you think is best? (Circle your choice).

1 or 2

you are here 4   13

additional data

§
Let’s take a bit of time to try and work out which strategy to use when adding data to your list in
this case.
Given the following list-creation code:

movies = ["The Holy Grail", "The Life of Brian", "The Meaning of Life"]

1 You were to work out the Python code required to insert the numeric year data into the preceding
list:

Insert the first year

BEFORE the second list
item. movies.insert(1, 1975)
Did you get the math right?
Insert the second year After the first insertion, th
BEFORE the fourth list item. list grows, so you have to takee
movies.insert(3, 1979) that into consideration when
working out where to do the
second insert.
movies.append(1983)
Then append the last year to
the end of the list.

2 You were also to write the Python code required to recreate the list with the data you need all in
one go:

movies = ["The Holy Grail", 1975,

“movies”
"The Life of Brian", 1979,
Assign all your data to the usl ere is
identifier. What was previo y th "The Meaning of Life", 1983]
replaced.
In this case, which of these two methods do you think is best? (You were to circle your choice.)

1 2 Yes, method 2 seems the betsmter

option here…that is, for a all
or

list like this. Also, there’s no

tricky counting to do.

14   Chapter 1
 meet python

Work with your list data

You often need to iterate over your list and perform some action on each item
as you go along. Of course, it is always possible to do something like this, Define a list
which works but does not scale: items with theannad populate its
movies. mes of two

fav_movies = ["The Holy Grail", "The Life of Brian"]

This is the list-processing

print(fav_movies[0])

e of print(fav_movies[1]) code.
Display the vallulist item
each individua.
on the screen
This code works as expected, making the data from the list appear on screen.
However, if the code is later amended to add another favorite movie to the list,
the list-processing code stops working as expected, because the list-processing code
does not mention the third item.
Big deal: all you need to do is add another print() statement, right?
Yes, adding one extra print() statement works for one extra movie, but
what if you need to add another hundred favorite movies? The scale of the
problem defeats you, because adding all those extra print() statements
becomes such a chore that you would rather find an excuse not to have to do.

It’s time to iterate

Processing every list item is such a common requirement that Python makes it
especially convenient, with the built-in for loop. Consider this code, which is
a rewrite of the previous code to use a for loop: Define a list
just as you didanbd populate it
efore.
ate
Use “for” to itderplaying
over the list, isch
the value of ea on fav_movies = ["The Holy Grail", "The Life of Brian"]
individual itemgo.
screen as you for each_flick in fav_movies:
This is the list-processing
print(each_flick) code, using a for loop.

Using a for loop scales and works with any size list.

you are here 4   15

 list processing

For loops work with lists of any size

Python’s for loop exists to process lists and other iterations in Python. Lists are
the most common iterated data structure in Python, and when you need to
iterate a list, it’s best to use for:
A colon “:” follows your
The keyword “for” The keyword “in” separates list name and indicates
indicates the start the target identifier from the start of your list-
of the loop and your list. processing code.
comes before the
target identifier.
for target identifer in
list :

list-processing code

ng code
The list-procesensited
MUST be ind loop.
under the for
The list-processing code is referred to by Python programmers as the suite.
The target identifier is like any other name in your code. As your list is
iterated over, the target identifier is assigned each of the data values in your
list, in turn. This means that each time the loop code executes, the target
identifier refers to a different data value. The loop keeps iterating until it
exhausts all of your list’s data, no matter how big or small your list is.
An alternative to using for is to code the iteration with a while loop.
Consider these two snippets of Python code, which perform the same action:

When you use “while”,

you have to worry about
“state information,” count = 0

which requires you while count < len(movies): for each_item in movies:
to employ a counting print(movies[count]) print(each_item)
identifier. count = count+1

When you use “for”, the

Python interpreter
worries about the “state
These while and for statements do the same thing. information” for you.

16   Chapter 1
 meet python

Q: So…when iterating over a list, I Q: Seeing as Python’s lists shrink and Q: What if I need to embed a double
should always use for instead of while? grow as needed, they must not support quote in a string?

A: A:
bounds-checking, right?
Yes, unless you have a really good
reason to use (or need the extra control
of) a while loop. The for loop takes care
A: Well, lists are dynamic, in that they
shrink and grow, but they are not magic,
You have two choices: either escape
the double quote like this: \”, or surround
your string with single quotes.

Q:
of working from the start of your list and in that they cannot access a data item
continuing to the end. It’s next to impossible that does not exist. If you try to access a Can I use any characters to name
to get stung by an off-by-one error when you nonexistent data item, Python responds with my identifiers?
use for. This is not the case with while. an IndexError, which means “out of

Q: bounds.”
A: No. Like most other programming
So, lists aren’t really like arrays
then, because they do so much more? Q: What’s with all the strange
languages, Python has some rules that
must be adhered to when creating names.

A:
references to Monty Python? Names can start with a letter character or

A:
Well…they are in that you can access an underscore, then include any number
individual data items in your list with the Ah, you spotted that, eh? It turns of letter characters, numbers, and/or
standard square bracket notation, but—as out that the creator of Python, Guido van underscores in the rest of the name. Strange
you’ve seen—Python’s lists can do so much Rossum, was reading the scripts of the characters (such as %$£) are not allowed
more. At Head First Labs, we like to think of Monty Python TV shows while designing his and you’ll obviously want to use names that
lists as “arrays on steroids.” new programming language. When Guido have meaning within the context of your

Q: And they work this way only in

needed a name for his new language, he
chose “Python” as a bit of a joke (or so the
legend goes).
code. Names like members, the_
time , and people are much better
Python 3, right? than m, t, and p, aren’t they?

A: No. There are certain enhancements Q: Do I need to know Monty Python in Q: Yes, good naming practice is
to lists that were added in Python 3, but order to understand the examples? always important. But what about case

A:
release 2 of Python has lists, too. All of what sensitivity?
you’ve learned about lists so far will work No, but as they say in the official
with lists in Releases 2 and 3 of Python. Python documentation: “it helps if you do.” A: Yes, Python is the “sensitive type,” in

Q: Why are we using Python 3? What’s

But don’t worry: you’ll survive, even if you’ve
never heard of Monty Python.
that Python code is case sensitive. This
means that msg and MSG are two different
wrong with Python 2, anyway? Lots of
programmers seem to be using it. Q: I notice that some of your strings
names, so be careful. Python (and IDLE)
will help with the problems that can occur as

A:
are surrounded with double quotes and a result of this. For instance, you can use
Lots of programmers are using Python others with single quotes. What’s the an identifier in your code only if it has been
2, but the future of Python development lies difference? given a value; unassigned identifiers cause

A:
with Release 3. Of course, moving the entire a runtime error. This means that if you type
Python community to Python 3 won’t happen There isn’t any. Python lets you use mgs when you meant msg, you’ll find out
overnight, so there’s an awful lot of projects either to create a string. The only rule is that pretty quickly when Python complains about
that will continue to run on Release 2 for the if you start a string with one of the quotes, your code having a NameError.
foreseeable future. Despite 2’s dominance then you have to end it with the same
at the moment, at Head First Labs we think quote; you can’t mix’n’match. As you may
the new bits in 3 are well worth the added have seen, IDLE uses single quotes when
investment in learning about them now. displaying strings within the shell.
Don’t worry: if you know 2, Python 3 is easy.

you are here 4   17

 lists within lists

Store lists within lists

As you’ve seen, lists can hold data of mixed type. But it gets even better than
that: lists can hold collections of anything, including other lists. Simply embed
the inner list within the enclosing list as needed.
There’s only one lead
Looking closely at the movie buff ’s data, it is possible to determine a structure actor listed here, but
which looks much like a list of lists: there could be more.

There’s a list of movie facts…

The Holy Grail, 1975, Terry Jones & Terry Gilliam, 91 mins
…which itself contains Graham Chapman
a list of lead actors…
Michael Palin, John Cleese, Terry Gilliam, Eric Idle & Terry Jones
…which itself
contains a list of
supporting actors.

In Python, you can turn this real list of data into code with little or no effort.
All you need to remember is that every list is a collection of items separated
from each other with commas and surrounded with square brackets. And, of
course, any list item can itself be another list:

The start of the

first, outer list
movies = [
The end of all the
"The Holy Grail", 1975, "Terry Jones & Terry Gilliam", 91,
lists is here.
["Graham Chapman",
["Michael Palin", "John Cleese", "Terry Gilliam", "Eric Idle", "Terry Jones"]]]

The start of the

second, inner list: The start of the third, inner This looks a little weird…until you
“movies[4]” inner list: “movies[4][1]” remember that there are th
opening square brackets, so three
So, a list within a list is possible, as is a list within a list within a list (as this also be three closing ones. ere must
example code demonstrates). In fact, it’s possible to nest lists within lists to
most any level with Python. And you can manipulate every list with its own list
methods and access it with the square bracket notation:

print(movies[4][1][3]) Eric Idle

A list within a list within a list Eric is this deeply nested, so he

can’t possibly be idle. §
18   Chapter 1
 meet python

Creating a list that contains another list is straightforward. But what happens when you try to process a list that
contains another list (or lists) using the for loop from earlier in this chapter?
Let’s use IDLE to work out what happens. Begin by creating the list of the movie data for “The Holy Grail” in
memory, display it on screen, and then process the list with your for loop:

>>> movies = ["The Holy Grail", 1975, "Terry Jones & Terry Gilliam", 91,
["Graham Chapman", ["Michael Palin", "John Cleese",
"Terry Gilliam", "Eric Idle", "Terry Jones"]]]
>>> print(movies)
['The Holy Grail', 1975, 'Terry Jones & Terry Gilliam', 91, ['Graham Chapman', ['Michael Palin',
'John Cleese', 'Terry Gilliam', 'Eric Idle', 'Terry Jones']]]
>>> for each_item in movies:
The list within a list within
print(each_item)
a list has been created in
memory.
The Holy Grail
1975 The “for” loop prints each item of
Terry Jones & Terry Gilliam the outer loop ONLY.
91
['Graham Chapman', ['Michael Palin', 'John Cleese', 'Terry Gilliam', 'Eric Idle', 'Terry Jones']]

The inner list within the inner list is printed “as-is.”

Your for loop is working OK. I think the

trouble is that you haven’t told it what to
do with any inner lists that it finds, so it
just prints everything, right?

Yes, that’s correct: the loop code isn’t complete.

At the moment, the code within the loop simply prints each list
item, and when it finds a list at a slot, it simply displays the entire
list on screen. After all, the inner list is just another list item as far as
the outer enclosing list is concerned. What’s we need here is some
mechanism to spot that an item in a list is in fact another list and take
the appropriate action.

That sounds a little tricky. But can Python help?

you are here 4   19

 looking for lists

Check a list for a list

Each time you process an item in your list, you need to check to see if the
item is another list. If the item is a list, you need to process the nested list
before processing the next item in your outer list. Deciding what to do when in
Python follows the familiar if... else... pattern:
A colon (:) follows your
The keyword “if” condition test.
indicates the start
of the decision code.
if some condition holds :
es if the
This code execsut(i.e., it’s TRUE). the "true" suite
condition hold
Note: both suites
else: Look! Another colon. are indented.

This code executes if the condit

does NOT hold (i.e., it’s FALSE)ion
the "false" suite
.

No surprises here, as the if statement in Python works pretty much as

expected. But what condition do you need to check? You need a way to
determine if the item currently being processed is a list. Luckily, Python ships
with a BIF that can help here: isinstance().
What’s cool about the isinstance() BIF is that it lets you check if a
specific identifier holds data of a specific type:

Create a short list and Let’s use the IDLE shell to learn a little about how isinstance() works:
assign it to an identifier.
>>> names = ['Michael', 'Terry']

Ask if “names” is a list (it is). >>> isinstance(names, list)

Refer to a Python type
here. In this case, the type
True
Assign a number to an >>> num_names = len(names)
identifier. >>> isinstance(num_names, list)
is “list”.
es” is a
Ask if “num_nam
False

list (it isn’t).

20   Chapter 1
 meet python

Here’s a copy of the current list-processing code. Your task is to rewrite this code using an if
statement and the isinstance() BIF to process a list that displays another list.

for each_item in movies:

print(each_item)

Write your
new code
here.

Q: Are there many of these BIFs in Q: How? Q: Why so many BIFs?

A: A:
Python?

A: Yes. At the last count, there were over

70 BIFs in Python 3.
At the Python or IDLE shell, type
dir(__builtins__) to see a list
of the built-in stuff that comes with Python
Why not? Because Python comes with
lots of built-in functionality, it can mean less
code for you to write. This Python philosophy

Q: (that’s two leading and trailing underscore is known as “batteries included”: there’s
Over 70! How am I to remember characters, by the way). The shell spits enough included with Python to let you do
that many, let alone find out what they all out a big list. Try it. All those lowercase most things well, without having to rely on
are? words are BIFs. To find out what any BIF code from third parties to get going. As well
does—like input(), for example—type as lots of BIFs, you’ll find that Python’s
A: You don’t have to worry about
remembering. Let Python do it for you.
help(input) at the shell for a
description of the BIFs function.
standard library is rich and packed with
features waiting to be exploited by you.

you are here 4   21

list the list

Here’s a copy of the current list-processing code. Your task was to rewrite this code using an if
statement and the isinstance() BIF to process a list that displays another list.

for each_item in movies:

print(each_item)

Process the “movies”

list as before. The inner loop
for each_item in movies: needs a new target
You need to check if the identifier.
if isinstance(each_item, list):
current item is a list.

If it is a list, use another for nested_item in each_item:
“for” loop to process the
nested list. print(nested_item)
else: If the current item
Did you manage to get your of the enclosing list
indentation right? print(each_item) isn’t a list, display it
on screen.

Let’s use IDLE to see if this code makes a difference to the output displayed on screen:

>>> for each_item in movies:

if isinstance(each_item, list):
for nested_item in each_item:
print(nested_item)
else:
print(each_item)

by
This is a little better, but notlist here
much…there’s another nested
The Holy Grail

ly.
that’s not being processed proper
1975
Terry Jones & Terry Gilliam
91
Graham Chapman
[‘Michael Palin’, ‘John Cleese’, ‘Terry Gilliam’, ‘Eric Idle’, ‘Terry Jones’]

22   Chapter 1
 meet python

Complex data is hard to process

The movie buff ’s data is complex. Let’s take another look at a subset of the
data and your Python code that processes it.

The outer, enclosing list

The data

The Holy Grail, 1975, Terry Jones & Terry Gilliam, 91 mins
Graham Chapman
Michael Palin, John Cleese, Terry Gilliam, Eric Idle & Terry Jones

An inner, nested list Another inner (inner), nested list

Your code Process the in

Process the outer, enclosing list. nested list. ner,

Yeah.. that’s almost

working...it’s just a
for each_item in movies: pity about that list of
if isinstance(each_item, list): supporting actors...
for nested_item in each_item:
print(nested_item)
else:
print(each_item)

Can you spot the problem with your Python

code as it is currently written? What do you
think needs to happen to your code to allow it to
process the movie buff’s data correctly?

you are here 4   23

nested lists

Handle many levels of nested lists

The data and your code are not in sync.
The movie buff ’s data is a list that contains a nested list that itself contains
a nested list. The trouble is that your code knows only how to process a list
nested inside an enclosing list.
The solution, of course, is to add more code to handle the additionally nested list. By
looking at the existing code, it’s easy to spot the code you need to repeat:

for each_item in movies:

if isinstance(each_item, list):
This code for nested_item in each_item:
processes a
nested list. print(nested_item)
else:
print(each_item)

Here’s where the

repeated code needs
The next to go.
iteration of your
code looks like
this.

for each_item in movies:

if isinstance(each_item, list):
The repeated co
for nested_item in each_item: replaces the “pride
if isinstance(nested_item, list): statement and innt()”
Note: in this another target troduces
called “deeper_itidentifier
for deeper_item in nested_item:
code, each
“if ” needs an
print(deeper_item) em”.
associated
else:
“el
se”. print(nested_item)
else:
print(each_item)

24   Chapter 1
 meet python

Let’s use IDLE once more to test this latest iteration of your code:
d
eeply neste
>>> for each_item in movies: Process a da nested list
list inside enclosing list.
inside an
if isinstance(each_item, list):
for nested_item in each_item:
if isinstance(nested_item, list):
for deeper_item in nested_item:
print(deeper_item)
else:
print(nested_item)
else:
print(each_item)

The Holy Grail

1975
It works! This time, you
Terry Jones & Terry Gilliam see all of your list data
91 on screen.
Graham Chapman
Michael Palin
John Cleese
Terry Gilliam
Eric Idle
Terry Jones

I just love that...in fact, I love it so

much I’ve decided to add another list to my
data. I want to include the other movies each
supporting actor has starred in. If I add the
data, can you change your code to print this
data, too?

That’s more list data and more Python code.

The data has to be embedded as another nested list within the already deeply
nested list of supporting actors. That’s possible to do, even though it makes
your head hurt just to think about a list of lists of lists of lists! Amending your
code is just a matter of adding another for loop and an if statement.

That doesn’t sound like too much trouble, does it?

you are here 4   25

avoid complexity

I think I’d rather have a root canal

than change that code again.

Adding another nested loop is a huge pain.

Your data is getting more complex (that mind-bending list
of lists of lists of lists) and, as a consequence, your code is
getting overly complex, too (that brain-exploding for loop
inside a for loop inside a for loop). And overly complex
code is rarely a good thing…

26   Chapter 1
 meet python

Wouldn’t it be dreamy if there were an

efficient way to process lists, preferably
using a technique that resulted in less code,
not more? But I know it’s just a fantasy...

you are here 4   27

reduce, reuse, recycle

Don’t repeat code; create a function

Take a look at the code that you’ve created so far, which (in an effort to save
you from having your brain explode) has already been amended to process yet
another nested list. Notice anything?
This code is essentially the
same as this code…

for each_item in movies:

…which is essentially
the same as this
if isinstance(each_item, list):
for nested_item in each_item: code…
if isinstance(nested_item, list):
…which is not
for deeper_item in nested_item: that much
if isinstance(deeper_item, list): different tha
for deepest_item in deeper_item: this code. n
print(deepest_item)
else:
print(deeper_item)
else:
h
print(nested_item)
There’s not mucong these
else: am
difference ts, either!
print(each_item) four statemen

This code is beginning to get a little scary…

Your code now contains a lot of repeated code. It’s also a mess to look at, even
though it works with the movie buff ’s amended data. All that nesting of for
loops is hard to read, and it’s even harder to ensure that the else suites are
associated with the correct if statement.
There has to be a better way…but what to do?
When code repeats in this way, most programmers look for a way to take
the general pattern of the code and turn it into a reusable function. And
Python programmers think this way, too. Creating a reusable function lets
you invoke the function as needed, as opposed to cutting and pasting existing
code.

So, let’s turn the repeating code into a function.

28   Chapter 1
 meet python

Create a function in Python

A function in Python is a named suite of code, which can also take an optional
list of arguments if required.
You define a Python function using the def statement, providing a name
for your function and specifying either an empty or populated argument list A colon (:) follows the
within parentheses. The standard form looks something like this: closing parenthesis
Argument lists are optional, and indicates the
but the parentheses are NOT. start of your
The keyword “def” functions code suite.
introduces the name
of the function.
def function name ( argument(s) ):

de
The function’sencoted under
function code suite

MUST be ind ent.

the def statem
What does your function need to do?
Your function needs to take a list and process each item in the list. If it finds a
nested list within the first list, the function needs to repeat. It can do this by
invoking itself on the nested list. In other words, the function needs to recur—
that is, invoke itself from within the funtion code suite.

Let’s call the function that you’ll create print_lol(). It takes

one argument: a list to display on screen. Grab your pencil and
complete the code below to provide the required functionality:

def print_lol(the_list):

for
if

else:

you are here 4   29

recursive function

You were to call the function that you’ll create print_lol(). It

takes one argument: a list to display on screen. You were to grab your
pencil and complete the code to provide the required functionality:

def print_lol(the_list):

If the item being processed

Process the provid ed for each_item in the_list: is itself a list, invoke the
list wit h a “fo r” loo

p. function.
if isinstance(each_item, list):
print_lol(each_item)
’T
If the item being processed ISN
else:
.
print(each_item) a list, display the item on screen

Let’s use IDLE one final time to test your new function. Will it work as well as your earlier code?

>>> def print_lol(the_list):

for each_item in the_list:
if isinstance(each_item, list): Define the function.
print_lol(each_item)
else:
print(each_item)

>>> print_lol(movies)
Invoke the function.
The Holy Grail
1975
Terry Jones & Terry Gilliam
91
It works, too! The recusrive fun
Graham Chapman
produces EXACTLY the same res ction
Michael Palin
the earlier code. ults as
John Cleese
Terry Gilliam
Eric Idle
Terry Jones

30   Chapter 1
 meet python

Recursion to the rescue!

The use of a recursive function has allowed you to reduce 14 lines of messy,
hard-to-understand, brain-hurting code into a six-line function. Unlike the
earlier code that needs to be amended to support additional nested lists
(should the movie buff require them), the recursive function does not need to
change to process any depth of nested lists properly.
Python 3 defaults its recursion limit to 1,000, which is a lot of lists of lists of lists
of lists…and this limit can be changed should you ever need even more depth
than that.

Ah, yes, that’s terrific! I can now

relax, knowing that your code can
process my movie data. I really
should’ve done this years ago...

What a great start!

By taking advantage of functions and recursion, you’ve solved the code
complexity problems that had crept into your earlier list-processing code.
By creating print_lol(), you’ve produced a reusable chunk of code that
can be put to use in many places in your (and others) programs.

You’re well on your way to putting Python to work!

you are here 4   31

 python toolbox

Your Python Toolbox

You’ve got Chapter 1 under your

CHAPTER 1
belt and you’ve added some key ƒƒ Run Python 3 from the command line or
CHAPTER 1

Python goodies to your toolbox. from within IDLE.

ƒƒ Identifiers are names that refer to data
objects. The identifiers have no “type,” but
the data objects that they refer to do.
ƒƒ print() BIF displays a message on
screen.

Python Lingo ƒƒ A list is a collection of data, separated

- a built-in function.
by commas and surrounded by square
• “BIF” brackets.
• “Suite” - a block of
Python code, which ƒƒ Lists are like arrays on steroids.
is indented to indicate grouping. ƒƒ Lists can be used with BIFs, but also
• “Batteries included
” - a way of support a bunch of list methods.
on comes
referring to the fact that Pythto get ƒƒ Lists can hold any data, and the data can be
with most everything you’ll need of mixed type. Lists can also hold other lists.
going quickly and productively. ƒƒ Lists shrink and grow as needed. All of the
memory used by your data is managed by

IDLE Notes
Python for you.
ƒƒ Python uses indentation to group statements
together.
• The IDLE shell lets you experiment wit
your code as you write it. h ƒƒ len() BIF provides a length of some data
object or count the number of items in a
• Adjust IDLE’s prefe
rences to suit the collection, such as a list.
way you work. ƒƒ The for loop lets you iterate a list and
• Remember: when wo is often more convenient to use that an
rking with the shell,
use Alt-P for Previous and use Alt equivalent while loop.
Next (but use Ctrl if you’re on -N for ƒƒ The if... else... statement lets you make
a Mac). decisions in your code.
ƒƒ isinstance() BIF checks whether
an identifier refers to a data object of some
specified type.
ƒƒ Use def to define a custom function.

32   Chapter 1
2 sharing your code

Modules of functions
I’d love to share...but
how am I supposed
to function without a
module?

Reusable code is great, but a shareable module is better. 

By sharing your code as a Python module, you open up your code to the entire Python
community…and it’s always good to share, isn’t it? In this chapter, you’ll learn how to
create, install, and distribute your own shareable modules. You’ll then load your module
onto Python’s software sharing site on the Web, so that everyone can benefit from your
work. Along the way, you’ll pick up a few new tricks relating to Python’s functions, too.

this is a new chapter   33

let’s share

It’s too good not to share

You’ve been showing your function to other programmers, and they like what
they see.

You should make your function shareable,

so that everyone can use it.

Yes, a function this good should be shared with the world.

Python provides a set of technologies that make this easy for you, which
includes modules and the distribution utilities:

Modules let you organize your code for optimal sharing.

The distribution utilities let you share your modules with the world.

Let’s turn your function into a module, then use the distribution utilities to
share your module with the wider Python programming community.

34   Chapter 2
 sharing your code

Turn your function into a module

A module is simply a text file that contains Python code. The main
requirement is that the name of the file needs to end in .py: the Python
extension. To turn your function into a module, save your code into an
appropriately named file:

Your code
from
Chapter 1 def print_lol(the_list):
for each_item in the_list:
if isinstance(each_item, list):
print_lol(each_item)
else:
print(each_item)

Let’s call
this file
“nester.py”.

Q: What’s the best Python editor?

A: The answer to that question really depends on who you ask. However, you
can, of course, use any text editor to create and save your function’s code in a
text file. Something as simple as NotePad on Windows works fine for this, as does
a full-featured editor such as TextMate on Mac OS X. And there’s also full-fledged Do this!
IDEs such as Eclipse on Linux, as well as the classic vi and emacs editors. And,
as you already know, Python comes with IDLE, which also includes a built-in
code editor. It might not be as capable as those other “real” editors, but IDLE is
installed with Python and is essentially guaranteed to be available. For lots of
jobs, IDLE’s edit window is all the editor you’ll ever need when working with your
Python code. Of course, there are other IDEs for Python, too. Check out WingIDE
Go ahead and create a text
for one that specifically targets Python developers. file called nester.py that
contains your function code
from the end of Chapter 1.

you are here 4   35

modules repository

Modules are everywhere

As might be expected, you’ll find Python modules in lots of places.

I’m preloaded with

lots of modules in the
Python Standard Library...
and they are already on your
computer.

If the Standard Library doesn’t do

it for you, why not try the Web?
I hear PyPI is where third-party
Python modules hang out.

ed
PyPI is pronounc
“pie-pie.”
Geek Bits
The Python Package Index (or PyPI for short) provides a
centralized repository for third-party Python modules on the
Internet. When you are ready, you’ll use PyPI to publish your
If you are already familiar with
module and make your code available for use by others. And your
Perl’s CPAN repository, you can
module is ready, but for one important addition. think of PyPI as the Python
equivalent.
What do you think is missing from your module?

36   Chapter 2
 sharing your code

Comment your code

It’s always a good idea to include comments with your code. As your plan to
share your module with the world, well-written comments help to document
your work.
In Python, a common commenting technique is to use a triple quote for
multiple-line comments. When you use a triple quote without assigning it to a
variable, everything between the triple quotes is considered a comment:
Hello! I’m a big string who
just happens to be a Python
comment, too. Nice, eh?
Start with a
triple quote…
"""This is the standard way to
include a multiple-line comment in
your code."""

…and end with a

triple quote.

Here is your module code (which is saved in the file nester.py). In

the spaces provided, use your pencil to compose two comments: the
Put your mod first to describe the module and the second to describe the function.
comment here.ule

def print_lol(the_list):

Add a commention
for your funct
here. for each_item in the_list:
if isinstance(each_item, list)
print_lol(each_item)
else:
print(each_item)

you are here 4   37

request for comments

Here is your module code (which is saved in the file nester.py). In

the spaces provided, you were asked to use your pencil to compose
two comments: the first to describe the module and the second to
describe the function.

“““This is the “nester.py" module, and it provides one function called

print_lol() which prints lists that may or may not include nested lists.”””
Did you def print_lol(the_list):
remember to ple
include the tri “““This function takes a positional argument called “the_list", which is any
quotes? Python list (of, possibly, nested lists). Each data item in the provided list
is (recursively) printed to the screen on its own line.”””
for each_item in the_list:
if isinstance(each_item, list):

There are no changes to thte

print_lol(each_item)
else:
actual code here; you’re jus
print(each_item) adding some comments.

Q: How do I know where the Python Q: Does it matter where I put my Q: Is there any other way to add a
modules are on my computer? nester.py module? comment to Python code?

A: Ask IDLE. Type import sys; A: For now, no. Just be sure to put it
somewhere where you can find it later. In
A: Yes. If you put a “#” symbol anywhere
on a line, everything from that point to the
sys.path (all on one line) into the IDLE
prompt to see the list of locations that your a while, you’ll install your module into your end of the current line is a comment (unless
Python interpreter searches for modules. local copy of Python, so that the interpreter the “#” appears within a triple quote, in

Q:
can find it without you having to remember which case it’s part of that comment). A lot
when you actually put it. of Python programmers use the “#” symbol
Hang on a second. I can use “;” to

Q:
to quickly switch on and off a single line of
put more than one line of code on the
So comments are like a funny- code when testing new functionality.
same line in my Python programs?
looking string surrounded by quotes?
A: Yes, you can. However, I don’t
recommend that you do so. Better to give A: Yes. When a triple-quoted string is
each Python statement its own line; it makes not assigned to a variable, it’s treated like a
your code much easier for you (and others) comment. The comments in your code are
to read. surrounded by three double quotes, but you
could have used single quotes, too.

38   Chapter 2
 sharing your code

Now that you’ve added your comments and created a module, let’s test that your code is still working properly.
Rather than typing your function’s code into IDLE’s prompt, bring the nester.py file into IDLE’s edit window,
and then press F5 to run the module’s code:

Note that the

comments are
color coded.

Nothing appears to happen, other than the Python shell “restarting” and an empty prompt appearing:

>>> ================================ RESTART ================================

>>>
>>>

What’s happened is that the Python interpreter has reset and the code in your module has executed. The code
defines the function but, other than that, does little else. The interpreter is patiently waiting for you to do
something with your newly defined function, so let’s create a list of lists and invoke the function on it:

>>> movies = [ Define the list of movies facts

"The Holy Grail", 1975, "Terry Jones & Terry Gilliam", 91, from Chapter 1.
["Graham Chapman",
["Michael Palin", "John Cleese", "Terry Gilliam", "Eric Idle", "Terry Jones"]]]

>>> print_lol(movies) Invoke the function on the list.

The Holy Grail
1975
Terry Jones & Terry Gilliam
91
Cool. Your code continues todata
Graham Chapman function as expected. The
Michael Palin in the list of lists is displayed
John Cleese on screen.
Terry Gilliam
Eric Idle
Terry Jones

you are here 4   39

 distribution plan

Prepare your distribution

In order to share your newly created module, you need to prepare a
distribution. This is the Python name given to the collection of files that
together allow you to build, package, and distribute your module.
Once a distribution exists, you can install your module into your local copy Do this!
of Python, as well as upload your module to PyPI to share with the world.
Follow along with the process described on these two pages to create a
distribution for your module.

1 Begin by creating a folder for your module. Follow along with each of
With the folder created, copy your nester.py module file into the the steps described on these
folder. To keep things simple, let’s call the folder nester: pages. By the time you reach
the end, your module will
have transformed into a
Python distribution.
The “nester.py”
module file.

The newly created

“nester” folder (or
nester directory).

2 Create a file called “setup.py” in your new folder.

This file contains metadata about your distribution. Edit this file by adding the following code:

Import the
“setup” function
from Python’s from distutils.core import setup
distribution utilities. Associate your module’s
setup( metadata with the setup
name = 'nester', function’s arguments.
These are the version = '1.0.0',
setup function’s py_modules = ['nester'],
argument names. author = 'hfpython', These are the
author_email = 'hfpython@headfirstlabs.com',
values Head First
Labs use with
url = 'http://www.headfirstlabs.com', their modules;
description = 'A simple printer of nested lists', your metadata
) will be different.

40   Chapter 2
 sharing your code

Build your distribution

You now have a folder with two files in it: your module’s code in nester.py Note: if you ar
and metadata about your module in setup.py. Now, it’s time to build your replace “python3e using Windows,
distribution. with “c:\Python3” in these commands
1\python.exe”.
3 Build a distribution file.
The distribution utilities include all of the smarts required to build a distribution. Open a terminal
window within your nester folder and type a single command: python3 setup.py sdist.
Enter the
command at
File Edit Window Help Build

$ python3 setup.py sdist

the prompt. running sdist
running check
warning: sdist: manifest template 'MANIFEST.in' does not exist

warning: sdist: standard file not found: should have README

A collection of writing manifest file 'MANIFEST'

status messages creating nester-1.0.0
appears on screen, making hard links in nester-1.0.0...
confirming the
hard linking nester.py -> nester-1.0.0
hard linking setup.py -> nester-1.0.0
creation of your creating dist
distribution. Creating tar archive
removing 'nester-1.0.0' (and everything under it)
$

4 Install your distribution into your local copy of Python.

Staying in the terminal, type this command: sudo python3 setup.py install.
File Edit Window Help Install
$ python3 setup.py install
running install
running build
running build_py
Another bunch of creating build
status messages creating build/lib
appear on screen, copying nester.py -> build/lib
confirming the running install_lib
copying build/lib/nester.py -> /Library/Frameworks/Python.
installation of framework/Versions/3.1/lib/python3.1/site-packages
your distribution. byte-compiling /Library/Frameworks/Python.framework/Versions/3.1/
lib/python3.1/site-packages/nester.py to nester.pyc
running install_egg_info
Writing /Library/Frameworks/Python.framework/Versions/3.1/lib/
python3.1/site-packages/nester-1.0.0-py3.1.egg-info

Your distribution is ready.

you are here 4   41

ready for distribution

A quick review of your distribution

Thanks to Python’s distribution utilities, your module has been transformed into
a distribution and installed into your local copy of Python.
You started with a single function, which you entered into a file called
nester.py, creating a module. You then created a folder called nester
to house your module. The addition of a file called setup.py to your
folder allowed you to build and install your distribution, which has resulted
in a number of additional files and two new folders appearing within
your nester folder. These files and folders are all created for you by the
distribution utilities.

Before Setup After Setup

nester nester
Your code is in A list of files in
this file. your distribution is
in this file.
nester.py MANIFEST
setup.py Here are your
new folders.
Your metadata is build
in this file.

lib Your code is

in this file.

nester.py

dist
This is your
distribution package.
Your code is in nester-1.0.0.tar.gz
this file. nester.py
nester.pyc A “compiled” version
of your code is in
setup.py
this file.
Your metadata is
in this file.
42   Chapter 2
 sharing your code

Import a module to use it

Now that your module is built, packaged as a distribution, and installed, let’s
see what’s involved in using it. To use a module, simply import it into your
programs or import it into the IDLE shell:

Use the Python

keyword “import”… …and provide the name
import nester of your module.

Note: you don’t need to

include the “.py” extension
when naming your module.
The import statement tells Python to include the nester.py module in
your program. From that point on, you can use the module’s functions as if
they were entered directly into your program, right? Well…that’s what you
might expect. Let’s check out the validity of your assumption.

Write a small program that imports your newly created module, defines a small list called “cast,”
and then uses the function provided by your module to display the contents of the list on screen.
Use the following list data (all strings): Palin, Cleese, Idle, Jones, Gilliam, and Chapman.

Open your program in IDLE’s edit window, and then press F5 to execute your code. Describe
what happens in the space below:

you are here 4   43

idle error

You were to write a small program that imports your newly created module, defines a small list
called “cast,” and then uses the function provided by your module to display the contents of the
list on screen. You were to use the following list data (all strings): Palin, Cleese, Idle, Jones,
Gilliam, and Chapman.

import nester
It’s a simple three-
line program. There’s
nothing too difficult cast = ['Palin’, 'Cleese’, 'Idle’, 'Jones’, 'Gilliam’, 'Chapman’]
here.
print_lol(cast)
Open your program in IDLE’s edit window, and then press F5 to execute your code. Describe
what happens in the space below:

But it didn’t work! IDLE gives an error, and the program does not run!

With your program in the IDLE edit window, pressing F5 (or choosing Run Module from the Run menu) does
indeed cause problems:

Your program does not appear to have executed and an error message is reported:

>>> ================================ RESTART ================================

>>>
Traceback (most recent call last):
File "/Users/barryp/HeadFirstPython/chapter2/try_nester.py", line 4, in <module>
print_lol(cast)
NameError: name 'print_lol' is not defined With your program in IDLE, pre
>>> a NameError…it looks like you ssing F5 causes
found!!! r fun ction can’t be

44   Chapter 2
 sharing your code

Python’s modules implement namespaces

All code in Python is associated with a namespace.
Code in your main Python program (and within IDLE’s shell) is associated That’s a double underscore in
with a namespace called __main__. When you put your code into its own front of the word “main” and
module, Python automatically creates a namespace with the same name as after it.
your module. So, the code in your module is associated with a namespace
called nester.

I guess namespaces are like family names?

If someone is looking for Chris, we need to know
if it’s Chris Murray or Chris Larkin, right? The
family name helps to qualify what we mean, as do
namespace names in Python.

Yes, namespace names are like family names.

When you want to refer to some function from a module
namespace other than the one you are currently in, you need
to qualify the invocation of the function with the module’s
namespace name.
So, instead of invoking the function as print_lol(cast)
you need to qualify the name as nester.print_lol(cast).
That way, the Python interpreter knows where to look. The
format for namespace qualification is: the module’s name, followed
by a period, and then the function name.

The module name, which

identifies the namespace. The function is then invoked as
nester.print_lol(cast) normal, with “cast” provided as
the list to process.
espace
A period separates the modulee. nam
name from the function nam

you are here 4   45

ready for pypi

Let’s test this. Staying at the IDLE shell, import your module, create the list, and then try to invoke the function
without a qualifying name. You’re expecting to see an error message:

>>> import nester

>>> cast = ['Palin', 'Cleese', 'Idle', 'Jones', 'Gilliam', 'Chapman']
>>> print_lol(cast)
Traceback (most recent call last):
As expected, your code has cau sed a
lify
NameError, because you did qua
n’t
File "<pyshell#4>", line 1, in <module>

the name.
print_lol(cast)
NameError: name 'print_lol' is not defined

When you qualify the name of the function with the namespace, things improve dramatically:

>>> nester.print_lol(cast)
Palin
Cleese
Idle
This time, things work as expect
Jones
the list are displayed on screen. ed…the data items in
Gilliam
Chapman

Geek Bits

When you use a plain import statement, such as import nester,

the Python interpreter is instructed to allow you to access nester’s
functions using namespace qualification. However, it is possible to be
more specific. If you use from nester import print_lol, the
specified function (print_lol in this case) is added to the current
namespace, effectively removing the requirement for you to use
namespace qualification. But you need to be careful. If you already have
a function called print_lol defined in your current namespace, the
specific import statement overwrites your function with the imported
one, which might not be the behavior you want.

Your module is now ready for upload to PyPI.

46   Chapter 2
 sharing your code

Register with the PyPI website

In order to upload your distribution to PyPI, you need to register with the
PyPI website. This is a relatively straightforward process.
Begin by surfing over to the PyPI website at http://pypi.python.org/ and
requesting a PyPI ID:

Provide the Username

you’d like to use. Don’t try to use “hfpython
that Username is already ta,”kenbec
.
ause

Enter your chosen password

twice for confirmation
purposes. Provide a valid email address.
t
Don’t worry abPoukey
providing a PGually have
(unless you act
one).

Don’t forget to click the

“I agree” checkbox before
clicking on the Register
button.

If all of your registration details are in order, a confirmation message is sent

to the email address submitted on the registration form. The email message
contains a link you can click to confirm your PyPI registration:

plete
Click the confirmation link to com
You are now registered with PyPI.
your PyPI registration.
you are here 4   47
 register and upload

Upload your code to PyPI

You’re ready to rock! The code in your function has been placed in a module,
used to create a distribution and installed into your local copy of Python. To
upload your distribution to PyPI, complete these two steps: command-line
registration with PyPI and command-line uploading.
It might seem strange to have to register with PyPI again, seeing as you just
did this with their website. However, the command-line uploading tool needs
to be made aware of your PyPI Username and Password, and that’s what this
registration does. Don’t worry: you have to do this only once.

Instruct setup to File Edit Window Help Register

$ python3 setup.py register
register your details. running register
running check
We need to know who you are, so please choose either:
Confirm that you 1. use your existing login,
want to use your 2. register as a new user,
3. have the server generate a new password for you (and email it to you), or
just-created PyPI 4. quit
credentials. Your selection [default 1]:
1
Username: hfpython
Password:
Use your PyPI Registering nester to http://pypi.python.org/pypi
settings and save Server response (200): OK

them for future

I can store your PyPI login so future submissions will be faster.
(the login will be stored in /Users/barryp/.pypirc)
use. Save your login (y/N)?y

Note: If you try to upload

With your registration details entered and saved, you are now ready to upload “nester”, you’ll get an error aasmodule called
th
your distribution to PyPI. Another command line does the trick: already taken. § at name’s

Instruct setup ftotware

File Edit Window Help Upload
$ python3 setup.py sdist upload
upload your so PyPI. running sdist
distribution to running check
reading manifest file ‘MANIFEST’
creating nester-1.0.0
making hard links in nester-1.0.0...
Setup confirms hard linking nester.py -> nester-1.0.0
that the upload hard linking setup.py -> nester-1.0.0
is successful. Your Creating tar archive
distribution is now removing ‘nester-1.0.0’ (and everything under it)
part of PyPI. running upload
Submitting dist/nester-1.0.0.tar.gz to http://pypi.python.org/pypi
Server response (200): OK
$

48   Chapter 2
 sharing your code

Welcome to the PyPI community

Congratulations! You are now a full-fledged, card-carrying member of the
PyPI community. Your distribution has joined the over 10,000 other uploads
on PyPI. Feel free to surf on over to the PyPI website to confirm the upload.
Programmers from all over the globe are now able to download, unpack, and
install your module into their local copy of Python, which is pretty cool when
you think about it.
You’ve now written and
published your code…
Sit back, put your feet up, and wait for the plaudits to
how cool is that?
begin…

Q: Q:
a result of your Java technologies compiling
your code.) The Python interpreter is smart
Which is best: plain imports or Is it really necessary for me to
enough to skip the translation phase the next
specific imports? install my modules into my local copy of
time your module is used, because it can

A:
Python? Can’t I just put them in any old
determine when you’ve made changes to
Neither, really. Most programmers folder and import them from there?
the original module code file. If your module
mix and match based on their own personal
preference and taste (although there are
plenty of programmers willing to argue that
A: Yes, it is possible. Just bear in mind
that Python looks for modules in a very
code hasn’t changed, no translation occurs
and the “compiled” code is executed. If your
code has changed, the translation occurs
their preferred way is the “one true way”). specific list of places (recall the import (creating a new pyc file) as needed. The
sys; sys.path trick from earlier upshot of all this is that when Python sees a
Note that the from module import in this chapter). If you put your modules pyc file, it tries to use it because doing so
function form pollutes your current in a folder not listed in Python’s path list, makes everything go much faster.
namespace: names already defined in your chances are the interpreter won’t find
current namespace are overwritten by the
imported names.
them, resulting in ImportErrors. Using the
distribution utilities to build and install your
Q: Cool. So I can just provide my
users with the pyc file?
Q:
module into your local copy of Python avoids
And when I press F5 in IDLE’s edit these types of errors.
A:pyc
Q:
No, don’t do that, because the use of
window, it’s as if the module’s code is
imported with an import statement, right? I noticed the distribution utiliites the file (if found) is primarily a runtime
optimization performed by the interpreter.

A:
created a file called nester.pyc.
Yes, that is essentially what happens. What’s up with that?
Q: So, can I delete the pyc file if I don’t
A:
The code in your edit window is compiled
and executed by Python, and any names need it?
That’s a very good question. When
in the edit window are imported into the
namespace being used by IDLE’s shell. This
the interpreter executes your module code
for the first time, it reads in the code and A: Sure, if you really want to. Just be
aware that you lose any potential runtime
is handy, because it makes it easy to test translates it into an internal bytecode format
functionality with IDLE. But bear in mind that which is ultimately executed. (This idea is optimization.
outside of IDLE, you still need to import your very similar to the way the Java JVM works:
module before you can use its functionality. your Java code is turned into a class file as

you are here 4   49

conflicting requests

With success comes responsibility

Lots of programmers from many different locations are using your module.
And some of these programmers are looking for more features.

Hang on a second.
We really love your code, I kinda like the way it
but is there any chance this works right now. I vote
thing could print the data NOT to change it.
to screen and indent each
nested list whenever one is
found?

Any changes to the way you

Likes what you’ve function works are likely to r
done, but could annoy this guy.
be happier.

Requests for change are inevitable

You need to keep your current users happy by maintaining the existing
functionality, while at the same time providing enhanced functionality to
those users that require it. This could be tricky.

What are your options here?

50   Chapter 2
 sharing your code

Life’s full of choices

When it comes to deciding what to do here, there’s
no shortage of suggestions.

That’s soooo easy. Simply create

another function called “print_lol2”,
right? You could then import the
function you want using the specific
form of the import statement. It’s not
that hard, really...

Yeah, that might just work.

You could edit your module’s code and define a new function called
print_lol2, then code up the function to perform the nested printing
When you want to use the original function, use this specific form of the
import statement: from nester import print_lol. When you want
to use the new, improved version of the function, use this import statement:
from nester import print_lol2.
Which would work, but…

But that suggestion is twice

the work...which might be OK
sometimes...but the creation of a
second, almost identical, function
seems wasteful to me.

Right. A second function is wasteful.

Not only are you introducing an almost identical function to your
module, which might create a potential maintenance nightmare, but
you’re also making things much more difficult for the users of your
module, who must decide ahead of time which version of the function
they need. Adding a second function makes your module’s application
programming interface (API) more complex than it needs to be.

There has to be a better strategy, doesn’t there?

you are here 4   51

add an argument

Control behavior with an extra argument

If you add an extra argment to your function, you can handle indentation
within your current code without too much trouble.

Yikes! I should’ve thought about that

myself... I probably need to go easy on
the coffee. Of course, it’s clear to me
now: adding another argument to your
function gives you options.

Take your function to the next level

At the moment, your function has a single argument: the_list. If you add
a second argument called level, you can use it to control indentation. A
positive value for level indicates the number of tab-stops to include when
displaying a line of data on screen. If level is 0, no indentation is used; if
it’s 1, use a single tab-stop; if it’s 2, use two tab-stops; and so on.
It’s clear you are looking at some sort of looping mechanism here, right? You
already know how to iterate over a variably sized list, but how do you iterate a
fixed number of times in Python?

Does Python provide any functionality that can help?

52   Chapter 2
 sharing your code

Before your write new code, think BIF

When you come across a need that you think is generic, ask yourself if there’s
a built-in function (BIF) that can help. After all, iterating a fixed number of
times is something you’ll need to do all the time.
And remember: Python 3 includes over 70 BIFs, so there’s a lot of
functionality waiting to be discovered.

Use your pencil to draw a line matching each BIF to the correct description.
The first one is done for you. Once you have all your lines drawn, circle the
BIF you think you need to use in the next version of your function.

BIF What the BIF does

list() Creates a numbered list of paired-data, starting

from 0.

range() Returns the unique identification for a Python

data object.

enumerate() A factory function that creates a new, empty list.

int() Returns the next item from a iterable data

structure such as a list.

id() Returns an iterator that generates numbers in a

specified range on demand and as needed.

next() Converts a string or another number to an

integer (if possible).

you are here 4   53

who did what

SOLUTION
You were to use your pencil to draw a line matching each BIF to the correct
description. Once you had all your lines drawn, you were to circle the BIF
you think you need to use in the next version of your function.

BIF What the BIF does

list() Creates a numbered list of paired-data, starting

from 0.

range() Returns the unique identification for a Python

data object.
This BIF
looks
interesting. enumerate() A factory function that creates a new, empty list.

int() Returns the next item from a iterable data

structure such as a list.

id() Returns an iterator that generates numbers in a

specified range on demand and as needed.

next() Converts a string or another number to an

integer (if possible).

The range() BIF iterates a fixed number of times

The range() BIF gives you the control you need to iterate a specific
number of times and can be used to generate a list of numbers from zero up-
to-but-not-including some number. Here’s how to use it: Generate numbers up-to-
but-not-including 4.
“num” is the target identifieer
and is assigned each of th ()”
for num in range(4):
numbers generated by “range
The numbers 0, 1, 2, and 3 will
print(num)
in turn.
appear on screen.
54   Chapter 2
 sharing your code

Q: Don’t I need to import the BIFs in Q: I get how range() works, but surely Q: So BIFs are actually good for me?

A:
order to use them in my program? I could just as easily use a while loop to

A:
do the same thing? BIFs exist to make your programming
No. For all intents and purposes, the
BIFs are specifically imported into every
Python program as well as IDLE.
A: Yes, you can, but it’s not as elegant
as using range(). Seriously, though,
experience as straightforward as possible
by providing a collection of functions that
provide common solutions to common

Q: the while equivalent not only requires problems. Since they are included with
So the BIFs must belong to the you to write more code, but it also makes it Python, you are pretty much assured that
__main__ namespace, right? your responsibility to worry about loop state, they have been tested to destruction and
do “exactly what it says on the tin.” You can
A:
whereas range() worries about this for
you. As a general rule, Python programmers depend on the BIFs. Using them gives your
No. They are automatically imported
look for ways to reduce the amount of code program a leg up and makes you look good.
into the __main__ namespace, but the So, yes, the BIFs are good for you!
BIFs have their very own namespace called they need to write and worry about, which
leads to better code robustness, fewer errors,
(wait for it) __builtins__.
and a good night’s sleep.

Now that you know a bit about the range() BIF, amend your function to use range() to
indent any nested lists a specific number of tab-stops.
Hint: To display a TAB character on screen using the print() BIF yet avoid taking a new-line
(which is print()’s default behavior), use this Python code: print("\t", end='').

"""This is the "nester.py" module and it provides one function called print_lol()
which prints lists that may or may not include nested lists."""
Include the name of the extra arg
def print_lol(the_list, ):
ument.
"""This function takes a positional argument called "the_list", which
is any Python list (of - possibly - nested lists). Each data item in the
provided list is (recursively) printed to the screen on it's own line."""

for each_item in the_list: Don’t forget to edit

if isinstance(each_item, list): the comment.
print_lol(each_item)
ed
else: Add code here to take the requir
number of tab-stops.

print(each_item)

you are here 4   55

using range

Now that you know a bit about the range() BIF, you were to amend your function to use
range() to indent any nested lists a specific number of tab-stops.
Hint: To display a TAB character on screen using the print() BIF yet avoid taking a new-line
(which is print()’s default behavior), use this Python code: print("\t", end='').
"""This is the "nester.py" module and it provides one function called print_lol()
which prints lists that may or may not include nested lists."""

def print_lol(the_list, level ):

"""This function takes a positional argument called "the_list", which
is any Python list (of - possibly - nested lists). Each data item in the
provided list is (recursively) printed to the screen on it's own line."""
A second argument called “level" is used to insert tab-stops when a nested list is encountered."""
for each_item in the_list:
if isinstance(each_item, list):
Use the value of “level” to con trol
print_lol(each_item)
how many tab-stops are use d.
else:
for tab_stop in range(level):
print("\t", end='') Display a TAB character for eac
print(each_item) of indentation. h level

It’s time to test the new version of your function. Load your module file into IDLE, press F5 to import the function
into IDLE’s namespace, and then invoke the function on your movies list with a second argument:
Invoke your function, being sure to provide
>>> print_lol(movies, 0) a second argument.
The Holy Grail
1975
The data in “movies” starts to appear on screen…
Terry Jones & Terry Gilliam
91
…then all hell breaks loose! Something is not righ
Traceback (most recent call last): t here.
File "<pyshell#2>", line 1, in <module>
print_lol(movies,0)
File "/Users/barryp/HeadFirstPython/chapter2/nester/nester.py", line 14, in print_lol
print_lol(each_item)
Here’s your clue as to
TypeError: print_lol() takes exactly 2 positional arguments (1 given)
what’s gone wrong.

Your code has a TypeError, which

56   Chapter 2 caused it to crash.
 sharing your code

Python tries its best to run your code

Unlike compiled languages (such as C, Java, C#, and others), Python doesn’t
completely check the validity of your code until it runs. This allows Python to do
some rather cool things that just aren’t possible in those other languages, such
as dynamically defining functions at runtime. This, of course, can be very
flexible and powerful.
The cost to you, however, is that you need to be very careful when writing
your code, because something that typically would be caught and flagged as
an “error” by a traditional, statically typed, compiled language often goes
unnoticed in Python.

...OK, C++ syntax fine...continuing to

parse...whoops! You’re trying to use a
function before it’s declared?!? That’s NOT
allowed around here... I’m outta here.

Please wait.
Compiling
your C++
code…
Ah ha! The old “calling
a function before you’ve defined
it yet” trick, eh? I’ll just make a note
in case you define it later at runtime.
You are planning to do that, right?
Please don’t disappoint me, or I’ll give
you an error...

Running your
Python code
right now…

Take another look at the error on the opposite

page. Why do you think Python is giving you this
particular error? What do you think is wrong?

you are here 4   57

 line check

Trace your code

When you’re trying to work out what went wrong with a program that looks
like it should be OK, a useful technique is to trace what happens as each line
of code executes. Here’s the code that you are currently working with. At
only three lines long (remember: the creation of the list is one line of code), it
doesn’t look like it should cause any trouble:

import nester Thes two lines look OK.

movies = [ "The Holy Grail", 1975, "Terry Jones & Terry Gilliam",
91,["Graham Chapman", ["Michael Palin",
"John Cleese", "Terry Gilliam", "Eric Idle",
"Terry Jones"]]]
h two arguments,
You are invoking the function wit
nester.print_lol(movies, 0) so that’s OK, too.

With the data assigned to the function’s arguments, the function’s code starts The “movies” list is assigned
to execute on each data item contained within the passed-in list: “the_list”, and the value 0 isto
assigned to “level”.
To save space, the
entire comment is def print_lol(the_list, level):
not shown.
"""This function ... """
Process each item in
the list… for each_item in the_list:
…then decide what if isinstance(each_item, list):
If the data item is a
to do next based on list, recursively invoke
whether or not the print_lol(each_item) the function…hang on
data item is a list. else: a second, that doesn’t
look right!?
for tab_stop in range(level):
print("\t", end='')
print(each_item)

58   Chapter 2
 sharing your code

Work out what’s wrong

There’s your problem: the recursive invocation of your function is using the
old function signature that required only one argument. The new version of
your function requires two arguments.
The fix is easy: provide the correct number of arguments when calling the
new version of your function. So, this snippet of code from your function:

if isinstance(each_item, list):
print_lol(each_item)

needs to be rewritten to specify the correct number of arguments:

if isinstance(each_item, list):
print_lol(each_item, level)

Not so fast. Surely the nested

list needs to be printed after a specific
number of tab-stops? At the moment, your
code sets “level” to 0 but never changes the
value, so “level” never has any impact on your
displayed output...

Right. Your use of “level” needs one final tweak.

The whole point of having level as an argument is to allow you to
control the nested output. Each time you process a nested list, you need
to increase the value of level by 1. Your code snippet needs to look
like this:

if isinstance(each_item, list):
print_lol(each_item, level+1)

of level by 1 each
Simply increment the value you function.
It’s time to perform that update. time you recursively invoke r

you are here 4   59

fresh pypi

Update PyPI with your new code

Go ahead and edit your nester.py module (in the nester folder) to
invoke your function properly. Now that you have a new version of your
module, it’s a good idea to update the distribution that you uploaded to PyPI.
With your code amended, there’s also a small change needed to your
distribution’s setup.py program. You’ve changed your API, so adjust the
value associated with version in setup.py. Let’s move from version 1.0.0
to 1.1.0:
from distutils.core import setup
Change the value
associated wit
setup(
to indicate to hP “version”
name = 'nester', this is indeed a yPI that
version. new
version = '1.1.0',
py_modules = ['nester'],
author = 'hfpython',
author_email = 'hfpython@headfirstlabs.com',
url = 'http://www.headfirstlabs.com',
description = 'A simple printer of nested lists',
)

Just as you did when you created and uploaded your distribution, invoke the
setup.py program within your distribution folder to perform the upload:

File Edit Window Help UploadAgain

$ python3 setup.py sdist upload
running sdist
running check
reading manifest file 'MANIFEST'
creating nester-1.1.0
making hard links in nester-1.1.0...
hard linking nester.py -> nester-1.1.0
Don’t you just hard linking setup.py -> nester-1.1.0
Creating tar archive
love those removing 'nester-1.1.0' (and everything under it)
“200 OK” running upload
messages? Submitting dist/nester-1.1.0.tar.gz to http://pypi.python.org/pypi
Server response (200): OK
$

Your new distribution is now available on PyPI.

60   Chapter 2
 sharing your code

Mark: Take a look at this, guys…the nester module has been

Cool.There’s a new updated on PyPI.
version of “nester” on
Bob: Version 1.1.0…
PyPI. Take a look.
Laura: I wonder what’s changed?
Mark: It still works with nested lists of lists, but now you can see the
nested structure on screen, which I think is pretty cool.
Laura: And useful. I’ve been waiting for that feature.
Bob: Eh…OK…but how do I upgrade my existing local copy?
Mark: Just follow the same steps as when you downloaded and
installed nester from PyPI the first time.
Bob: So I download the package file, unpack it, and ask setup.py to
install it into my Python for me?
Mark: Yes. It couldn’t be any easier.
Laura: And what about my existing version of nester; what
happens to that “old” version?
Bob: Yeah…do I have two nester modules now?
Mark: No. When you use setup.py to install the latest version
it becomes the current version and effectively replaces the previous
module, which was the 1.0.0 release.
Bob Mark Laura
Bob: And PyPI knows to give you the latest version of the module, too,
right?
Mark: Yes, when you surf the PyPI website and search for nester,
you are always provided with the latest version of the module.
Laura: Well, I use this module all the time and I’ve been waiting for
this feature. I think I’ll update right away.
Mark: I’ve already upgraded mine, and it works a treat.
Bob: Yeah, I use it a lot, too, so I guess I’ll keep my system up to date
and install the latest version. It’s probably not a good idea to rely on
out-of-date software, right?
Mark: I’d say. And, there’s nothing quite like progress.
Laura: Catch you later, guys, I’ve got work to do.
Bob: Me, too. I’m off to PyPI to grab the latest nester and install
it into my local copy of Python. I’ll give it a quick test to confirm all is
OK.
Mark: Later, dudes…

you are here 4   61

unhappy user

You’ve changed your API

Your new version of nester is indeed better, but not for all your users.

Ah, phooey! I can’t believe it... I installed

the latest version of “nester” from PyPI, and
now all of my code that uses your function is
not working. What did you do?!?

Get with the program,

Bob. You have TypeError’s
everywhere...

In your rush to release the lates and greatest version of your module, you
forgot about some of your existing users. Recall that not all of your users want
the new nested printing feature. However, by adding the second argument
to print_lol(), you’ve changed your function’s signature, which means
your module has a different API. Anyone using the old API is going to have
problems.
The ideal solution would be to provide both APIs, one which switches on the
new feature and another that doesn’t. Maybe the feature could be optional?

But how would that work?

62   Chapter 2
 sharing your code

Use optional arguments

To turn a required argument to a function into an optional argument, provide
the argument with a default value. When no argument value is provided, the
default value is used. When an argument value is provided, it is used instead
of the default. The key point is, of course, that the default value for the
argument effectively makes the argument optional.
To provide a default value for any function argument, specify the default
Both argumen
are REQUIREDts.
value after the argument name:

def print_lol(the_list, level):

The addition of a
def print_lol(the_list, level=0): default value has
turned “level” into an
OPTIONAL argument.

With the default value for the argument defined, you can now invoke the
function in a number of different ways:

Invoke the function and

provide both arguments.
nester.print_lol(movies, 0)
Invoke the function with
one argument and use
nester.print_lol(movies) the default value for
Invoke the function with the second.
both arguments, but nester.print_lol(movies, 2)
provide an alternative
starting value for the
second argument.

Your function now supports different signatures, but the

functonality remains as it was.

you are here 4   63

idle session

Amend your code to give the level argument a default value of 0 and then load your code into the IDLE editor.
Press F5 to load the code into the shell and then follow along to confirm that the latest version of your function
works as expected. Start be defining a short list of lists and use the function to display the the list on screen:
>>> names = ['John', 'Eric', ['Cleese', 'Idle'], 'Michael', ['Palin']]
>>> print_lol(names, 0)
John
Eric
The standard behavior works
Cleese as expected, with nested
Idle lists indented.
Michael
Palin

Now try to do the same thing without specifiying the second argument. Let’s rely on the default value kicking in:
>>> print_lol(names)
John

ument, the
Eric
Cleese Without specifying the second . arg
Idle default is used and works, too
Michael
Palin

Now specify a value for the second argument and note the change in the function’s behavior:
>>> print_lol(names, 2)
John
Eric Specify an alternative value for
Cleese argument and the indenting sta the second
Idle that level. rt s from
Michael
Palin

One final example provides what looks like a silly value for the second argument. Look what happens:
>>> print_lol(names, -9)
John
tch es OFF the indenting, as thexae
neg ati ve val ue eff ect ively swi ctly
Using a a positive integer. This looks
Eric
“le vel ” is unl ikely to bec om e
Cleese count for sion 1.0.0, right?
Idle like the original output from ver
Michael
Palin

64   Chapter 2
 sharing your code

Your module supports both APIs

Well done! It looks like your module is working well, as both APIs, the original
1.0.0 API and the newer 1.1.0 API, can now be used.
Let’s take a moment to create and upload a new distibution for PyPI. As
before, let’s amend the version setting in the setup.py program:

name = 'nester', Once again, be sure to change

version = '1.2.0', the value associated with
py_modules = ['nester'],
“version” in “setup.py”.

And with the code changes applied, upload this new version of your
distribution to PyPI:

File Edit Window Help UploadThree

$ python3 setup.py sdist upload
running sdist
running check
reading manifest file 'MANIFEST'
creating nester-1.2.0
making hard links in nester-1.2.0...
hard linking nester.py -> nester-1.2.0
hard linking setup.py -> nester-1.2.0
This all looks Creating tar archive
fine and removing 'nester-1.2.0' (and everything under it)
running upload
dandy. Submitting dist/nester-1.2.0.tar.gz to http://pypi.python.org/pypi
Server response (200): OK
$

Success! The messages from setup.py confirm that the your latest version
of nester is up on PyPI. Let’s hope this one satisfies all of your users.

Consider your code carefully. How might some of

your users still have a problem with this version of
your code?

you are here 4   65

faulty default

Your API is still not right

Although the API lets your users invoke the function in its original form, the
nesting is switched on by default. This behavior is not required by everyone and
some people aren’t at all happy.

Another
version of “nester” has I can’t believe it! My
been released...but its programs were back to running
default behavior might not fine, but now everything is
be what you want. indented. Has this thing
changed again?!?

Funny...it works fine

for me.

Of course, if you have some functionality that really ought to be optional

(that is, not the default), you should adjust your code to make it so. But how?
One solution is to add a third argument which is set to True when the
indenting is required and False otherwise. If you ensure that this argument
is False by default, the original functonality becomes the default behavior
and users of your code have to request the new indenting feature explicitly.

Let’s look at adding this final revision.

66   Chapter 2
 sharing your code

1 Amend your module one last time to add a third argument to your function. Call your argument
indent and set it initially to the value False—that is, do not switch on indentation by default.
In the body of your function, use the value of indent to control your indentation code.
Note: to save a bit of space, the comments from the module are not shown here. Of course, you
need to make the necessary adjustments to your comments to keep them in sync with your code.

Put the extra argument

here.

def print_lol(the_list, , level=0):

What needs to
for each_item in the_list:
go in here?
if isinstance(each_item, list):
print_lol(each_item, , level+1)
else:

de
Add a line of con
for tab_stop in range(level):

to control whers. print("\t", end='')

indenting occu print(each_item)

2 With your new code additions in place, provide the edit you would recommend making to the
setup.py program prior to uploading this latest version of your module to PyPI:

3 Provide the command you would use to upload your new distribution to PyPI:

you are here 4   67

adding an argument

1 You were to amend your module one last time to add a third argument to your function. You were
to call your argument indent and set it initially to the value False—that is, do not switch on
indentation by default. In the body of your function, you were to use the value of indent to
control your indentation code.

Did you include the

default value?

def print_lol(the_list, indent=False , level=0):

s changed,
Your signatureuphadate this
for each_item in the_list: so be sure to
if isinstance(each_item, list): invocation.
print_lol(each_item, indent , level+1)
else:
if indent : Don’t forget the colon at the end
of the “if” line.
A simple “if” s
statement doe
for tab_stop in range(level):

the trick.
print("\t", end='')
print(each_item)
A sweet alternative to this “for”
is this code: print("\t" * level, loop
end='').

2 With your new code additions in place, you were to provide the edit you would recommend
making to the setup.py program prior to uploading this latest version of your module to PyPI:

Edit “setup.py” so that it reads: version = ‘1.3.0’,

It’s a new version of your module,
the value associated with “versi so be sure to change
on” in your “setup.py” file.

3 You were to provide the command you would use to upload your new distribution to PyPI:

python3 setup.py sdisk upload

use “C:\Python31\python.exe”
Remember: if you are on Windows
instead of “python3”.

68   Chapter 2
 sharing your code

A final test of the functonality should convince you that your module is now working exactly the way you and your
users want it to. Let’s start with the original, default behavior:

>>> names = ['John', 'Eric', ['Cleese', 'Idle'], 'Michael', ['Palin']]

>>> print_lol(names)
John
The original, default functionalit
is restored (that should please y
Eric
Cleese
Idle Bob).
Michael
Palin

Next, turn on indentation by providing True as the second argument:

>>> names = ['John', 'Eric', ['Cleese', 'Idle'], 'Michael', ['Palin']]

>>> print_lol(names, True)
John
Eric
By providing a second argument ent,
Cleese
it’s possible to switch on ind ed
Idle
output (keeping Laura hap py) .
Michael
Palin

And, finally, control where indentation begins by providing a third argument value:

>>> names = ['John', 'Eric', ['Cleese', 'Idle'], 'Michael', ['Palin']]

>>> print_lol(names, True, 4)
John
Eric
Indenting from a specific
Cleese tab-stop is also possible.
Idle
Michael
Palin

Go ahead and edit your

Do this! setup.py file; then upload
your distribution to PyPI.

you are here 4   69

one module for all

Your module’s reputation is restored

Congratulations! Word of your new and improved module is spreading fast.

Lots of PyPI hits

My programs are back to
already. I told you
working the way I want
this was good.
them to, so I’m a happy
guy. Thanks!
Great work! I love
that I can switch
indentation on and off.

This is as close as Bob gets

to a smile. But trust us,
he’s happy. §

Your Python skills are starting to build

You’ve created a useful module, made it shareable, and uploaded it to the
PyPI website. Programmers all over the world are downloading and using
your code in their projects.

Keep up the good work.

70   Chapter 2
 sharing your code

Your Python Toolbox

You’ve got Chapter 2 under your

CHAPTER 2
belt and you’ve added some key
Python goodies to your toolbox. ƒƒ A module is a text file that contains Python
code.
ƒƒ The distribution utilities let you turn your
module into a shareable package.

Python Lingo e
ƒƒ The setup.py program provides

a “triple-quoted string” to includ

metadata about your module and is used
• Use
code . to build, install, and upload your packaged
a multiple-line comment in your distribution.

• “PyPI” is the Pyth

on Package Index and ƒƒ Import your module into other programs
is well worth a visit. using the import statement.

• A “namespace” is a
place in Python’s ƒƒ Each module in Python provides its own

memory where names exist.

namespace, and the namespace name
is used to qualify the module’s functions
• Python’s main name
space is known as when invoking them using the module.
__main__.
function() form.
ƒƒ Specifically import a function from a module
into the current namespace using the from
IDLE Notes module import function form of
the import statement.
• Press F5 to “run” the code in the IDLE ƒƒ Use # to comment-out a line of code or
edit window. to add a short, one-line comment to your
program.
• When you press F5
to “load” a
code into the IDLE shell, the momodule’s ƒƒ The built-in functions (BIFs) have their own
names are specifically imported intdule’s namespace called __builtins__,
IDLE’s namespace. This is a conve o which is automatically included in every
when using IDLE. Within your co nience
Python program.

need to use the import statementde, you ƒƒ The range() BIF can be used with for to
explicitly. iterate a fixed number of times.
ƒƒ Including end=’’ as a argument to the
print() BIF switches off its automatic
inclusion of a new-line on output.
ƒƒ Arguments to your functions are optional if
you provide them with a default value.

you are here 4   71

3 files and exceptions

Dealing with errors

I always thought he was

exceptional...especially when it
comes to processing my files.

It’s simply not enough to process your list data in your code.
You need to be able to get your data into your programs with ease, too. It’s no surprise
then that Python makes reading data from files easy. Which is great, until you consider
what can go wrong when interacting with data external to your programs…and there
are lots of things waiting to trip you up! When bad stuff happens, you need a strategy for
getting out of trouble, and one such strategy is to deal with any exceptional situations
using Python’s exception handling mechanism showcased in this chapter.

this is a new chapter   73

 getting data in

Data is external to your program

Most of your programs conform to the input-process-output model: data comes
in, gets manipulated, and then is stored, displayed, printed, or transferred.

I’m ready for your

DBMS data...just give it to
me, baby!
Data comes fro m
lots of places.

So far, you’ve learned how to process data as well as display it on screen.

But what’s involved in getting data into your programs? Specifically, what’s
involved in reading data from a file?

How does Python read data from a file?

74   Chapter 3
 files and exceptions

It’s all lines of text

The basic input mechanism in Python is line based: when read into your
program from a text file, data arrives one line at a time.
Python’s open() BIF lives to interact with files. When combined with a for
statement, reading files is straightforward.

Your data in a text

file called “sketch.txt”.

Your data
open() as individual
lines.

When you use the open() BIF to access your data in a file, an iterator is
created to feed the lines of data from your file to your code one line at a time.
But let’s not get ahead of ourselves. For now, consider the standard open-
process-close code in Python:

Do this!

Open… the_file = open('sketch.txt')

Create a folder called
# Do something with the data HeadFirstPython and a
…Process… subfolder called chapter3.
# in "the_file".
With the folders ready,
…Close. the_file.close() download sketch.txt from
the Head First Python support
website and save it to the
chapter3 folder.

Let’s use IDLE to get a feel for Python’s file-input mechanisms.

you are here 4   75

idle session

Start a new IDLE sesson and import the os module to change the current working directory to the folder that
contains your just-downloaded data file:
>>> import os Import “os” from the Standard Library.
>>> os.getcwd() What’s the current working directory?
'/Users/barryp/Documents'
Change to the folder that contains your data file.
>>> os.chdir('../HeadFirstPython/chapter3')
>>> os.getcwd() Confirm you are now in the right place.
'/Users/barryp/HeadFirstPython/chapter3'

Now, open your data file and read the first two lines from the file, displaying them on screen:
>>> data = open('sketch.txt') Open a named file and assign the file to a file object called “data”.
>>> print(data.readline(), end='')
Man: Is this the right room for an argument?
Use the “readline()” method to grab a
>>> print(data.readline(), end='') line from the file, then use the “print()”
Other Man: I've told you once. BIF to display it on screen.

Let’s “rewind” the file back to the start, then use a for statement to process every line in the file:
>>> data.seek(0) Use the “seek()” method to return to the start of the file.
0 And yes, you can use “tell()” with Python’s files, too.
>>> for each_line in data:
This code should look familiar: it’s
iteration using the file’s data as a standard
print(each_line, end='')

input.
Man: Is this the right room for an argument?
Other Man: I've told you once.
Man: No you haven't!
Other Man: Yes I have. Every line of the data is
Man: When? displayed on screen (although
Other Man: Just now. for space reasons, it is
Man: No you didn't! abridged here).
...
Man: (exasperated) Oh, this is futile!!
(pause)
Other Man: No it isn't!
Man: Yes it is!
>>> data.close() Since you are now done with the file, be sure to close it.

76   Chapter 3
 files and exceptions

Take a closer look at the data

Look closely at the data. It appears to conform to a specific format:
A colon, followed by a The line spoken by the
The cast space character cast member
member’s role
Man: Is this the right room for an argument?
Other Man: I’ve told you once.
Man: No you haven’t!
Other Man: Yes I have.
Man: When?
Other Man: Just now.
Man: No you didn’t!

With this format in mind, you can process each line to extract parts of the line
as required. The split() method can help here:

Man: Is this the right room for an argument?

()” method
Invoke the “splhit the “each_line” each_line.split(":") This tells “split()”
associated wit ak the string what to split on.
string and bre is found.
whenever a “:”
Man
Is this the right room for an argument?

The split() method returns a list of strings, which are assigned to a list of
target identifiers. This is known as multiple assignment:

e left… …are assigned the strings return

A list of target identifiers on th ed by “split()”.

(role, line_spoken) = each_line.split(":")

Using the example data from abo

ve, “role” is …“line_spoken”: is assigned the string “Is this
assigned the string “Man”, wh ?”
ereas… the right room for an argument
Well? Is it? §
you are here 4   77
idle session

Let’s confirm that you can still process your file while splitting each line. Type the following code into IDLE’s shell:
>>> data = open('sketch.txt') Open the data file.
>>> for each_line in data:
(role, line_spoken) = each_line.split(':') each part from
Process the data, extractinhg par t on screen.
print(role, end='') each line and displaying eac
print(' said: ', end='')
print(line_spoken, end='')

Man said: Is this the right room for an argument?

Other Man said: I've told you once.
Man said: No you haven't!
Other Man said: Yes I have.
Man said: When? This all looks OK.
Other Man said: Just now.
Man said: No you didn't!
Other Man said: Yes I did!
Man said: You didn't!
Other Man said: I'm telling you, I did!
Man said: You did not!
Other Man said: Oh I'm sorry, is this a five minute argument, or the full half hour?
Man said: Ah! (taking out his wallet and paying) Just the five minutes.
Other Man said: Just the five minutes. Thank you.
Other Man said: Anyway, I did.
Man said: You most certainly did not!
Traceback (most recent call last):
sly
File "<pyshell#10>", line 2, in <module> Whoops! There’s something seriou
(role, line_spoken) = each_line.split(':') wrong here.
ValueError: too many values to unpack

It’s a ValueError, so
that must mean there’s
something wrong with the
data in your file, right?

78   Chapter 3
 files and exceptions

Know your data

Your code worked fine for a while, then crashed with a runtime error. The
problem occurred right after the line of data that had the Man saying, “You
most certainly did not!”
Let’s look at the data file and see what comes after this successfully processed
line:

M a n : You didn't!
Other Man: I'm telling you, I did!
Man: You did not!
Other Man: Oh I'm sorry, is this a five minute argument, or the full half hour?
Man: Ah! (taking out his wallet and paying) Just the five minutes.
Other Man: Just the five minutes. Thank you.
Other Man: Anyway, I did.
The error occurs AFTER this line
Man: You most certainly did not! of data.
Other Man: Now let's get one thing quite clear: I most definitely told you!
Man: Oh no you didn't!
Other Man: Oh yes I did!

Notice anything?
Notice anything about the next line of data?
The next line of data has two colons, not one. This is enough extra data
to upset the split() method due to the fact that, as your code currently
stands, split()expects to break the line into two parts, assigning each to
role and line_spoken, respectively.
When an extra colon appears in the data, the split() method breaks the Do this!
line into three parts. Your code hasn’t told split() what to do with the third
part, so the Python interpreter raises a ValueError, complains that you
have “too many values,” and terminates. A runtime error has occurred.

To help diagnose this

problem, let’s put your
code into its own file called
sketch.py. You can copy
What approach might you take to solve this data- and paste your code from
processing problem? the IDLE shell into a new
IDLE edit window.

you are here 4   79

ask for help

Know your methods and ask for help

It might be useful to see if the split() method includes any functionality
that might help here. You can ask the IDLE shell to tell you more about the
split() method by using the help() BIF.

>>> help(each_line.split) Looks like “split()” takes

Help on built-in function split: an optional argument.
split(beans)
split(...)
S.split([sep[, maxsplit]]) -> list of strings

Return a list of the words in S, using sep as the

delimiter string. If maxsplit is given, at most maxsplit
splits are done. If sep is not specified or is None, any
whitespace string is a separator and empty strings are
removed from the result.

The optional argument to split() controls how many breaks occur within
your line of data. By default, the data is broken into as many parts as is
possible. But you need only two parts: the name of the character and the line
he spoke.
If you set this optional argument to 1, your line of data is only ever broken
into two pieces, effectively negating the effect of any extra colon on any line.

Let’s try this and see what happens. split(beans, 1)

Geek Bits
IDLE gives you searchable access to the entire Python
documentation set via its Help ➝ Python Docs menu option (which
will open the docs in your web browser). If all you need to see is the
documentation associated with a single method or function, use
the help() BIF within IDLE’s shell.

80   Chapter 3
 files and exceptions

Here’s the code in the IDLE edit window. Note the extra argument to the split() method.

The extra argument

controls how “split()”
splits.

With the edit applied and saved, press F5 (or select Run Module from IDLE’s Run menu) to try out this version of
your code:

>>> ================================ RESTART ================================

>>>
Man said: Is this the right room for an argument?
Other Man said: I've told you once.
Man said: No you haven't!
Other Man said: Yes I have.
The displayed output is
Man said: When? abridged to allow the
Other Man said: Just now. important stuff to fit on
... this page.
Other Man said: Anyway, I did.
Man said: You most certainly did not!
Other Man said: Now let's get one thing quite clear: I most definitely told you!
Man said: Oh no you didn't!
Cool. You made it past the
Other Man said: Oh yes I did!
line with two colons…
Man said: Oh no you didn't!
ere’s
Other Man said: Oh yes I did! …but your joy is short lived. Th
Man said: Oh look, this isn't an argument! ANOTHER ValueError!!
Traceback (most recent call last):
File "/Users/barryp/HeadFirstPython/chapter4/sketch.py", line 5, in <module>
(role, line_spoken) = each_line.split(':', 1)
ValueError: need more than 1 value to unpack

That’s enough to ruin your day. What could be wrong now?

you are here 4   81

missing colon

Know your data (better)

Your code has raised another ValueError, but this time, instead of
complaining that there are “too many values,” the Python interpreter is
complaining that it doesn’t have enough data to work with: “need more than
1 value to unpack.” Hopefully, another quick look at the data will clear up the
mystery of the missing data.

Other Man: Now let's get one thing quite clear: I most definitely told you!
Man: Oh no you didn't!
Other Man: Oh yes I did!
Man: Oh no you didn't!
Other Man: Oh yes I did!
Man: Oh look, this isn't an argument!
(pause)
Other Man: Yes it is! What’s this?!? Some of the ta
Man: No it isn’t! the expected format…which da doesn’t conform to
can’t be good.
(pause)
Man: It's just contradiction!
Other Man: No it isn't!

The case of the missing colon

Some of the lines of data contain no colon, which causes a problem when
the split() method goes looking for it. The lack of a colon prevents
split() from doing its job, causes the runtime error, which then results in
the complaint that the interpreter needs “more than 1 value.”

It looks like you still have

problems with the data in
your file. What a shame it’s
not in a standard format.

82   Chapter 3
 files and exceptions

Two very different approaches

Or you could decide to let the

When you have to deal with a errors occur, then simply handle
bunch of exceptional situations, the each error if and when it happens.
best approach is to add extra logic. That would be exceptional.
If there’s more stuff to worry
about, you need more code.

Joe
Jill

Jill’s suggested approach certainly works: add the extra logic required to work out
whether it’s worth invoking split() on the line of data. All you need to do is
work out how to check the line of data.
Joe’s approach works, too: let the error occur, spot that it has happened, and then
recover from the runtime error…somehow.

Which approach works best here?

you are here 4   83

find the substring

Add extra logic

Let’s try each approach, then decide which works best here.
In addition to the split() method, every Python string has the find()
method, too. You can ask find() to try and locate a substring in another
string, and if it can’t be found, the find() method returns the value -1. If
the method locates the substring, find() returns the index position of the
substring in the string.

Assign a string to the each_line variable that does not contain a colon, and then use the find() method to
try and locate a colon:

>>> each_line = "I tell you, there's no such thing as a flying circus."
>>> each_line.find(':')
-1 The string does NOT contain a colon, so “find()” returns -1 for NOT FOUND.
Press Alt-P twice to recall the line of code that assigns the string to the variable, but this time edit the string to
include a colon, then use the find() method to try to locate the colon:

>>> each_line = "I tell you: there's no such thing as a flying circus."
>>> each_line.find(':')
10
The string DOES contain a colon, so “find()” returns a positive index value.

And you thought this approach wouldn’t

work? Based on this IDLE session, I
think this could do the trick.

84   Chapter 3
 files and exceptions

Adjust your code to use the extra logic technique demonstrated on the previous page to deal
with lines that don’t contain a colon character.

data = open('sketch.txt') What condition needs to go here?

for each_line in data:

if
(role, line_spoken) = each_line.split(':', 1)
print(role, end='')
print(' said: ', end='')
print(line_spoken, end='')

data.close()

Can you think of any potential problems with this technique?

Grab your pencil and write down any issues you might have with
this approach in the space provided below:

you are here 4   85

substring found

You were to adjust your code to use the extra logic technique to deal with lines that don’t contain
a colon character:

It takes a few seconds to get

head around this condition, butyour
data = open('sketch.txt')

does work. it
for each_line in data:
if not each_line.find(':') == -1:
the (role, line_spoken) = each_line.split(':', 1)
Note the use of hich
“not” keyword, luwe of
print(role, end='')

negates the va print(' said: ', end='')

the condition. print(line_spoken, end='')

data.close()

You were to think of any potential problems with this technique,

grabbing your pencil to write down any issues you might have
with this approach.
issues
It’s OK if your. Just so
are different e similar
long as they ar
to these.

There might be a problem with this code if the format of the data file
changes, which will require changes to the condition.

The condition used by the if statement is somewhat hard to read and

understand.

This code is a little “fragile”…it will break if another exceptional

situation arises.

86   Chapter 3
 files and exceptions

Test Drive
Amend your code within IDLE’s edit window, and press F5 to see if it works.

No errors
this time.

Your program works…although it is fragile. Maybe it’s time for a

If the format of the file changes, your code will different approach? One
need to change, too, and more code generally means that doesn’t require extra
more complexity. Adding extra logic to handle logic, eh?
exceptional situations works, but it might cost you
in the long run.

you are here 4   87

exceptional catch

Handle exceptions
Have you noticed that when something goes wrong with your code, the
Python interpreter displays a traceback followed by an error message?
The traceback is Python’s way of telling you that something unexpected has
occurred during runtime. In the Python world, runtime errors are called
exceptions.

Whoooah! I don’t
know what to do with this
error, so I’m gonna raise
an exception...this really is
someone else’s problem.

>>> if not each_

Traceback (most r
File “<pyshell
It looks
Oooh, yuck’s! a bug. (role, line_
like there ValueError: too m

Of course, if you decide to ignore an exception when it occurs, your program

crashes and burns.
But here’s the skinny: Python let’s you catch exceptions as they occur, which
gives you with a chance to possibly recover from the error and, critically, not
crash.
By controlling the runtime behavior of your program, you can ensure (as
much as possible) that your Python programs are robust in the face of most
runtime errors.

Try the code first. Then deal with errors as they happen.

88   Chapter 3
 files and exceptions

Try first, then recover

Rather than adding extra code and logic to guard against bad things
happening, Python’s exception handling mechanism lets the error occur,
spots that it has happened, and then gives you an opportunity to recover.
During the normal flow of control, Python tries your code and, if nothing goes
wrong, your code continues as normal. During the exceptional flow of control,
Python tries your code only to have something go wrong, your recovery code
executes, and then your code continues as normal.

Exceptional flow
Python tries
Normal flow your code, but
fails!

Python tries
your code.
Crash!
Your recovery
code executes.
Your exception
It’s all OK, so is handled.
keep going…

The try/except mechanism Then you keep

going…
Python includes the try statement, which exists to provide you with a way to
systematically handle exceptions and errors at runtime. The general form of
the try statement looks like this:

try:
Both “try” your code (which might cause a runtime erro
and “except” r)
are Python
keywords.
except:
your error-recovery code

you are here 4   89

allow errors

Hang on, here! Are you

actually letting errors
occur on purpose? ARE
YOU MAD?!?

No. Not mad. And, yes. Letting errors occur.

If you try to code for every possible error, you’ll be at it for a
long time, because all that extra logic takes a while to work out.
Paradoxically, when you worry less about covering every
possible error condition, your coding task actually gets easier.

90   Chapter 3
 files and exceptions

Identify the code to protect

In order to plug into the Python exception handling mechanism, take a
moment to identify the code that you need to protect.

Study your program and circle the line or lines of code that you
think you need to protect. Then, in the space provided, state why.

data = open('sketch.txt')

for each_line in data:

(role, line_spoken) = each_line.split(':', 1)
print(role, end='')
print(' said: ', end='')
print(line_spoken, end='')

data.close()
State your reason
why here.

Q: Something has been bugging me for a while. When the split() method executes, it passes back a list, but the target identifiers
are enclosed in regular brackets, not square brackets, so how is this a list?

A: Well spotted. It turns out that there are two types of list in Python: those that can change (enclosed in square brackets) and those that
cannot be changed once they have been created (enclosed in regular brackets). The latter is an immutable list, more commonly referred
to as a tuple. Think of tuples as the same as a list, except for one thing: once created, the data they hold cannot be changed under any
circumstances. Another way to think about tuples is to consider them to be a constant list. At Head First, we pronounce “tuple” to rhyme with
“couple.” Others pronounce “tuple” to rhyme with “rupal.” There is no clear concensus as to which is correct, so pick one and stick to it.

you are here 4   91

code to protect

You were to study your program and circle the line or lines of
code that you think you need to protect. Then, in the space
provided, you were to state why.

data = open('sketch.txt')

for each_line in data:

(role, line_spoken) = each_line.split(':', 1)

These four
print(role, end='')
print(' said: ', end='')
lines of code
print(line_spoken, end='') all need to be
protected.
data.close()

If the call to “split()” fails, you don’t want the three “print()”
statements executing, so it’s best to protect all four lines of the “if”
suite, not just the line of code that calls “split()”.

OK. I get that the code Yeah...good point. It’s

can be protected from an probably best to ignore
error. But what do I do when it, right? I wonder how...
an error actually occurs?

92   Chapter 3
 files and exceptions

Take a pass on the error

With this data (and this program), it is best if you ignore lines that don’t
conform to the expected format. If the call to the split() method causes
an exception, let’s simply pass on reporting it as an error.
When you have a situation where you might be expected to provide code, but
don’t need to, use Python’s pass statement (which you can think of as the
empty or null statement.)
Here’s the pass statement combined with try:

data = open('sketch.txt')

for each_line in data:

try:

This code is (role, line_spoken) = each_line.split(':', 1)

protected from print(role, end='')
runtime errors. print(' said: ', end='')
print(line_spoken, end='')
except:
If a runtime error
pass occurs, this code is
executed.
data.close()

Now, no matter what happens when the split() method is invoked, the
try statement catches any and all exceptions and handles them by ignoring the
error with pass.

Let’s see this code in action. Do this!

Make the required changes

to your code in the IDLE
edit window.

you are here 4   93

idle session

With your code in the IDLE edit window, press F5 to run it.

>>> ================================ RESTART ================================

>>>
Man said: Is this the right room for an argument?
Other Man said: I’ve told you once.
Man said: No you haven’t!
Other Man said: Yes I have. This code works, and there are
Man said: When? no runtime errors, either.
...
OOther Man said: Nonsense!
Man said: (exasperated) Oh, this is futile!!
Other Man said: No it isn’t!
Man said: Yes it is!

So...both approaches work.

But which is better?

94   Chapter 3
 files and exceptions

Tonight’s talk: Approaching runtime errors with extra code

and exception handlers

Extra Code: Exception Handler:

By making sure runtime errors never happen, I keep
my code safe from tracebacks.
At the cost of added complexity….
Complexity never hurt anyone.
I’ll be sure to remind you of that the next time
you’re debugging a complex piece of code at 4
o’clock in the morning.
I just don’t get it. You’re more than happy for your
code to explode in your face…then you decide it’s
probably a good idea to put out the fire?!?
Yes. I concentrate on getting my work done first and
foremost. If bad things happen, I’m ready for them.
But the bad things still happen to you. They never
happen with me, because I don’t let them.
Until something else happens that you weren’t
expecting. Then you’re toast.
Well…that depends. If you’re smart enough—and,
believe me, I am—you can think up all the possible
runtime problems and code around them.
Sounds like a whole heap of extra work to me.
Hard work never hurt anyone.
You did hear me earlier about debugging at 4 AM,
right? Sometimes I think you actually enjoy writing
code that you don’t need…
Of course all my code is needed! How else can you
code around all the runtime errors that are going to
happen?
Yeah…how many?
Um, uh…most of them, I guess.
You don’t know, do you? You’ve no idea what will
happen when an unknown or unexpected runtime
error occurs, do you?
Look: just cut it out. OK?

you are here 4   95

more errors

What about other errors?

It is true that both approaches work, but let’s consider what happens when
other errors surface.

Ummm...I’m not totally

OK, guys. How does you sure, to be honest...
code react when the
data file is deleted? I guess that’ll produce
another exception, but
what happens in my code
is anyone’s guess.

Handling missing files

Frank’s posed an interesting question and, sure
enough, the problem caused by the removal of
the data file makes life more complex for Jill and
Joe. When the data file is missing, both versions of Rename the data file, then
the program crash with an IOError. run both versions of your
program again to confirm
Do this! that they do indeed raise an
IOError and generate a
traceback.

96   Chapter 3
 files and exceptions

Add more error-checking code…

If you’re a fan of the “let’s not let errors happen” school of thinking, your
first reaction will be to add extra code to check to see if the data file exists
before you try to open it, right?
Let’s implement this strategy. Python’ s os module has some facilities that can
help determine whether a data file exists, so we need to import it from the
Standard Library, then add the required check to the code:

import os
Check whether
the file exists.
if os.path.exists('sketch.txt'):
data = open('sketch.txt')

for each_line in data:

if not each_line.find(':') == -1:
All of this (role, line_spoken) = each_line.split(':', 1)
code remains print(role, end='')
unchanged.
print(' said: ', end='')
print(line_spoken, end='')

data.close()
Inform the else:
user of the
bad news. print('The data file is missing!')

A quick test of the code confirms that this new problem is dealt with properly. With this new version of your code
in IDLE’s edit window, press F5 to confirm all is OK.

>>> ================================ RESTART ================================

>>>
The data file is missing! Exactly what was expected. Cool.
>>>

you are here 4   97

take it to another level

…Or add another level of exception handling

If you are a fan of the “handle exceptions as they occur” school of thinking, you’ll
simply wrap your code within another try statement.

Add another
“try” statement. try:
data = open('sketch.txt')

for each_line in data:

try:
As with the (role, line_spoken) = each_line.split(':', 1)
other program,
all of this print(role, end='')
code remains print(' said: ', end='')
unchanged.
print(line_spoken, end='')
except:
pass

data.close()
Give the user except:
the bad news. print('The data file is missing!')

Another quick test is required, this time with the version of your program that uses exception handling. Press F5 to
give it a spin.

>>> ================================ RESTART ================================

>>>
As expected, this version of th e
too.
program handles the missing file,
The data file is missing!
>>>

98   Chapter 3
 files and exceptions

So, which approach is best?

Well…it depends on who you ask! Here are both versions of your code:

This version uses extra logic

to handle File I/O errors.

This version uses another “try”

statement to handle File I/O err
ors.

Let’s ask a simple question about these two versions of your program: What do each
of these programs do?

Grab your pencil. In box 1, write down what you think the
program on the left does. In box 2, write down what you think
the program on the right does.

you are here 4   99

keep it simple

You were to grab your pencil, then in box 1, write down what you
thought the program on the left does. In box 2, write down what
ually need you thought the program on the right does.
There’s a lot to write, so youionactthan was
more space for your descripte.
provided on the previous pag
1 The code on the right starts by importing the “os” library, and then it uses “path.exists” to
make sure the data file exists, before it attempts to open the data file. Each line from
the file is then processed, but only after it has determined that the line conforms to the
required format by checking first for a single “:” character in the line. If the “:” is found,
the line is processed; otherwise, it’s ignored. When we’re all done, the data file is closed. And
you get a friendly message at the end if the file is not found.

Now…that’s more like it.

2
The code on the right opens a data file, processes each line in that file, extracts the data of
interest and displays it on screen. The file is closed when done. If any exceptions occur, this
code handles them.

Complexity is rarely a good thing

Do you see what’s happening here?
As the list of errors that you have to worry about grows, the complexity of
the “add extra code and logic” solution increases to the point where it starts
to obscure the actual purpose of the program.
This is not the case with the exceptions handling solution, in which it’s obvious
what the main purpose of the program is.
By using Python’s exception-handling mechanism, you get to concentrate on
what your code needs to do, as opposed to worrying about what can go wrong
and writing extra code to avoid runtime errors.
Prudent use of the try statement leads to code that is easier to read, easier
to write, and—perhaps most important—easier to fix when something goes
wrong.
Concentrate on what your code needs to do.

100   Chapter 3
 files and exceptions

You’re done…except for one small thing

Your exception-handling code is good. In fact, your code might be too good in
that it is too general.
At the moment, no matter what error occurs at runtime, it is handled by your
code because it’s ignored or a error message is displayed. But you really need to
worry only about IOErrors and ValueErrors, because those are the types
of exceptions that occurred earlier when your were developing your program.
Although it is great to be able to handle all runtime errors, it’s probably
unwise to be too generic…you will want to know if something other than
an IOError or ValueError occurs as a result of your code executing at
runtime. If something else does happen, your code might be handling it in an
inappropriate way.

try:
data = open('sketch.txt')

for each_line in data:

try:
(role, line_spoken) = each_line.split(':', 1)
print(role, end='')
print(' said: ', end='')
print(line_spoken, end='')
except:
pass
an d t hi s co d e runs when ANYthat
This code oc curs within the
code
ru nt im e er ro r
is being tried.
data.close()
except:
print('The data file is missing!')

As your code is currently written, it is too generic. Any runtime error that
occurs is handled by one of the except suites. This is unlikely to be what
you want, because this code has the potential to silently ignore runtime errors.

You need to somehow use except in a less generic way.

you are here 4   101

specify exceptions

Be specific with your exceptions

If your exception-handling code is designed to deal with a specific type of
error, be sure to specify the error type on the except line. In doing so, you’ll
take your exception handling code from generic to specific.

try:
data = open('sketch.txt')

for each_line in data:

try:
(role, line_spoken) = each_line.split(':', 1)
print(role, end='')
print(' said: ', end='')
print(line_spoken, end='')
except ValueError:
pass
Specify the type
error you are ha of runtime
data.close() ndling.
except IOError:
print('The data file is missing!')

Of course, if an different type of runtime error occurs, it is no longer handled

by your code, but at least now you’ll get to hear about it. When you are
specific about the runtime errors your code handles, your programs no longer
silently ignore some runtime errors.

...and it lets you avoid

adding unnecessary code
Using “try/except”
and logic to your programs.
lets you concentrate
That works for me!
on what your code
needs to do...

102   Chapter 3
 files and exceptions

Your Python Toolbox

You’ve got Chapter 3 under your

CHAPTER 3
belt and you’ve added some key
Python techiques to your toolbox. ƒƒ Use the open() BIF to open a disk file,
creating an iterator that reads data from
the file one line at a time.
ƒƒ The readline() method reads a

go
single line from an opened file.

P y th o n L i n ƒƒ The seek() method can be used to

s a result
n” occurs a
“rewind” a file to the beginning.
• An “exceptio ror, producing a ƒƒ The close() method closes a
e er
of a runtim previously opened file.
traceback. d
is a detaile error
ƒƒ The split() method can break a
r a ce b a c k ”
• A “t the runtim
e string into a list of parts.
rip t io n o f
desc curred.
ƒƒ An unchangeable, constant list in Python
that has oc is called a tuple. Once list data is
assigned to a tuple, it cannot be changed.

IDLE Notes
Tuples are immutable.
ƒƒ A ValueError occurs when your data
does not conform to an expected format.
• Access Python
by choosing Pyth’s documentation ƒƒ An IOError occurs when your data
IDLE’s Help men on Docs from cannot be accessed properly (e.g.,
documentation seu. The Python 3 perhaps your data file has been moved or
your favorite webt should open in renamed).
browser. ƒƒ The help() BIF provides access to
Python’s documentation within the IDLE
shell.
ƒƒ The find() method locates a specific
substring within another string.
ƒƒ The not keyword negates a condition.
ƒƒ The try/except statement provides
an exception-handling mechanism,
allowing you to protect lines of code that
might result in a runtime error.
ƒƒ The pass statement is Python’s empty
or null statement; it does nothing.

you are here 4   103

4 persistence

Saving data to files

I’m in a bit of a pickle...my

data is not as persistent as
it could be.

It is truly great to be able to process your file-based data.

But what happens to your data when you’re done? Of course, it’s best to save your
data to a disk file, which allows you to use it again at some later date and time. Taking
your memory-based data and storing it to disk is what persistence is all about. Python
supports all the usual tools for writing to files and also provides some cool facilities for
efficiently storing Python data. So…flip the page and let’s get started learning them.

this is a new chapter   105

save your work

Programs produce data

It’s a rare program that reads data from a disk file, processes the data, and
then throws away the processed data. Typically, programs save the data they
process, display their output on screen, or transfer data over a network.
You have lots of choices
about which type of disk
file to use.

Your data is ready.

Where should I put it
for you?
DBMS

Hey, that’s nice

output...exactly
what I want.

Before you learn what’s involved in writing data to disk, let’s process the data
from the previous chapter to work out who said what to whom.

When that’s done, you’ll have something worth saving.

106   Chapter 4
 persistence

Code Magnets
Add the code magnets at the bottom of this page to your existing
code to satisfy the following requirements:

1. Create an empty list called man.

2. Create an empty list called other.
3. Add a line of code to remove unwanted whitespace from the
line_spoken variable.
4. Provide the conditions and code to add line_spoken to the
correct list based on the value of role.
5. Print each of the lists (man and other) to the screen.

try:
data = open('sketch.txt')
for each_line in data:
try:
(role, line_spoken) = each_line.split(':', 1)

except ValueError:
pass
data.close()

Here are your magnets.

except IOError:
print('The datafile is missing!')

elif role
== 'Other
Man':
man = []
if role == 'Man':
other.app
end(line_
line_spoken = line_spoken.strip() other = [] spoken)

man.appen print(man)
d(line_sp
print(other) oken)

you are here 4   107

process and print

Code Magnets Solution

Your were to add the code magnets to your existing code to satisfy
the following requirements:

1. Create an empty list called man.

2. Create an empty list called other.
3. Add a line of code to remove unwanted whitespace from the
line_spoken variable.
4. Provide the conditions and code to add line_spoken to the
correct list based on the value of role.
5. Print each of the lists (man and other) to the screen.

Assign an empty list to

man = [] “man” and “other”.
other = []

try:
data = open('sketch.txt')
for each_line in data:
try:

Assign the (role, line_spoken) = each_line.split(':', 1)

stripped string
back onto itself.
line_spoken = line_spoken.strip() The “strip()” method
removes unwanted
whitespace from a string.
if role == 'Man':

man.append(line_spoken)

elif role == 'Other Man':

Update one of the lists
“elif” means based on who said what.
“else if.” other.append(line_spoken)

except ValueError:
pass
data.close()
except IOError:
print('The datafile is missing!')

print(man)
Conclude by displaying the
print(other) processed data on screen.

108   Chapter 4
 persistence

Test Drive
Load your code into IDLE’s edit window and take it for a spin by pressing F5. Be sure to save your program into the
same folder that contains sketch.txt.

The code in IDLE’s

edit window

And here’s what appears

on screen: the contents of
the two lists.

It worked, as expected.

Surely Python’s open() BIF

can open files for writing as well
as reading, eh?

Yes, it can.

When you need to save data to a file, the

open() BIF is all you need.

you are here 4   109

open and close

Open your file in write mode

When you use the open() BIF to work with a disk file, you can specify an
access mode to use. By default, open() uses mode r for reading, so you don’t
need to specify it. To open a file for writing, use mode w: The access model
to use

out = open("data.out", "w")

The data file
object
The name of the file to
write to
By default, the print() BIF uses standard output (usually the screen) when
displaying data. To write data to a file instead, use the file argument to
specify the data file object to use:

print("Norwegian Blues stun easily.", file=out)

What gets written to the file

The name of the data
file object to write to
When you’re done, be sure to close the file to ensure all of your data is written
to disk. This is known as flushing and is very important:

This is VERY important

out.close() when writing to files.

Geek Bits

When you use access mode w, Python opens your named file
for writing. If the file already exists, it is cleared of its contents, or
clobbered. To append to a file, use access mode a, and to open a
file for writing and reading (without clobbering), use w+. If you
try to open a file for writing that does not already exist, it is first
created for you, and then opened for writing.

110   Chapter 4
 persistence

At the bottom of your program, two calls to the print() BIF

display your processed data on screen. Let’s amend this code to
save the data to two disk files instead.

Call your disk files man_data.txt (for what the man said) and
other_data.txt (for what the other man said). Be sure to
both open and close your data files, as well as protect your code
man = [] against an IOError using try/except.
other = []

try:
data = open('sketch.txt')
for each_line in data:
try:
(role, line_spoken) = each_line.split(':', 1)
line_spoken = line_spoken.strip()
if role == 'Man':
man.append(line_spoken)
elif role == 'Other Man':
other.append(line_spoken)
except ValueError:
pass
data.close()
except IOError:
print('The datafile is missing!')

Go on, try. Open your two

data files here.

Specify the files to

print(man, )

write to when you invoke

print(other, )
Be sure to “print()”.
close your
files.
Handle any
exceptions here.

you are here 4   111

save to disk

At the bottom of your program, two calls to the print() BIF

display your processed data on screen. You were to amend this
code to save the data to two disk files instead.

You were to call your disk files man_data.txt (for what the
man said) and other_data.txt (for what the other man said).
You were to make sure to both open and close your data files, as
man = [] well as protect your code against an IOError using try/except.

other = []

try:
data = open('sketch.txt')
for each_line in data:
try:
(role, line_spoken) = each_line.split(':', 1)
line_spoken = line_spoken.strip()
if role == 'Man': All of this code is
man.append(line_spoken) unchanged.
elif role == 'Other Man':
other.append(line_spoken)
except ValueError:
pass
data.close()
except IOError:

Did you remember to open

print('The datafile is missing!')

your files in WRITE mode?

try:
man_file = open(‘man_data.txt’, ‘w’) Open your two files, and assign
each to file objects.
other_file = open(‘other_data.txt’, ‘w’)

print(man, file=man_file ) Use the “print()” BIF to save th

named lists to named disk files. e
print(other, file=other_file )

man_file.close() Don’t forget to close BOTH files.

other_file.close()
except IOError: Handle an I/O exception, should
one occur.
print('File error.’)

112   Chapter 4
 persistence

Test Drive
Perform the edits to your code to replace your two print() calls with your new file I/O code. Then, run your
program to confirm that the data files are created:

Before your program runs,

there are no data files in
your folder, just your code.

When you run

your code, this
is all you see,
what looks like
an “empty” IDLE
shell.

After your program

runs, two new files are
CREATED in your folder.

That code worked, too. You’ve created two data files, each holding the data
from each of your lists. Go ahead and open these files in your favorite editor
to confirm that they contain the data you expect.

Consider the following carefully: what happens to

your data files if the second call to print() in
your code causes an IOError?

you are here 4   113

close files

Files are left open after an exception!

When all you ever do is read data from files, getting an IOError is annoying,
but rarely dangerous, because your data is still in your file, even though you
might be having trouble getting at it.
It’s a different story when writing data to files: if you need to handle an
IOError before a file is closed, your written data might become corrupted
and there’s no way of telling until after it has happened.

try: Ok.
man_file = open('man_data.txt', 'w') OK
other_file = open('other_data.txt', 'w') OK

print(man, file=man_file) OK
print(other, file=other_file) Not OK!!

Crash!

man_file.close() These two lines of code

other_file.close()
DON’T get to run.
except IOError: OK
print('File error.') OK

Your exception-handling code is doing its job, but you now have a situation
where your data could potentially be corrupted, which can’t be good.
What’s needed here is something that lets you run some code regardless of
whether an IOError has occured. In the context of your code, you’ll want
to make sure the files are closed no matter what.

114   Chapter 4
 persistence

Extend try with finally

When you have a situation where code must always run no matter what errors
occur, add that code to your try statement’s finally suite:

try:
man_file = open('man_data.txt', 'w')
other_file = open('other_data.txt', 'w')
No changes
here, except
that… print(man, file=man_file)
print(other, file=other_file)

except IOError:
print('File error.')

…the calls to finally:

“close()” are
moved to here. man_file.close()
other_file.close()

If no runtime errors occur, any code in the finally suite executes. Equally,
if an IOError occurs, the except suite executes and then the finally
suite runs.
No matter what, the code in the finally suite always runs.
By moving your file closing code into your finally suite, you are reducing
the possibility of data corruption errors.
This is a big improvement, because you’re now ensuring that files are closed
properly (even when write errors occur).
But what about those errors?

How do you find out the specifics of the error?

you are here 4   115

no dumb questiohns

Q: I’m intrigued. When you stripped the line_spoken data Q: But surely Python can work out how many variables are
of unwanted whitespace, you assigned the result back to the referring to any one particular string?

A:
line_spoken variable. Surely invoking the strip() method on
line_spoken changed the string it refers to? It does, but only for the purposes of garbage collection. If you

A: No, that’s not what happens. Strings in Python are immutable,

which means that once a string is created, it cannot be changed.
have a line of code like print('Flying Circus'), the
string is not referred to by a variable (so any variable reference
counting that’s going on isn’t going to count it) but is still a valid string

Q: But you did change the line_spoken string by removing

object (which might be referred to by a variable) and it cannot have
its data changed under any circumstances.
any unwanted whitespace, right?
Q:
A:
So Python variables don’t actually contain the data
Yes and no. What actually happens is that invoking the assigned to them?
strip() method on the line_spoken string creates a
new string with leading and trailing whitespace removed. The new
string is then assigned to line_spoken, replacing the data that
A: That’s correct. Python variables contain a reference to a
data object.The data object contains the data and, because you
was referred to before. In effect, it is as if you changed line_ can conceivably have a string object used in many different places
spoken, when you’ve actually completely replaced the data it throughout your code, it is safest to make all strings immutable so
refers to. that no nasty side effects occur.

Q: So what happens to the replaced data?

Q: Isn’t it a huge pain not being able to adjust strings “in
place”?
A: Python’s built-in memory management technology reclaims the
RAM it was using and makes it available to your program. That is, A: No, not really. Once you get used to how strings work, it
unless some other Python data object is also referring to the string. becomes less of an issue. In practice, you’ll find that this issue rarely
trips you up.
Q: What? I don’t get it.
Q: Are any other Python data types immutable?
A: It is conceivable that another data object is referring to the
string referred to by line_spoken. For example, let’s assume A: Yes, a few. There’s the tuple, which is an immutable list. Also,
you have some code that contains two variables that refer to the all of the number types are immutable.
same string, namely “Flying Circus.” You then decide that one of
the variables needs to be in all UPPERCASE, so you invoke the Q: Other than learning which is which, how will I know when
upper() method on it. The Python interperter takes a copy of the something is immutable?

A:
string, converts it to uppercase, and returns it to you. You can then
assign the uppercase data back to the variable that used to refer to Don’t worry: you’ll know. If you try to change an immutable
the original data. value, Python raises a TypeError exception.

Q: And the original data cannot change, because there’s Q: Of course: an exception occurs. They’re everywhere in
another variable referring to it? Python, aren’t they?

A: Precisely. That’s why strings are immutable, because you never

know what other variables are referring to any particular string.
A: Yes. Exceptions make the world go ’round.

116   Chapter 4
 persistence

Knowing the type of error is not enough

When a file I/O error occurs, your code displays a generic “File Error”
message. This is too generic. How do you know what actually happened?

It could be that the Yeah, or it could be

file can be opened but a permission error, or
Maybe the problem maybe your disk is full?
not written to?
is that you can’t open
the file?

Who knows?

It turns out that the Python interpreter knows…and it will give up the details
if only you’d ask.
When an error occurs at runtime, Python raises an exception of the specific
type (such as IOError, ValueError, and so on). Additionally, Python
creates an exception object that is passed as an argument to your except
suite.

Let’s use IDLE to see how this works.

you are here 4   117

idle session

Let’s see what happens when you try to open a file that doesn’t exist, such as a disk file called missing.txt.
Enter the following code at IDLE’s shell:

>>> try:
data = open('missing.txt')
print(data.readline(), end='')
except IOError:
print('File error')
finally:
data.close()

File error
There’s your error message, but…
Traceback (most recent call last):
File "<pyshell#8>", line 7, in <module> …what’s this?!? Another except
data.close() raised and it killed your code. ion was
NameError: name 'data' is not defined

As the file doesn’t exist, the data file object wasn’t created, which subsequently makes it impossible to call the
close() method on it, so you end up with a NameError. A quick fix is to add a small test to the finally
suite to see if the data name exists before you try to call close(). The locals() BIF returns a collection of
names defined in the current scope. Let’s exploit this BIF to only invoke close() when it is safe to do so:

The “in” operator tests This is just the bit of code that
finally: for membership. needs to change. Press Alt-P
if 'data' in locals(): edit your code at IDLE’s shell.to
data.close()

File error
No extra exceptions this time.
Just your error message.

Here you’re searching the collection returned by the locals() BIF for the string data. If you find it, you can
assume the file was opened successfully and safely call the close() method.

If some other error occurs (perhaps something awful happens when your code calls the print() BIF), your
exception-handling code catches the error, displays your “File error” message and, finally, closes any opened file.
But you still are none the wiser as to what actually caused the error.

118   Chapter 4
 persistence

When an exception is raised and handled by your except suite, the Python interpreter passes an exception object
into the suite. A small change makes this exception object available to your code as an identifier:

Give your exception object

except IOError as err: a name…
print('File error: ' + err)
…then use it as part of

your error message.

But when you try to run your code with this change made, another exception is raised:

Traceback (most recent call last):

File "<pyshell#18>", line 5, in <module> Whoops! Yet another
print('File error:' + err) exception; this time it’s a
TypeError: Can't convert 'IOError' object to str implicitly “TypeError”.

This time your error message didn’t appear at all. It turns out exception objects and strings are not compatible
types, so trying to concatenate one with the other leads to problems. You can convert (or cast) one to the other
using the str() BIF:

Use the “str()” BIF to for th

except IOError as err: exception object to behave ce e
like a string.
print('File error: ' + str(err))

And you now get a specific erry or

Now, with this final change, your code is behaving exactly as expected:

message that tells you exactl

File error: [Errno 2] No such file or directory: 'missing.txt'
what went wrong.

Of course, all this extra logic is starting to obscure the

real meaning of your code.
Wouldn’t it be dreamy if there
were a way to take advantage of
these mechanisms without the code
bloat? I guess it’s just a fantasy...

you are here 4   119

try with

Use with to work with files

Because the use of the try/except/finally pattern is so common when
it comes to working with files, Python includes a statement that abstracts
away some of the details. The with statement, when used with files, can
dramatically reduce the amount of code you have to write, because it negates
the need to include a finally suite to handle the closing of a potentially
opened data file. Take a look:

try: This is the usual “try/

except/finally” pattern.
data = open('its.txt', "w")
print("It's...", file=data)
except IOError as err:
print('File error: ' + str(err))
finally:
if 'data' in locals():
data.close()

try:
with open('its.txt', "w") as data:
print("It's...", file=data)
except IOError as err:
The use of “with” print('File error: ' + str(err))
negates the need for
the “finally” suite.
When you use with, you no longer have to worry about closing any opened
files, as the Python interpreter automatically takes care of this for you. The
with code on the the right is identical in function to that on the left. At Head
First Labs, we know which approach we prefer.

Geek Bits

The with statement takes advantage of a Python technology

called the context management protocol.

120   Chapter 4
 persistence

Grab your pencil and rewrite this try/except/finally code to use

with instead. Here’s your code with the appropriate finally
suite added:

try:
man_file = open('man_data.txt', 'w')
other_file = open('other_data.txt', 'w')

print(man, file=man_file)
print(other, file=other_file)
except IOError as err:
print('File error: ' + str(err))
finally:
if 'man_file' in locals():
man_file.close()
if 'other_file' in locals():
other_file.close()

Write your
“with” code
here.

you are here 4   121

no finally

You were to grab your pencil and rewrite this try/except/finally

code to use with instead. Here’s your code with the appropriate
finally suite added:

try:
man_file = open('man_data.txt', 'w')
other_file = open('other_data.txt', 'w')

print(man, file=man_file)
print(other, file=other_file)
except IOError as err:
print('File error: ' + str(err))
finally:
if 'man_file' in locals():
man_file.close()
if 'other_file' in locals():
other_file.close()

try:
with open(‘man_data.txt', ‘w') as man_file:
Using two “with”
statements to rewrite print(man, file=man_file)
the code without the with open(‘other_data.txt', ‘w') as other_file:
“finally” suite.
print(other, file=other_file)
except IOError as err:
print(‘File error: ' + str(err))
ls into one
Or combine the two “open()” cal Note the use of the comma.
“with” statement.
with open('man_data.txt', 'w') as man_file, open('other_data.txt’, 'w’) as other_file:
print(man, file=man_file)
print(other, file=other_file)

122   Chapter 4
 persistence

Test Drive
Add your with code to your program, and let’s confirm that it continues to function as expected. Delete the two
data files you created with the previous version of your program and then load your newest code into IDLE and give
it a spin.

IDLE
No errors in theindicate
shell appears toram ran
that the prog
successfully.

If you check your folder, your two data files should’ve reappeared. Let’s take a closer look at the data file’s contents
by opening them in your favorite text editor (or use IDLE).

Here’s what the

man said.

Here’s what the

other man said.

You’ve saved the lists in two files containing what the Man said and what the Other
man said. Your code is smart enough to handle any exceptions that Python or
your operating system might throw at it.

Well done. This is really coming along.

you are here 4   123

unsuitable format

Default formats are unsuitable for files

Although your data is now stored in a file, it’s not really in a useful format.
Let’s experiment in the IDLE shell to see what impact this can have.

Use a with statement to open your data file and display a single line from it:

>>> with open('man_data.txt') as mdf:

Note: no need to close your e,
print(mdf.readline())
because “with” does that forfilyou
.
['Is this the right room for an argument?', "No you haven't!", 'When?', "No you didn't!", "You
didn't!", 'You did not!', 'Ah! (taking out his wallet and paying) Just the five minutes.',
'You most certainly did not!', "Oh no you didn't!", "Oh no you didn't!", "Oh look, this isn't
an argument!", "No it isn't!", "It's just contradiction!", 'It IS!', 'You just contradicted
me!', 'You DID!', 'You did just then!', '(exasperated) Oh, this is futile!!', 'Yes it is!']

Yikes! It would appear your list is converted to a large string by print()

when it is saved. Your experimental code reads a single line of data from the
file and gets all of the data as one large chunk of text…so much for your code
saving your list data.

What are your options for dealing with this problem?

Geek Bits

By default, print() displays your data in a format that mimics

how your list data is actually stored by the Python interpreter.
The resulting output is not really meant to be processed further…
its primary purpose is to show you, the Python programmer,
what your list data “looks like” in memory.

124   Chapter 4
 persistence

I guess I could write some custom

parsing code to process the “internal
It might be worth looking at
format” used by “print()”. It shouldn’t
using something other than a plain
take me all that long...
“print()” to format the data prior
to saving it to the data file? I’d
certainly look into it.

Parsing the data in the file is a possibility…although it’s complicated by all those
square brackets, quotes, and commas. Writing the required code is doable,
but it is a lot of code just to read back in your saved data.
Of course, if the data is in a more easily parseable format, the task would likely be
easier, so maybe the second option is worth considering, too?

Can you think of a function you created from earlier

in this book that might help here?

you are here 4   125

nonstandard output

Why not modify print lol()?

Recall your print_lol() function from Chapter 2, which takes any list (or
list of lists) and displays it on screen, one line at a time. And nested lists can
be indented, if necessary.
This functionality sounds perfect! Here’s your code from the nester.py
module (last seen at the end of Chapter 2):

r
This code currently displays you
data on the screen.

Amending this code to print to a disk file instead of the screen (known as
standard output) should be relatively straightforward. You can then save your
data in a more usable format.

Standard Output The default place where your code writes its
data when the “print()” BIF is used. This is typically the screen.
In Python, standard output is referred to as “sys.stdout” and
is importable from the Standard Library’s “sys” module.

126   Chapter 4
 persistence

1 Let’s add a fourth argument to your print_lol() function to identify a place to write your
data to. Be sure to give your argument a default value of sys.stdout, so that it continues to
write to the screen if no file object is specified when the function is invoked.
Fill in the blanks with the details of your new argument. (Note: to save on space, the comments
have been removed from this cod, but be sure to update your comments in your nester.py
module after you’ve amended your code.)

def print_lol(the_list, indent=False, level=0, ):

for each_item in the_list:

if isinstance(each_item, list):
print_lol(each_item, indent, level+1, )
else:
if indent:
for tab_stop in range(level):
print("\t", end='', )
print(each_item, )

2 What needs to happen to the code in your with statement now that your amended print_lol()
function is available to you?

3 List the name of the module(s) that you now need to import into your program in order to support your
amendments to print_lol().

you are here 4   127

extend your function

1 You were to add a fourth argument to your print_lol() function to identify a place to write
your data to, being sure to give your argument a default value of sys.stdout so that it
continues to write to the screen if no file object is specified when the function is invoked.
You were to fill in the blanks with the details of your new argument. (Note: to save on space, the
comments have been removed from this code, but be sure to update those in your nester.py
module after you’ve amended your code).
e it a
Add the fourth argument and giv
default value.
def print_lol(the_list, indent=False, level=0, fh=sys.stdout ):

for each_item in the_list:

if isinstance(each_item, list): Note: the
print_lol(each_item, indent, level+1, fh ) signature has
changed.
else:
if indent:
for tab_stop in range(level): Adjust the two
print("\t", end='', file=fh ) calls to “print()”
to use the new
print(each_item, file=fh )
argument.

2 What needs to happen to the code in your with statement now that your amended print_lol()
function is available to you?

The code needs to be adjusted so that instead of using the

“print()” BIF, the code needs to invoke “print_lol()” instead.

3 List the name of the module(s) that you now need to import into your program in order to support your
amendments to print_lol().

The program needs to import the amended “nester” module.

128   Chapter 4
 persistence

Test Drive
Before taking your code for a test drive, you need to do the following:
1. Make the necessary changes to nester and install the amended module into your Python
environment (see Chapter 2 for a refresher on this). You might want to upload to PyPI, too.
2. Amend your program so that it imports nester and uses print_lol() instead of print()
within your with statement. Note: your print_lol() invocation should look something like this:
print_lol(man, fh=man_file).
When you are ready, take your latest program for a test drive and let’s see what happens:

As before, there’s no
output on screen.

Let’s check the contents of the files to see what they look like now.

What the man

said is now
legible.
And here’s what
the other man
said.

This is looking good. By amending your nester module, you’ve provided a

facility to save your list data in a legible format. It’s now way easier on the eye.

But does this make it any easier to read the data back in?

you are here 4   129

brittle code

Hang on a second...haven’t you been

here before? You’ve already written
code to read in lines from a data file and
put ‘em into lists...do you like going around
in circles?!?

That’s a good point.

This problem is not unlike the problem from the
beginning of the chapter, in that you’ve got lines of
text in a disk file that you need to process, only now
you have two files instead of one.
You know how to write the code to process your
new files, but writing custom code like this is
specific to the format that you’ve created for this
problem. This is brittle: if the data format changes,
your custom code will have to change, too.
Ask yourself: is it worth it?

130   Chapter 4
 persistence

Custom Code Exposed

This week’s interview:
When is custom code appropriate?

Head First: Hello, CC, how are you today? Custom Code: What?!? That’s where I excel:
creating beautifully crafted custom solutions for folks
Custom Code: Hi, I’m great! And when I’m not
with complex computing problems.
great, there’s always something I can do to fix things.
Nothing’s too much trouble for me. Here: have a Head First: But if something’s been done before,
seat. why reinvent the wheel?
Head First: Why, thanks. Custom Code: But everything I do is custom-
made; that’s why people come to me…
Custom Code: Let me get that for you. It’s my
new custom SlideBack&Groove™, the 2011 model, Head First: Yes, but if you take advantage of other
with added cushions and lumbar support…and it coders’ work, you can build your own stuff in half
automatically adjusts to your body shape, too. How the time with less code. You can’t beat that, can you?
does that feel?
Custom Code: “Take advantage”…isn’t that like
Head First: Actually [relaxes], that feels kinda exploitation?
groovy.
Head First: More like collaboration, sharing,
Custom Code: See? Nothing’s too much trouble participation, and working together.
for me. I’m your “go-to guy.” Just ask; absolutely
Custom Code: [shocked] You want me to give my
anything’s possible when it’s a custom job.
code…away?
Head First: Which brings me to why I’m here. I
Head First: Well…more like share and share alike.
have a “delicate” question to ask you.
I’ll scratch your back if you scratch mine. How does
Custom Code: Go ahead, shoot. I can take it. that sound?
Head First: When is custom code appropriate? Custom Code: That sounds disgusting.
Custom Code: Isn’t it obvious? It’s always Head First: Very droll [laughs]. All I’m saying is
appropriate. that it is not always a good idea to create everything
from scratch with custom code when a good enough
Head First: Even when it leads to problems down
solution to the problem might already exist.
the road?
Custom Code: I guess so…although it won’t be as
Custom Code: Problems?!? But I’ve already told
perfect a fit as that chair.
you: nothing’s too much trouble for me. I live to
customize. If it’s broken, I fix it. Head First: But I will be able to sit on it!
Head First: Even when a readymade solution Custom Code: [laughs] You should talk to my
might be a better fit? buddy Pickle…he’s forever going on about stuff like
this. And to make matters worse, he lives in a library.
Custom Code: Readymade? You mean (I hate to
say it): off the shelf? Head First: I think I’ll give him a shout. Thanks!
Head First: Yes. Especially when it comes to Custom Code: Just remember: you know where to
writing complex programs, right? find me if you need any custom work done.

you are here 4   131

in a pickle

Pickle your data

Python ships with a standard library called pickle, which can save and load
almost any Python data object, including lists.
Once you pickle your data to a file, it is persistent and ready to be read into
another program at some later date/time:

Your data as it appears The pickle engine

in Python’s memory

['Is this the right room for an

argument?', "No you haven't!", Your
'When?', "No you didn't!", "You pickled
didn't!", 'You did not!', 'Ah!
(taking out his wallet and paying)
[‘Is this the right room
for an argument?’, “No
you haven’t!”, ‘When?’,
“No you didn’t!”, “You
data
didn’t!”, ‘You did not!’,

Just the five minutes.', 'You ‘Ah! (taking out his wallet
and paying) Just the five
minutes.’, ‘You most
certainly did not!’, “Oh

most certainly did not!', "Oh

no you didn’t!”, “Oh no
you didn’t!”, “Oh look,
this isn’t an argument!”,
“No it isn’t!”, “It’s
just contradiction!”,

no you didn't!", "Oh no you

‘It IS!’, ‘You just
contradicted me!’, ‘You
DID!’, ‘You did just
then!’, ‘(exasperated)
Oh, this is futile!!’,

didn't!", "Oh look, this isn't ‘Yes it is!’]

an argument!", "No it isn't!",

"It's just contradiction!", 'It
IS!', 'You just contradicted
me!', 'You DID!', 'You did just
Out comes the
then!', '(exasperated) Oh, this
is futile!!', 'Yes it is!']
Feed your Python pickled version of
data to pickle. your data.

You can, for example, store your pickled data on disk, put it in a database,
or transfer it over a network to another computer.
When you are ready, reversing this process unpickles your persistent pickled
data and recreates your data in its original form within Python’s memory: Your data is recreated
in Python’s memory,
The same pickle engine exactly as before.

Your
pickled ['Is this the right room for an
argument?', "No you haven't!",
data 'When?', "No you didn't!", "You
didn't!", 'You did not!', 'Ah!
[‘Is this the right room
for an argument?’, “No
you haven’t!”, ‘When?’,
(taking out his wallet and paying)
Just the five minutes.', 'You
“No you didn’t!”, “You
didn’t!”, ‘You did not!’,
‘Ah! (taking out his wallet
and paying) Just the five
minutes.’, ‘You most
certainly did not!’, “Oh
no you didn’t!”, “Oh no
you didn’t!”, “Oh look,
this isn’t an argument!”,
most certainly did not!', "Oh
no you didn't!", "Oh no you
“No it isn’t!”, “It’s
just contradiction!”,
‘It IS!’, ‘You just
contradicted me!’, ‘You
DID!’, ‘You did just

didn't!", "Oh look, this isn't

then!’, ‘(exasperated)
Oh, this is futile!!’,
‘Yes it is!’]

an argument!", "No it isn't!",

"It's just contradiction!", 'It

Out comes the

IS!', 'You just contradicted
Feed your pickled me!', 'You DID!', 'You did just
data to pickle. Python version of then!', '(exasperated) Oh, this

your pickled data.

is futile!!', 'Yes it is!']

132   Chapter 4
 persistence

Save with dump and restore with load

Using pickle is straightforward: import the required module, then use
dump() to save your data and, some time later, load() to restore it. The
only requirement when working with pickled files is that they have to be
opened in binary access mode:

Always import pickle

remember ...
to import
the “pickle” with open('mydata.pickle', 'wb') as mysavedata:
The “b” tells
module. pickle.dump([1, 2, 'three'], mysavedata) Python to open
your data files
To save your data, ...
in BINARY
use “dump()”. with open('mydata.pickle', 'rb') as myrestoredata:
mode.
a_list = pickle.load(myrestoredata)
Assign youdr ata Restore your data from your
restored ntifier. print(a_list) file using “load()”.
to an ide
r program, you can
Once your data is back in youobj ect.
treat it like any other data
What if something goes wrong?
If something goes wrong when pickling or unpickling your data, the pickle
module raises an exception of type PickleError.

w Here’s a snippet of your code as it currently stands. Grab your

pencil and strike out the code you no longer need, and then
replace it with code that uses the facilities of pickle instead.
Add any additional code that you think you might need, too.

try:
with open('man_data.txt', 'w') as man_file, open('other_data.txt', 'w') as other_file:
nester.print_lol(man, fh=man_file)
nester.print_lol(other, fh=other_file)
except IOError as err:
print('File error: ' + str(err))

you are here 4   133

significance of the pickle

print(‘Pickling error: ‘ + str(perr))

Here’s a snippet of your code as it currently stands. You were to grab
your pencil and strike out the code you no longer need, and then
Import “pickle”
the top of yournear
replace it with code that uses the facilities pickle instead. You
were also to add any additional code that you think you might need.
program.
import pickle Change the access mode to
be “writeable, binary”.
try: ‘wb' 'wb'
with open('man_data.txt', 'w') as man_file, open('other_data.txt', 'w') as other_file:
nester.print_lol(man, fh=man_file) pickle.dump(man, man_file)
nester.print_lol(other, fh=other_file) pickle.dump(other, other_file)
except IOError as err:

rint_lol()”
Replace the two calls to “nester.p
print('File error: ' + str(err))

except pickle.PickleError as perr: with calls to “pickle.dump()”.

print('Pickling error: ' + str(perr))

Don’t forget to handle any except

that can occur. ions

Q: When you invoked print_lol() earlier, you provided only two arguments, even though the function signature requires you to
provide four. How is this possible?

A: When you invoke a Python function in your code, you have options, especially when the function provides default values for some
arguments. If you use positional arguments, the position of the argument in your function invocation dictates what data is assigned to which
argument. When the function has arguments that also provide default values, you do not need to always worry about positional arguments
being assigned values.

Q: OK, you’ve completely lost me. Can you explain?

A: Consider print(), which has this signature: print(value, sep=' ', end='\n', file=sys.stdout). By
default, this BIF displays to standard output (the screen), because it has an argument called file with a default value of sys.stdout.
The file argument is the fourth positional argument. However, when you want to send data to something other than the screen, you do not need
to (nor want to have to) include values for the second and third positional arguments. They have default values anyway, so you need to provide
values for them only if the defaults are not what you want. If all you want to do is to send data to a file, you invoke the print() BIF like this:
print("Dead Parrot Sketch", file='myfavmonty.txt') and the fourth positional argument uses the value
you specify, while the other positional arguments use their defaults. In Python, not only do the BIFs work this way, but your custom functions
support this mechamism, too.

134   Chapter 4
 persistence

Test Drive
Let’s see what happens now that your code has been amended to use the standard pickle module instead of
your custom nester module. Load your amended code into IDLE and press F5 to run it.

Once again, you

get no visual clue
that something has
happened.

So, once again, let’s check the contents of the files to see what they look like now:

The is the man’s

pickled data.

The is the other

man’s pickled data.

It appears to have worked…but these files look like gobbledygook! What gives?
Recall that Python, not you, is pickling your data. To do so efficiently, Python’s
pickle module uses a custom binary format (known as its protocol). As you
can see, viewing this format in your editor looks decidedly weird.

Don’t worry: it is supposed to look like this.

you are here 4   135

idle session

pickle really shines when you load some previously pickled data into another program. And, of course, there’s
nothing to stop you from using pickle with nester. After all, each module is designed to serve different
purposes. Let’s demonstrate with a handful of lines of code within IDLE’s shell. Start by importing any required
modules:
>>> import pickle
>>> import nester

No surprises there, eh?

Next up: create a new identifier to hold the data that you plan to unpickle.Create an empty list called new_man:
>>> new_man = []

Yes, almost too exciting for words, isn’t it? With your list created. let’s load your pickled data into it. As you are
working with external data files, it’s best if you enclose your code with try/except:

>>> try:
with open('man_data.txt', 'rb') as man_file:
new_man = pickle.load(man_file)
except IOError as err:
print('File error: ' + str(err))
except pickle.PickleError as perr:
print('Pickling error: ' + str(perr))

This code is not news to you either. However, at this point, your data has been unpickled and assigned to the
new_man list. It’s time for nester to do its stuff:
>>> nester.print_lol(new_man)
Is this the right room for an argument?
No you haven’t!
Not all the data is shown
When?
here, but trust us: it’s all
No you didn’t!
there.
...
You did just then!
(exasperated) Oh, this is futile!!
Yes it is!

And to finish off, let’s display the first line spoken as well as the last:
>>> print(new_man[0])
Is this the right room for an argument?
>>> print(new_man[-1])
Yes it is!

See: after all that, it is the rig

ht room! §
136   Chapter 4
 persistence

Generic file I/O with pickle is the way to go!

Now, no matter what data you create

and process in your Python programs,
you have a simple, tested, tried-
and-true mechanism for saving and
restoring your data. How cool is that?

Python takes care of your file I/O details, so you can concentrate on what
your code actually does or needs to do.
As you’ve seen, being able to work with, save, and restore data in lists is a
breeze, thanks to Python. But what other data structures does Python
support out of the box?

Let’s dive into Chapter 5 to find out.

you are here 4   137

 python toolbox

Your Python Toolbox

You’ve got Chapter 4 under your

CHAPTER 4
belt and you’ve added some key
CHAPTER 4

ƒƒ The strip() method removes

Python techiques to your toolbox.
unwanted whitespace from strings.
ƒƒ The file argument to the print()
BIF controls where data is sent/saved.
ƒƒ The finally suite is always executed
no matter what exceptions occur within a
try/except statement.
ƒƒ An exception object is passed into the

go
except suite and can be assigned to

P y th on L i n an identifier using the as keyword.

pe s ” - d at a types ƒƒ The str() BIF can be used to access

y
table t ce assigned
• “Immu
the stringed representation of any data
hat, on
in Python tnot have that value
object that supports the conversion.

a value, can ƒƒ The locals() BIF returns a collection

changed. of
of variables within the current scope.

k li n g ” - t he process rsistence ƒƒ The in operator tests for membership.

• “Pic e
g a d a t a object to p
savin ƒƒ The “+” operator concatenates two
storage. f strings when used with strings but adds

lin g ” - th e process o two numbers together when used with

k
• “Unpic
g a s a ved data object numbers.
restorin tence storage.
from persis
ƒƒ The with statement automatically
arranges to close all opened files, even
when exceptions occur. The with
statement uses the as keyword, too.
ƒƒ sys.stdout is what Python calls
“standard output” and is available from
the standard library’s sys module.
ƒƒ The standard library’s pickle module
lets you easily and efficiently save and
restore Python data objects to disk.
ƒƒ The pickle.dump() function saves
data to disk.
ƒƒ The pickle.load() function
restores data from disk.

138   Chapter 4
 5 comprehending data

Work that data!

Life could be so much
easier if only she’d let me
help her extract, sort, and
comprehend her data...

Data comes in all shapes and sizes, formats and encodings.

To work effectively with your data, you often have to manipulate and transform it into a
common format to allow for efficient processing, sorting, and storage. In this chapter, you’ll
explore Python goodies that help you work your data up into a sweat, allowing you to
achieve data-munging greatness. So, flip the page, and let’s not keep the coach waiting…

this is a new chapter   139

coaching crisis

Coach Kelly needs your help

I’m too busy on the track

to waste time fiddling with
my computer. Can you help
me process my athlete data?

The coach is an old friend, and you’d love to help. His crack squad of U10
athletes has been training hard. With each 600m run they do, Coach Kelly
has recorded their time in a text file on his computer. There are four files in james.txt
all, one each for James, Sarah, Julie, and Mikey.

julie.txt 2-34,3:21,2.34,2.45,3.01,2:01,2:01,3:10,2-22

2.59,2.11,2:11,2:23,3-10,2-23,3:10,3.21,3-21

2:22,3.01,3:01,3.02,3:02,3.02,3:22,2.49,2:38
2:58,2.58,2:39,2-25,2-55,2:54,2.18,2:55,2:55
sarah.txt mikey.txt

Initially, the coach needs a quick way to know the top three fastest times for each
athlete.
Can you help?

140   Chapter 5
 comprehending data

Do this!

Before proceeding with this

chapter, take a few moments
to download the four data
files from the Head First
Python support website.

Let’s begin by reading the data from each of the files into its own list. Write a short program to
process each file, creating a list for each athlete’s data, and display the lists on screen.
Hint: Try splitting the data on the commas, and don’t forget to strip any unwanted whitespace.

Write your
code here.

you are here 4   141

let’s split

Let’s begin by reading the data from each of the files into its own list. You were to write a short
program to process each file, creating a list for each athlete’s data. You were then to display the
lists on screen. Open the file.
with open(‘james.txt’) as jaf:
data = jaf.readline() Read the line of data.
james = data.strip().split(‘,’) Convert the data to a list.
with open(‘julie.txt’) as juf:
data = juf.readline()
Open each of
the data files julie = data.strip().split(‘,’)
in turn, read
the line of with open(‘mikey.txt’) as mif:
data from the
file, and create data = mif.readline()
a list from the mikey = data.strip().split(‘,’)
line of data.
with open(‘sarah.txt’) as saf:
data = saf.readline()
sarah = data.strip().split(‘,’)

print(james)
print(julie)
print(mikey) Display the four lists on screen.

print(sarah)

Q: That data.strip().split(',') line looks a little weird. Can you explain what’s going on?

A: That’s called method chaining. The first method, strip(), is applied to the line in data, which removes any unwanted whitespace
from the string. Then, the results of the stripping are processed by the second method, split(','), creating a list. The resulting list is
then applied to the target identifier in the previous code. In this way, the methods are chained together to produce the required result. It helps
if you read method chains from left to right.

142   Chapter 5
 comprehending data

Test Drive
Load your code into IDLE and run it to confirm that it’s all OK for now:

Here’s your
program as
displayed in
IDLE.

And here’s the output

produced by running
your code.

So far, so good. Coach Kelly’s data is now represented by four lists in Python’s
memory. Other than the use of method chaining, there’s nothing much new
here, because you’ve pretty much mastered reading data from files and using
it to populate lists.
There’s nothing to show the coach yet, so no point in disturbing him until his
data is arranged in ascending order, which requires you to sort it.

Let’s look at your sorting options in Python.

you are here 4   143

 in-place or copied sorting

Sort in one of two ways

When it comes to sorting your data using Python, you have two options.
In-place sorting takes your data, arranges it in the order you specify, and
then replaces your original data with the sorted version. The original ordering
is lost. With lists, the sort() method provides in-place sorting:

The Python “In-

Engine transformplace Sorting”
s, then replaces.
The original, a
unordered dat ]
, 5 ]
2,
6
1, 2, 3, 4, 5, 6
4, [
,
, 3
[ 1
The original data has
now been ordered (and
replaced).

Copied sorting takes your data, arranges it in the order you specify, and
then returns a sorted copy of your original data. Your original data’s ordering
is maintained and only the copy is sorted. In Python, the sorted() BIF
supports copied sorting.

The Python “Cop

Engine transformied Sorting”
s and returns.

The original, a
unordered dat [ 1, 3
, 4, 2
, 6, 5
] ]
6, 5
2,
, 4, The original, unordered data
, 3 remains UNTOUCHED.
[ 1

]
4 , 5, 6
2, 3,
[ 1,
a has now been
The dat pied).
ordered (and co
144   Chapter 5
 comprehending data

Let’s see what happens to your data when each of Python’s sorting options is used. Start by creating an unordered
list at the IDLE shell:

>>> data = [6, 3, 1, 2, 4, 5] Create a list of

>>> data unordered data and
[6, 3, 1, 2, 4, 5] assign to a variable.
Perform an in-place sort using the sort() method that is built in as standard to every Python list:

Perform IN-PLACE sorting on

>>> data.sort()
the data.
>>> data
[1, 2, 3, 4, 5, 6] The data’s ordering has changed.
Reset data to its original unordered state, and then perform a copied sort using the sorted() BIF:

>>> data = [6, 3, 1, 2, 4, 5]

et.
The data’s ordering has been res
>>> data
[6, 3, 1, 2, 4, 5]
>>> data2 = sorted(data)
Perform COPIED sorting on the
>>> data data.
[6, 3, 1, 2, 4, 5] Same as it ever was.
>>> data2
The copied data is ordered
from lowest to highest.
[1, 2, 3, 4, 5, 6]

Either sorting option works with the coach’s data, but let’s use a
copied sort for now to arrange to sort the data on output. In the
space below, provide four amended print() statements to
replace those at the bottom of your program.

you are here 4   145

all sorted

Either sorting option works with the coach’s data, but let’s use
a copied sort for now to arrange to sort the data on output. You
were to provide four amended print() statements to replace
those at the bottom of your program.

print(sorted(james))
Simply call
“sorted()” on the print(sorted(julie))
data BEFORE
you display it on print(sorted(mikey))
screen.
print(sorted(sarah))

Q: What happens to the unsorted data when I use sort()? Q: And there’s no way to get the original data back?

A: For all intents and purposes, it disappears. Python takes a copy,

sorts it, and then replaces your original data with the sorted version.
A:sorted()
the
No. If the ordering of the original data is important to you, use
BIF to transform your data into a sorted copy.

Geek Bits

You’ve already seen method chaining, and now it’s time to say
“hello” to function chaining. Function chaining allows you to apply
a series of functions to your data. Each function takes your data,
performs some operation on it, and then passes the transformed
data on to the next function. Unlike method chains, which read
from left to right, function chains read from right to left (just to
keep things interesting).

146   Chapter 5
 comprehending data

Test Drive
Let’s see if this improves your output in any way. Make the necessary amendments to your code and
run it.

updates
Here’s thede. But look at T
to the co The data is notHIS!
sorted…which is, at all
weird. like,

Look at this: 2-55 is

coming BEFORE 2.18…now
that is weird. Hey, it looks like your data values are
not uniform. Is the problem with all those
periods, dashes, and colons?

Yes. The minute and seconds separators are

confusing Python’s sorting technology.
When recording his athletes’ times in each of their files, Coach
Kelly sometimes used a different character to separate minutes
from seconds. It looks like you need to fix your data.

you are here 4   147

time trials

The trouble with time

Well…there’s never enough of it, is there?
Let’s look closely at the coach’s data to see what the problem is. Here’s Sarah
raw data again: Oh, look: what a lovely
bunch of strings...

2:58,2.58,2:39,2-25,2-55,2:54,2.18,2:55,2:55
sarah.txt

Recall that data read from a file comes into your program as text, so Sarah’s
data looks like this once you turn it into a list of “times”:

['2:58', '2.58', '2:39’, '2-25', '2-55', '2:54’, '2.18', '2:55', '2:55']

h the coach
These are all strings, even thoug
thinks they’re times.

And when you sort Sarah’s data, it ends up in this order (which isn’t quite
what you were expecting): I don’t get what the
problem is...they’re all
times to me.
['2-25', '2-55', '2.18', '2.58', '2:39', '2:54', '2:55', '2:55', '2:58']

Whoops! That’s not Whoops again. 2:39

right. How can 2.18 can’t come between
come after 2-55? 2.58 and 2:54, can it?

Python sorts the strings, and when it comes to strings, a dash comes before a
period, which itself comes before a colon. As all the strings start with 2, the
next character in each string acts like a grouping mechanism, with the dashed
times grouped and sorted, then the period times, and finally the colon times.

Nonuniformity in the coach’s data is causing the sort to fail.

148   Chapter 5
 comprehending data

Code Magnets
Let’s create a function called sanitize(), which takes as input
a string from each of the athlete’s lists. The function then processes
the string to replace any dashes or colons found with a period and
returns the sanitized string. Note: if the string already contains a
period, there’s no need to sanitize it.
Rearrange the code magnets at the bottom of the page to provide
the required functionality.

def sanitize(time_string):

return(mins + '.' + secs)

Return the sanitized time

the caller of this function. string to

elif ':' i
Your magnets are
n time_str
ing:
waiting. time_string.split(splitter)
return(time_stri
if '-' in ng)
time_strin
g:
else:
(mins, secs) =
'-'
splitter = ':' ter =
split

you are here 4   149

sanitzing function

Code Magnets Solution

You were to create a function called sanitize(), which takes
as input a string from each of the athlete’s lists. The function then
processes the string to replace any dashes or colons found with a
period and returns the sanitized string. Note: if the string already
contains a period, there’s no need to sanitize it.
You were to rearrange the code magnets at the bottom of the
previous page to provide the required functionality.

def sanitize(time_string):

if '-' in time_string:

splitter = '-'
Use the “in”
operator to elif ':' in time_string:
check if the
string contains splitter = ':'
a dash or a
colon. else: Do nothing if the string do
return(time_string)
NOT need to be sanitized. es

(mins, secs) = time_string.split(splitter)

return(mins + '.' + secs) Split the string to extract the

minutes and seconds parts.

Of course, on its own, the sanitize() function is not enough. You need
to iterate over each of your lists of data and use your new function to convert
each of the athlete’s times into the correct format.

Let’s put your new function to work right away.

150   Chapter 5
 comprehending data

Let’s write the code to convert your existing data into a sanitized version of itself. Create four
new lists to hold the sanitized data. Iterate over each athlete’s list data and append each
sanitized string from each list to the appropriate new list. Conclude your program by printing a
sorted copy of each new list to the screen.

with open('james.txt') as jaf: data = jaf.readline()

The code that
reads the data
james = data.strip().split(',')

from the data with open('julie.txt') as juf: data = juf.readline()

files remains julie = data.strip().split(',')
unchanged with open('mikey.txt') as mif: data = mif.readline()
(and has been
compressed to fit
mikey = data.strip().split(',')

on this page). with open('sarah.txt') as saf: data = saf.readline()

sarah = data.strip().split(',')

Add your new

code here.

What happens
to the four
“print()” print( )

statements? print( )

print( )

print( )

you are here 4   151

sanitized for your protection

Let’s write the code to convert your existing data into a sanitized version of itself. You were to
create four new lists to hold the sanitized data. You were then to iterate over each athlete’s data
and append each sanitized string from each list to an appropriate new list. You were to conclude
your program by printing a sorted copy of each new list to the screen.

with open('james.txt') as jaf: data = jaf.readline()

james = data.strip().split(',')
with open('julie.txt') as juf: data = juf.readline()
julie = data.strip().split(',')
with open('mikey.txt') as mif: data = mif.readline()
mikey = data.strip().split(',')
with open('sarah.txt') as saf: data = saf.readline()
sarah = data.strip().split(',')

clean_james = []
Create four clean_julie = []
new, initially
empty lists. clean_mikey = []
clean_sarah = []

for each_t in james:

clean_james.append(sanitize(each_t))
for each_t in julie:
clean_julie.append(sanitize(each_t)) Take each of the data items in
the original lists, sanitize them,
for each_t in mikey: and then append the sanitized
clean_mikey.append(sanitize(each_t)) data to the appropriate new
list.
for each_t in sarah:
clean_sarah.append(sanitize(each_t))
The four “print()”
statements now print( sorted(clean_james) )
display the new print( sorted(clean_julie) )
lists, which are
sorted, too. print( sorted(clean_mikey) )

print( sorted(clean_sarah) )

152   Chapter 5
 comprehending data

Test Drive
Combine your sanitize() function with your amended code from the previous page, and then
press F5 in IDLE to confirm the sorting is now working as expected.

The sanitized data only

contains ‘.’ as a separator.

The ordering works, because all Four sorted

times are now comparable. the
lists

This output looks much better.

It’s taken a bit of work, but now the data from each of the four files is both
sorted and uniformly formatted. By preprocessing your data before you
sort it, you’ve helped ensure Python’s sorting technology performs correctly.

Geek Bits

By default, both the sort() method and the sorted()

BIF order your data in ascending order. To order your data in
descending order, pass the reverse=True argument to
either sort() or sorted() and Python will take care of
things for you.

you are here 4   153

duplicated code

Hang on a sec! Something doesn’t feel quite

right...look at all that duplicated code, as well as
all those duplicated lists. This duplication is bad,
right? Is this really the best you can do?

That’s right. Duplicated code is a problem.

As things stand, your code creates four lists to hold the data as read
from the data files. Then your code creates another four lists to hold
the sanitized data. And, of course, you’re iterating all over the
place…
There has to be a better way to write code like this.
Transforming lists is such a common requirement that Python
provides a tool to make the transformation as painless as
possible. This tool goes by the rather unwieldly name of list
comprehension. And list comprehensions are designed to reduce
the amount of code you need to write when transforming one list
into another.

154   Chapter 5
 comprehending data

Comprehending lists
Consider what you need to do when you transform one list into another. Four
things have to happen. You need to:

1 Create a new list to hold the transformed data.

2 Iterate each data item in the original list.

3 With each iteration, perform the transformation.

4 Append the transformed data to the new list.

1. Create.
clean_mikey = [] 2. Iterate.

for each_t in mikey:

3. Transform.
clean_mikey.append(sanitize(each_t))

4. Append.

Here’s the same functionality as a list comprehension, which involves

You get to pick the target
creating a new list by specifying the transformation that is to be applied to each
identifier to use (just like with
of the data items within an existing list.
regular iterations).

clean_mikey = [sanitize(each_t) for each_t in mikey]

The new list is created… …by applying a …to each …within an existing list.
transformation… data item…

What’s interesting is that the transformation has been reduced to a single line
of code. Additionally, there’s no need to specify the use of the append()
method as this action is implied within the list comprehension. Neat, eh?

you are here 4   155

idle session

Let’s see some other list comprehension examples. Open up your IDLE shell and follow along with these one-liner
transformations.
Start by transforming a list of minutes into a list of seconds:
>>> mins = [1, 2, 3]
>>> secs = [m * 60 for m in mins]
>>> secs Simply multiply the
[60, 120, 180]
minute values by 60.

How about meters into feet?

>>> meters = [1, 10, 3]

Yes, there are 3.281

>>> feet = [m * 3.281 for m in meters]
>>> feet
feet in a meter.
[3.281, 32.81, 9.843]

Given a list of strings in mixed and lowercase, it’s a breeze to transform the strings to UPPERCASE:

>>> lower = ["I", "don't", "like", "spam"]

>>> upper = [s.upper() for s in lower]
>>> upper Every string comes with
['I', "DON'T", 'LIKE', 'SPAM'] the “upper()” method.

Let’s use your sanitize() function to transform some list data into correctly formatted times:

>>> dirty = ['2-22', '2:22', '2.22']

>>> clean = [sanitize(t) for t in dirty]
It’s never been so easy to tur n something
dirty into something clean. §
>>> clean
['2.22', '2.22', '2.22']

It’s also possible to assign the results of the list transformation back onto the original target identifier. This
example transforms a list of strings into floating point numbers, and then replaces the original list data:
>>> clean = [float(s) for s in clean]
>>> clean
The “float()” BIF converts to flo
[2.22, 2.22, 2.22] ating point.
And, of course, the transformation can be a function chain, if that’s what you need:

>>> clean = [float(sanitize(t)) for t in ['2-22', '3:33', '4.44']]

e data
Combining transformations on th
>>> clean

items is supported, too!

[2.22, 3.33, 4.44]

156   Chapter 5
 comprehending data

Now that you know about list comprehensions, let’s write four of
them to process the coach’s four lists of timing values. Transform
each of your lists into sorted, sanitized version of themselves.
Grab your pencil and in the space provided, scribble the list
comprehensions you plan to use.

Q: So…let me get this straight: list comprehensions are good and list iterations are bad, right?

A: No, that’s not the best way to look at it. If you have to perform a transformation on every item in a list, using a list comprehension is the
way to go, especially when the transformation is easily specified on one line (or as a function chain). List iterations can do everything that list
comprehensions can, they just take more code, but iterations do provide more flexibility should you need it.

Geek Bits

Python’s list comprehension is an example of the language’s

support for functional programming concepts. There’s plenty of
debate about the best way to develop program code: either
procedurally, using functional programming techniques, or
using object orientation. At Head First Labs, we try not to get
involved in this debate, other than to rejoice in the fact that
Python supports, in one way or another, all three of these
programming practices.

you are here 4   157

list comprehensions

Now that you know about list comprehensions, you were to write
four of them to process the coach’s four lists of timing values. You
were to transform each of your lists into sorted, sanitized version
of themselves. You were to grab your pencil and in the space
provided, scribble the list comprehensions you plan to use.

The list comprehension

performs the sorted([sanitize(t) for t in james])
transformation, and the
new list is then ordered
by the “sorted()” BIF.
sorted([sanitize(t) for t in julie])
Rinse and repeat
sorted([sanitize(t) for t in mikey]) for the other lists.
sorted([sanitize(t) for t in sarah])

Be careful about where you use

the sorted() BIF when defining
your list comprehensions.
You may have been tempted to use the
function chain sorted(sanitize(t))
within your list comprehension. Don’t be. Recall that the
transformation works on one list item at a time, not the
entire list. In this example, the sorted() BIF expects
to sort a list, not an individual data item.

The beauty of list comprehensions

The use of list comprehensions with the coach’s athlete data has resulted
in a lot less code for you to maintain. Additionally, as you get used to list
comprehension syntax and usage, you’ll find that their use is natural and
matches the way your brain thinks about your data and the transformations
that you might want to apply.

Let’s confirm that your new code is working as expected.

158   Chapter 5
 comprehending data

Test Drive
Replace your list iteration code from earlier with your four new (beautiful) list comprehensions. Run
your program to confirm that the results have not changed.

Your new list comprehensions

produce EXACTLY the
same output as your earlier
list iterations.

As expected, the outout matches that from earlier.

You’ve written a program that reads Coach Kelly’s data from his data files,
stores his raw data in lists, sanitizes the data to a uniform format, and then
sorts and displays the coach’s data on screen. And all in 25 lines of code.
It’s probably safe to let the coach take a look at your output now.

What will the coach think?

you are here 4   159

list slicing

Have you not been drinking enough

water? I wanted the three fastest times
for each athlete...but you’ve given me
everything and it contains duplicates!

In your haste to sanitize and sort your data, you forgot to worry about what
you were actually supposed to be doing: producing the three fastest times for each
athlete. And, of course, there’s no place for any duplicated times in your
output.
Accessing the first three data items from any list is easy. Either specify each list
item individually using the standard notation or use a list slice:

james[0]
Access each data item
you need individually. james[1]
james[2]

Use a list slice to access fromdinlistg

james[0:3] item 0 up-to-but-not-inclu
list item 3.
But…what about removing duplicates from your list?

160   Chapter 5
 comprehending data

Iterate to remove duplicates

Processing a list to remove duplicates is one area where a list comprehension
can’t help you, because duplicate removal is not a transformation; it’s more of
a filter. And a duplicate removal filter needs to examine the list being created
as it is being created, which is not possible with a list comprehension.
To meet this new requirement, you’ll need to revert to regular list iteration
code.

Assume that the fourth from last line of code from your current program is changed to this:
james = sorted([sanitize(t) for t in james])
That is, instead of printing the sanitized and sorted data for James to the screen, this line of
code replaces James’s unordered and nonuniform data with the sorted, sanitized copy.
Your next task is to write some code to remove any duplicates from the james list produced
by the preceding line of code. Start by creating a new list called unique_james, and then
populate it with the unique data items found in james. Additionally, provide code to display only
the top three fastest times for James.
Hint: you might want to consider using the not in operator.

you are here 4   161

top three

Assume that the fourth from last line of code from your current program is changed to this:
james = sorted([sanitize(t) for t in james])
That is, instead of printing the sanitized and sorted data for James to the screen, this line of
code replaces James’s unordered and non-uniform data with the sorted, sanitized copy.
Your next task was to write some code to remove any duplicates from the james list produced
by the preceding line of code. You were to start by creating a new list called unique_james
and then populate it with the unique data items found in james. Additionally, you were to
provide code to only display the top three fastest times for James.

Create the
empty list to unique_james = []
hold the unique
data items. Iterate over the
existing data…
for each_t in james: …and if the data item ISN’T
already in the new list…
if each_t not in unique_james:
m to
unique_james.append(each_t) …append the unique data ite
the new list.

Slice the first

three data items
from the list and print(unique_james[0:3])
display them on
screen.

Do this!

Repeat the code on this page

for the rest of the coach’s lists:
julie, mikey & sarah. Add
all of your new code to your
existing program.

162   Chapter 5
 comprehending data

Test Drive
Take all of the recent amendments and apply them to your program. Run this latest code within IDLE
when you are ready.

Sort and sanitize

each list.

Remove the
duplicates.

Looking good!

It worked!

You are now displaying only the top three times for each athlete, and the
duplicates have been successfully removed.
The list iteration code is what you need in this instance. There’s a little bit of
duplication in your code, but it’s not too bad, is it?

you are here 4   163

duplicated code to remove duplication?

“Not too bad”...you’re kidding, right?!?

Surely there’s something that can be done
with all that duplicated duplicate code?

The irony is hard to avoid, isn’t it?

The code that removes duplicates from your lists is itself
duplicated.
Sometimes such a situation is unavoidable, and sometimes
creating a small function to factor out the duplicated code can
help. But something still doesn’t feel quite right here…

164   Chapter 5
 comprehending data

Wouldn't it be dreamy if there were a way to

quickly and easily remove duplicates from an
existing list? But I know it's just a fantasy...

you are here 4   165

factory functions

Remove duplicates with sets

In addition to lists, Python also comes with the set data structure, which
behaves like the sets you learned all about in math class.
The overriding characteristics of sets in Python are that the data items in a set
are unordered and duplicates are not allowed. If you try to add a data item to a set
that already contains the data item, Python simply ignores it.
Create an empty set using the set() BIF, which is an example of a factory
function:
Create a new,
empty set, and
assign it to a distances = set()
variable.

It is also possible to create and populate a set in one step. You can provide a list
of data values between curly braces or specify an existing list as an argument
to the set() BIF, which is the factory function:

Any duplicates in
the supplied list
distances = {10.6, 11, 8, 10.6, "two", 7} of data values are
ignored.

Any duplicates in distances = set(james)

the “james” list are
ignored. Cool.

Factory Function: A factory function is used to make new

data items of a particular type. For instance, “set()” is
a factory function because it makes a new set. In the
real world, factories make things, hence the name.

166   Chapter 5
 comprehending data

Tonight’s talk: Does list suffer from set envy?

List: Set:
[sings] “Anything you can do, I can do better. I can
do anything better than you.” I’m resisting the urge to say, “No, you can’t.”
Instead, let me ask you: what about handling
duplicates? When I see them, I throw them away
automatically.
Can you spell “d-a-t-a l-o-s-s”? Getting rid of data
automatically sounds kinda dangerous to me.
But that’s what I’m supposed to do. Sets aren’t
allowed duplicate values.
Seriously?
Yes. That’s why I exist…to store sets of values.
Which, when it’s needed, is a real lifesaver.

And that’s all you do?

That’s all I need to do.

And they pay you for that?!?

Very funny. You’re just being smug in an effort
to hide from the fact that you can’t get rid of
duplicates on your own.

Have you ever considered that I like my duplicate

values. I’m very fond of them, you know.
Yeah, right. Except when you don’t need them.
Which isn’t very often. And, anyway, I can always
rely on the kindness of others to help me out with
I think you meant to say, “the kindness of set()”,
any duplicates that I don’t need.
didn’t you?

To extract the data you need, replace

Do this! all of that list iteration code in your
current program with four calls to
sorted(set(...))[0:3].

you are here 4   167

code review

Head First
Code Review
The Head First Code Review Team has taken your code and
annotated it in the only way they know how: they’ve scribbled I think we
all over it. Some of their comments are confirmations of what can make a few
you might already know. Others are suggestions that might improvements here.
make your code better. Like all code reviews, these comments
are an attempt to improve the quality of your code.
def sanitize(time_string): A comment would
if '-' in time_string: be nice to have
splitter = '-'
here.
elif ':' in time_string:
splitter = ':'
else:
return(time_string)
(mins, secs) = time_string.split(splitter)
return(mins + '.' + secs)

What happens with open('james.txt') as jaf: Meet the Head

First Code Review
if one of these james = data.strip().split(',')
data = jaf.readline()
Team.
files is missing?!?
Where’s your
You
with open('julie.txt') as juf:
t io n ha n d li n g There’s a bit of duplication here.
excep data = juf.readline()
small
code? julie = data.strip().split(',')
could factor out the code into a
is call
function; then, all you need to do
hlete
the function for each of your at
with open('mikey.txt') as mif:

an
data files, assigning the result to
data = mif.readline()
mikey = data.strip().split(',')
athlete list.
with open('sarah.txt') as saf:

There’s a lot data = saf.readline()

going on here,
sarah = data.strip().split(',')

but we find it’s print(sorted(set([sanitize(t) for Ah, OK. We get it.

not too hard to print(sorted(set([sanitize(t) for
t in james]))[0:3])
The slice is applied to
u
t in julie]))[0:3])
understand if yo print(sorted(set([sanitize(t) for t in mikey]))[0:3]) the list produced by
e
read it from th print(sorted(set([sanitize(t) for t in sarah]))[0:3])
“sorted()”, right?
inside out.

168   Chapter 5
 comprehending data

Let’s take a few moments to implement the review team’s suggestion to turn those four with
statements into a function. Here’s the code again. In the space provided, create a function to
abstract the required functionality, and then provide one example of how you would call your
new function in your code:

with open('james.txt') as jaf:

data = jaf.readline()
james = data.strip().split(',')

with open('julie.txt') as juf:

data = juf.readline()
julie = data.strip().split(',')

with open('mikey.txt') as mif:

data = mif.readline()
mikey = data.strip().split(',')

with open('sarah.txt') as saf:

data = saf.readline()
sarah = data.strip().split(',')

Write your new

function here.

Provide one
example call.

you are here 4   169

statement to function

You were to take a few moments to implement the review team’s suggestion to turn those four
with statements into a function. In the space provided, your were to create a function to
abstract the required functionality, then provide one example of how you would call your new
function in your code:

with open('james.txt') as jaf:

data = jaf.readline()
james = data.strip().split(',')

with open('julie.txt') as juf:

data = juf.readline()
julie = data.strip().split(',')

with open('mikey.txt') as mif:

data = mif.readline()
mikey = data.strip().split(',')

with open('sarah.txt') as saf:

data = saf.readline()
sarah = data.strip().split(',')
Accept a filename as
the sole argument.
def get_coach_data(filename):
Create a new
function. try:
Open the file, and
with open(filename) as f: read the data.
Add the suggested
exception-handling data = f.readline() Perform the split/strip tri
code. the data prior to returningckitonto
return(data.strip().split(‘,')) the calling code.
except IOError as ioerr:
print(‘File error: ' + str(ioerr))
return(None) Tell your user about the errorone”
(if it occurs) and return “N
to indicate failure.
Calling the function sarah = get_coach_data(‘sarah.txt')
is straightforward.
Provide the name of the file to
process.

170   Chapter 5
 comprehending data

Test Drive
It’s time for one last run of your program to confirm that your use of sets produces the same results
as your list-iteration code. Take your code for a spin in IDLE and see what happens.

As expected, your latest

code does the business.
Looking good!

Excellent!

You’ve processed the coach’s data perfectly, while That’s great work, and just
taking advantage of the sorted() BIF, sets, what I need. Thanks! I’m
and list comprehensions. As you can imagine, you looking forward to seeing
can apply these techniques to many different you on the track soon...
situations. You’re well on your way to becoming a
Python data-munging master!

you are here 4   171

 python toolbox

Your Python Toolbox

You’ve got Chapter 5 under your
belt and you’ve added some more
CHAPTER 5

Python techiques to your toolbox.

t h o n L in g o
Py ing - trans
forms
ƒƒ The sort() method changes the

lace” sort .
ordering of lists in-place.
• “In-p
eplaces
and then r
ƒƒ The sorted() BIF sorts most any data

so r tin g - transforms structure by providing copied sorting.

• “Copie
d”
eturns. ƒƒ Pass reverse=True to either
and then r - reading
sort() or sorted() to arrange your
C h a in in g ”
• “Met
hod t , applies a .
data in descending order.
t t o r ig h
from lef f methods to data ƒƒ When you have code like this:
collection o ” - reading
new_l = []
h a in in g for t in old_l:
• “Func
tion C t, applies a
to lef
new_l.
from right f functions to data. append(len(t))
collection o rewrite it to use a list comprehension,
like this:

More Python Lin new_l = [len(t) for t

•“List Com
go in old_l]

preh ƒƒ To access more than one data item from

a transformatio ension” - specify a list, use a slice. For example:
opposed to using n on one line (as my_list[3:6]
an iteration). accesses the items from index location 3
• A “slice”
- acc up-to-but-not-including index location 6.
item from a list ess more than one
. ƒƒ Create a set using the set() factory
• A “set” - function.
a co
unordered data llection of
contains no dupli items that
cates.

172   Chapter 5
6 custom data objects

Bundling code with data

The object of my desire

[sigh] is in a class of
her own.

It’s important to match your data structure choice to your data. 

And that choice can make a big difference to the complexity of your code. In Python,
although really useful, lists and sets aren’t the only game in town. The Python dictionary
lets you organize your data for speedy lookup by associating your data with names, not
numbers. And when Python’s built-in data structures don’t quite cut it, the Python class
statement lets you define your own. This chapter shows you how.

this is a new chapter   173

additional data

Coach Kelly is back

(with a new file format)
I love what you’ve done, but I can’t tell which line
of data belongs to which athlete, so I’ve added some
information to my data files to make it easy for you to
figure it out. I hope this doesn’t mess things up much.

The output from your last program in Chapter 5 was exactly what the coach
was looking for, but for the fact that no one can tell which athlete belongs to
which data. Coach Kelly thinks he has the solution: he’s added identification This is “sarah2.txt”, with
data to each of his data files: extra data added.

Sarah Sweeney,2002-6-17,2:58,2.58,2:39,2-25,2-55,2:54,2.18,2:55,2:55,2:22,2-21,2.22

Sarah’s full name Sarah’s timing data

Sarah’s date of birth

If you use the split() BIF to extract Sarah’s data into a list, the first data Do this!
item is Sarah’s name, the second is her date of birth, and the rest is Sarah’s
timing data.

Let’s exploit this format and see how well things work. Grab the updated files from the
Head First Python website.
174   Chapter 6
 custom data objects

Code Magnets
Let’s look at the code to implement the strategy outlined at the bottom of the previous page. For
now, let’s concentrate on Sarah’s data. Rearrange the code magnets at the bottom of this page to
implement the list processing required to extract and process Sarah’s three fastest times from Coach
Kelly’s raw data.

Hint: the pop() method removes and returns a data item from the specified list location.

The “sanitize()” function is as

def sanitize(time_string):
if '-' in time_string:
it was in Chapter 5.
splitter = '-'
elif ':' in time_string:
splitter = ':'
else:
return(time_string)
(mins, secs) = time_string.split(splitter)
return(mins + '.' + secs)

on is
def get_coach_data(filename): The “get_coach_data()” functi
try: also from the last chapter.
with open(filename) as f:
data = f.readline()
return(data.strip().split(','))
except IOError as ioerr:
print('File error: ' + str(ioerr))
return(None)

Rearrange the
magnets here.

(sarah_name, sarah_dob)
sarah "'s fastes =
t times ar
e: " +
print(sarah_name +
get_coach_data('sarah2.txt') =
sarah.pop(0), sarah.pop
(0)

str(sorted(set([sanitize(t) for t in sarah]))[0:3]))

you are here 4   175
sarah’s times

Code Magnets Solution

Let’s look at the code to implement the strategy outlined earlier. For now, let’s concentrate on
Sarah’s data.
You were to rearrange the code magnets at the bottom of the previous page to implement the list
processing required to extract and process Sarah’s three fastest times from Coach Kelly’s raw data.

def sanitize(time_string):
if '-' in time_string:
splitter = '-'
elif ':' in time_string:
splitter = ':'
else:
return(time_string)
(mins, secs) = time_string.split(splitter)
return(mins + '.' + secs)

def get_coach_data(filename):
try:
with open(filename) as f:
data = f.readline()
return(data.strip().split(','))
except IOError as ioerr: Use the function to turn
print('File error: ' + str(ioerr)) Sarah’s data file into a list,
return(None) and then assign it to the
“sarah” variable.

sarah = get_coach_data('sarah2.txt')

(sarah_name, sarah_dob) = sarah.pop(0), sarah.pop(0)

print(sarah_name +
The “pop(0)” call "'s fastest times are: " +

returns and str(sorted(set([sanitize(t) for t in sarah]))[0:3]))

removes data from
the front of a
list.. Two calls to A custom message within
“pop(0)” remove the call to “print()” is used
the first two data to display the results you’re
values and assigns after.
them to the named
variables.

176   Chapter 6
 custom data objects

Test Drive
Let’s run this code in IDLE and see what happens.

Your latest code

This output
is much more
understandable.

This program works as expected, and is fine…except that you have to name and create
Sarah’s three variables in such as way that it’s possible to identify which name, date of birth,
and timing data relate to Sarah. And if you add code to process the data for James, Julie,
and Mikey, you’ll be up to 12 variables that need juggling. This just about works for now
with four athletes. But what if there are 40, 400, or 4,000 athletes to process?
Although the data is related in “real life,” within your code things are disjointed, because
the three related pieces of data representing Sarah are stored in three separate variables.

you are here 4   177

 keys and values

Use a dictionary to associate data

Lists are great, but they are not always the best data structure for every
situation. Let’s take another look at Sarah’s data:

Sarah’s full name Sarah’s date of birth

Sarah’s timing data

Sarah Sweeney,2002-6-17,2:58,2.58,2:39,2-25,2-55,2:54,2.18,2:55,2:55,2:22,2-21,2.22

There’s a definite structure here: the athlete’s name, the date of birth, and
then the list of times.
Let’s continue to use a list for the timing data, because that still makes sense.
But let’s make the timing data part of another data structure, which associates
all the data for an athlete with a single variable.
We’ll use a Python dictionary, which associates data values with keys:

The “keys”
Name "Sarah Sweeney"
The associated data, also known
as the “values”
DOB "2002-6-17"

Times [2:58,2.58,2:39,2-25,2-55,2:54,2.18,2:55,2:55,2:22,2-21,2.22]

Dictionary A built-in data structure (included with

Python) that allows you to associate data with keys, as
opposed to numbers. This lets your in-memory data
closely match the structure of your actual data.

178   Chapter 6
 custom data objects

Tonight’s talk: To use a list or not to use a list?

Dictionary: List:
Hi there, List. I hear you’re great, but not always
the best option for complex data. That’s where I
come in.
What?!? Haven’t you heard? You can put anything
into a list, anything at all.

True. But when you do, you lose any structure

associated with the data you are processing.
Well…assuming, of course, that structure is
important to you.

Isn’t it always?
Ummm, uh…I guess so.

You guess so? When it comes to modeling your data

in code, it’s best not to guess. Be firm. Be strong. Be
assertive. Use a dictionary. That sounds like a slogan from one of those awful
self-help conferences. Is that where you heard it?

[laughs] Oh, I do love your humor, List, even when

you know you’re on thin ice. Look, the rule is
simple: if your data has structure, use a dictionary, not a
list. How hard is that?
Not that hard, really. Unless, of course, you are a
list, and you miss being used for every piece of data
in a program…

Which rarely makes sense. Knowing when to use a

list and when to use a dictionary is what separates
the good programmers from the great ones, right?
I guess so. Man, I do hate it when you’re right!

Geek Bits

The Python dictionary is known by different names in other programming languages. If you hear other
programmers talking about a “mapping,” a “hash,” or an “associative array,” they are talking about a “dictionary.”

you are here 4   179

idle session

Let’s see the Python dictionary in action. Follow along with this IDLE session on your computer, ensuring that you
get the same results as shown.
Start by creating two empty dictionaries, one using curly braces and the other using a factory function:

>>> cleese = {}
>>> palin = dict()
Both techniques create
>>> type(cleese) an empty dictionary, as
<class 'dict'> confirmed.
>>> type(palin)
<class 'dict'>

Add some data to both of these dictionaries by associating values with keys. Note the actual structure of the data
is presenting itself here, as each dictionary has a Name and a list of Occupations. Note also that the palin
dictionary is being created at the same time:
>>> cleese['Name'] = 'John Cleese'
>>> cleese['Occupations'] = ['actor', 'comedian', 'writer', 'film producer']
>>> palin = {'Name': 'Michael Palin', 'Occupations': ['comedian', 'actor', 'writer', 'tv']}

With your data associated with keys (which are strings, in this case), it is possible to access an individual data item
using a notation similar to that used with lists:

>>> palin['Name']
Use square brackets to index into the dictionary to access
data items, but instead of numbers, index with keys.
Use numbers to access a list item stored at a particular dictionary key.
'Michael Palin'

Think of this as “index-chaining” and read from right to left: “…the last
>>> cleese['Occupations'][-1]

item of the list associated with Occupations…”.

'film producer'

As with lists, a Python dictionary can grow dynamically to store additional key/value pairings. Let’s add some data
about birthplace to each dictionary:
Provide the data associated
>>> palin['Birthplace'] = "Broomhill, Sheffield, England" with the new key.
>>> cleese['Birthplace'] = "Weston-super-Mare, North Somerset, England"

Unlike lists, a Python dictionary does not maintain insertion order, which can result in some unexpected
behavior. The key point is that the dictionary maintains the associations, not the ordering:

>>> palin
{'Birthplace': 'Broomhill, Sheffield, England', 'Name': 'Michael Palin', 'Occupations':
['comedian', 'actor', 'writer', 'tv']}
>>> cleese
{'Birthplace': 'Weston-super-Mare, North Somerset, England', 'Name': 'John Cleese',
'Occupations': ['actor', 'comedian', 'writer', 'film producer']}
The ordering maintained by Python is different from how the data
was inserted. Don’t worry about it; this is OK.

180   Chapter 6
 custom data objects

It’s time to apply what you now know about Python’s dictionary to your code. Let’s continue to
concentrate on Sarah’s data for now. Strike out the code that you no longer need and replace it
with new code that uses a dictionary to hold and process Sarah’s data.
def sanitize(time_string):
if '-' in time_string:
splitter = '-'
elif ':' in time_string:
splitter = ':'
else:
return(time_string)
(mins, secs) = time_string.split(splitter)
return(mins + '.' + secs)

def get_coach_data(filename):
try:
with open(filename) as f:
data = f.readline()
return(data.strip().split(','))
except IOError as ioerr:
print('File error: ' + str(ioerr))
Strike out the code
you no longer need. return(None)

sarah = get_coach_data('sarah2.txt')
(sarah_name, sarah_dob) = sarah.pop(0), sarah.pop(0)
print(sarah_name + "'s fastest times are: " +

Add your dictionary

str(sorted(set([sanitize(t) for t in sarah]))[0:3]))

using and processing

code here.

you are here 4   181

dictionary data

It’s time to apply what you now know about Python’s dictionary to your code. Let’s continue to
concentrate on Sarah’s data for now. You were to strike out the code that you no longer needed
and replace it with new code that uses a dictionary to hold and process Sarah’s data.
def sanitize(time_string):
if '-' in time_string:
splitter = '-'
elif ':' in time_string:
splitter = ':'
else:
return(time_string)
(mins, secs) = time_string.split(splitter)
return(mins + '.' + secs)

def get_coach_data(filename):
try:
with open(filename) as f:
data = f.readline()
return(data.strip().split(','))
except IOError as ioerr:
print('File error: ' + str(ioerr)) You don’t need this
return(None)
code anymore.

sarah = get_coach_data('sarah2.txt')
(sarah_name, sarah_dob) = sarah.pop(0), sarah.pop(0)
print(sarah_name + "'s fastest times are: " +
str(sorted(set([sanitize(t) for t in sarah]))[0:3]))
Create an empty
dictionary. sarah_data = {}
sarah_data[‘Name’] = sarah.pop(0) Populate the dictionary wit
associating the data from thh ethfile data by
sarah_data[‘DOB’] = sarah.pop(0) the dictionary keys.. e with
sarah_data[‘Times’] = sarah
print(sarah_data[‘Name’] + “’s fastest times are: “ +
str(sorted(set([sanitize(t) for t in sarah_data[‘Times’]]))[0:3]))
Refer to the dictionary when
processing the data.

182   Chapter 6
 custom data objects

Test Drive
Let’s confirm that this new version of your code works exactly as before by testing your code within
the IDLE environment.

Your dictionary code

produces the same results
as earlier.

Which, again, works as expected…the difference being that you can now more easily
determine and control which identification data associates with which timing data,
because they are stored in a single dictionary.
Although, to be honest, it does take more code, which is a bit of a bummer. Sometimes the
extra code is worth it, and sometimes it isn’t. In this case, it most likely is.

Let’s review your code to see if we can improve anything.

you are here 4   183

code review

Head First
Code Review
It’s great to see
The Head First Code Review Team has been at it
you taking some of our
again: they’ve scribbled all over your code. Some
suggestions on board.
of their comments are confirmations; others are
Here are a few more...
suggestions. Like all code reviews, these comments
are an attempt to improve the quality of your code.
def sanitize(time_string):
if '-' in time_string:
splitter = '-'
elif ':' in time_string:
splitter = ':'
else:
return(time_string)
(mins, secs) = time_string.split(splitter)
return(mins + '.' + secs)

def get_coach_data(filename):
try:
with open(filename) as f:
ry as you go
data = f.readline()
Rather than building the dictiona
? In fact, in
along, why not do it all in one go
return(data.strip().split(','))

sense to do
this situation, it might even make
except IOError as ioerr:

oach_data()
this processing within the get_c
print('File error: ' + str(ioerr))

return a
return(None)

function and have the function

to a list.
sarah = get_coach_data('sarah2.txt')
populated dictionary as opposed
e the
Then, all you need to do is creat
ng an
sarah_data = {}
sarah_data['Name'] = sarah.pop(0) dictionary from the data file usi
sarah_data['DOB'] = sarah.pop(0)
appropriate function call, right?
sarah_data['Times'] = sarah

print(sarah_data['Name'] + "'s fastest times are: " +

str(sorted(set([sanitize(t) for t in sarah_data['Times']]))[0:3]))

You might want to consider moving this code into the get_coach_data() function, too, because
doing so
would rather nicely abstract away these processing details. But whether you do or not is up to
you. It’s
your code, after all!

184   Chapter 6
 custom data objects

Actually, those review comments are really useful. Let’s take the time to
apply them to your code. There are four suggestions that you need to
adjust your code to support:
1. Create the dictionary all in one go.

2. Move the dictionary creation code into the get_coach_data()

function, returning a dictionary as opposed to a list.
3. Move the code that determines the top three times for each athlete
into the get_coach_data() function.
4. Adjust the invocations within the main code to the new version of the
get_coach_data() function to support it’s new mode of operation.
Grab your pencil and write your new get_coach_data() function
in the space provided below. Provide the four calls that you’d make to
process the data for each of the athletes and provide four amended
print() statements:

you are here 4   185

reviews are in

You were to take the time to apply the code review comments to your
code. There were four suggestions that you needed to adjust your code
to support:
1. Create the dictionary all in one go.

2. Move the dictionary creation code into the get_coach_data()

function, returning a dictionary as opposed to a list.
3. Move the code that determines the top three times for each athlete
into the get_coach_data() function.
4. Adjust the invocations within the main code to the new version of the
get_coach_data() function to support its new mode of operation.
You were to grab your pencil and write your new get_coach_data()
function in the space provided below, as well as provide the four calls
that you’d make to process the data for each of the athletes and provide
four amended print() statements:

def get_coach_data(filename):
try:
with open(filename) as f:
data = f.readline()
1. Create a temporary templ = data.strip().split(‘,’)
list to hold the data return({‘Name’ : templ.pop(0), 2. The dictionary creation code
BEFORE creating the now part of the function. is
dictionary all in one go. ‘DOB’ : templ.pop(0),
‘Times’: str(sorted(set([sanitize(t) for t in templ]))[0:3])})
except IOError as ioerr: e
3. The code that determinesththe
print(‘File error: ‘ + str(ioerr)) top three scores is part of
function, too.
return(None)

We are showing only these tw line

4. Call the function james = get_coach_data(‘james2.txt’) code for one athlete (becauseo rep s of
for an athlete and for the other three is a trivial eat ing it
adjust the “print()” exercise).
statement as needed.
print(james[‘Name’] + “’s fastest times are: “ + james[‘Times’])

186   Chapter 6
 custom data objects

Test Drive
Let’s confirm that all of the re-factoring suggestions from the Head First Code Review Team are
working as expected. Load your code into IDLE and take it for a spin.

All of the data processing is

moved into the function.

tidied up and
This code has been considerably lete associated
now displays the name of the ath
with their times.

Looking
good!

To process additional athletes, all you need is two lines of code: the first invokes
the get_coach_data() function and the second invokes print().
And if you require additional functionality, it’s no big deal to write more
functions to provide the required functionality, is it?

you are here 4   187

associate custom code with custom data

Wait a minute...you’re using a dictionary to

keep your data all in one place, but now you’re
proposing to write a bunch of custom functions
that work on your data but aren’t associated with
it. Does that really make sense?

Keeping your code and its data together is good.

It does indeed make sense to try and associate the functions with the
data they are meant to work on, doesn’t it? After all, the functions
are only going to make sense when related to the data—that is, the
functions will be specific to the data, not general purpose. Because this
is the case, it’s a great idea to try and bundle the code with its data.
But how? Is there an easy way to associate custom code, in the form
of functions, with your custom data?

188   Chapter 6
 custom data objects

Bundle your code and its data in a class

Like the majority of other modern programming languages, Python lets you
create and define an object-oriented class that can be used to associate code
with the data that it operates on.

Why would anyone

want to do this?

Using a class helps reduce complexity.

By associating your code with the data it works on, you reduce
complexity as your code base grows.

So what’s the big

deal with that?

Reduced complexity means fewer bugs.

Reducing complexity results in fewer bugs in your code.
However, it’s a fact of life that your programs will have
functionality added over time, which will result in additional
complexity. Using classes to manage this complexity is a very
good thing.

Yeah? But...who really

cares?

Fewer bugs means more maintainable code.

Using classes lets you keep your code and your data together in
one place, and as your code base grows, this really can make
quite a difference. Especially when it’s 4 AM and you’re under a
deadline…

you are here 4   189

 get some class

Define a class
Python follows the standard object-oriented programming model of
providing a means for you to define the code and the data it works on as a
class. Once this definition is in place, you can use it to create (or instantiate)
data objects, which inherit their characteristics from your class.
Within the object-oriented world, your code is often referred to as the class’s
methods, and your data is often referred to as its attributes. Instantiated
data objects are often referred to as instances.

The “raw” data

"Sarah Sweeney","2002-6-17",[2:58,2.58,2:39,2-25,2-55,2:54,2.18,2:55,2:55,2:22,2-21,2.22]

The factory has been

primed with your class.
The Object
Factory

Here are your instantiated obj ects,

which are packaged to conta in your
code and its associated da ta .

Mikey’s object instance

Julie’s object James’s object
instance instance
Each object is created from the class and shares a similar set of
characteristics. The methods (your code) are the same in each instance, but
each object’s attributes (your data) differ because they were created from your
raw data.

Let’s look at how classes are defined in Python.

190   Chapter 6
 custom data objects

Use class to define classes

Python uses class to create objects. Every defined class has a special method
called __init__(), which allows you to control how objects are initialized.
Methods within your class are defined in much the same way as functions,
that is, using def. Here’s the basic form:

Give your class a nice,

The keyword starts descriptive name.
the definition.

class Athlete:
Don’t forget the colon!
def __init__(self):
# The code to initialize a "Athlete" object.
...

That’s a double underscore before

and after the word “init”. The code that initializes each
object goes in here.

Creating object instances

With the class in place, it’s easy to create object instances. Simply assign a call
to the class name to each of your variables. In this way, the class (together
with the __init__() method) provides a mechanism that lets you create
a custom factory function that you can use to create as many object
instances as you require:
The brackets tell Python to
create a new “Athlete” obj
which is then assigned to a ect,
a = Athlete() variable.
b = Athlete()
All of these variables are
unique and are “of type” c = Athlete()
Athlete. d = Athlete()

Unlike in C++-inspired languages, Python has no notion of defining a

constructor called “new.” Python does object contruction for you, and then
lets you customize your object’s initial state using the __init__() method.

you are here 4   191

note to self

The importance of self

To confirm: when you define a class you are, in effect, defining a custom factory
function that you can then use in your code to create instances:

a = Athlete() Invoke the class’s custom factory

The target identifier that holds function.
a reference to your instance
When Python processes this line of code, it turns the factory function call into
the following call, which identifies the class, the method (which is automatically
set to __init__()), and the object instance being operated on:

The name of the class

The target identifier
Athlete().__init__(a) of the object instance

The name of the method

Now take another look at how the __init__() method was defined in the
class:

def __init__(self):
# The code to initialize an "Athlete" object.
...

Check out what Python turns your object creation invocation into. Notice
anything?

The target identifer is assigned to the self argument.

This is a very important argument assignment. Without it, the Python interpreter
can’t work out which object instance to apply the method invocation to. Note
that the class code is designed to be shared among all of the object instances:
the methods are shared, the attributes are not. The self argument helps
identify which object instance’s data to work on.

192   Chapter 6
 custom data objects

Every method’s first argument is self

In fact, not only does the __init__() method require self as its first
argument, but so does every other method defined within your class.
Python arranges for the first argument of every method to be the invoking
(or calling) object instance. Let’s extend the sample class to store a value in a
object attribute called thing with the value set during initialization. Another
method, called how_big(), returns the length of thing due to the use of
the len() BIF:

class Athlete:
def __init__(self, value=0):
The “init” code now
assigns a supplied value self.thing = value Note the use of
to a class attribute def how_big(self): “self” to identify
called “self.thing”. the calling object
return(len(self.thing)) instance.

The “how_big()” method return

length of “self.thing”. s the

When you invoke a class method on an object instance, Python arranges for
the first argument to be the invoking object instance, which is always assigned
to each method’s self argument. This fact alone explains why self is
so important and also why self needs to be the first argument to every object
method you write:

What you write: What Python executes:

d = Athlete("Holy Grail") Athlete.__init__(d, "Holy Grail")

The target
The class The method indentifer (or
instance)

d.how_big() Athlete.how_big(d)

you are here 4   193

idle session

Let’s use IDLE to create some object instances from a new class that you’ll define. Start by creating a small class
called Athlete: No te the default values for
>>> class Athlete: two of the arguments.
def __init__(self, a_name, a_dob=None, a_times=[]):
self.name = a_name
ree att rib utes are init iali ze d and assigned to three class
Th ument data.
attributes using the supplied arg
self.dob = a_dob
self.times = a_times

With the class defined, create two unique object instances which derive their characteristcs from the Athlete
class:

>>> sarah = Athlete('Sarah Sweeney', '2002-6-17', ['2:58', '2.58', '1.56'])

>>> james = Athlete('James Jones')
Create two unique athletes ith
>>> type(sarah) “james” using the default arg(w
ument
<class '__main__.Athlete'> values).
>>> type(james) Confirm that both “sarah” and
<class '__main__.Athlete'> “james” are athletes.
Even though sarah and james are both athletes and were created by the Athlete class’s factory function,
they are stored at different memory addreses:

>>> sarah
es on our computer, which wil l
<__main__.Athlete object at 0x14d23f0> These are the memory addressed on yours. The key poi nt is
>>> james differ from the values report and “james” differ.
<__main__.Athlete object at 0x14cb7d0> the memory address for “sarah”
Now that sarah and james exist as object instances, you can use the familiar dot notation to access the
attributes associated with each:

>>> sarah.name
'Sarah Sweeney'
>>> james.name
'James Jones'
>>> sarah.dob
'2002-6-17'
The “james” object instance has
>>> james.dob
“dob”, so nothing appears on scr no value for
>>> sarah.times een.
['2:58', '2.58', '1.56']
>>> james.times
[]

194   Chapter 6
 custom data objects

Here’s your code (except for the santize() function, which doesn’t
need to change). With your pencil, write code to define the Athlete
class. In addition to the __init__() method, define a new method
called top3() that, when invoked, returns the top three times.

Be sure to adjust the get_coach_data() function to return an

Athlete object as opposed to a dictionary, and don’t forget to amend
your print() statements, too.
Write your Athlete
class code here.

def get_coach_data(filename):
try:

to ensure
with open(filename) as f:
What needs to change here hle object
this function returns an At te
data = f.readline()

as opposed to a dictionary?
templ = data.strip().split(',')
return({'Name' : templ.pop(0),
'DOB' : templ.pop(0),
'Times': str(sorted(set([sanitize(t) for t in templ]))[0:3])})
except IOError as ioerr:
print('File error: ' + str(ioerr))
return(None)
This line of code
james = get_coach_data('james2.txt')
needs to change, too.

print(james['Name'] + "'s fastest times are: " + james['Times'])

you are here 4   195

class athlete

Here’s your code (except for the santize() function, which doesn’t
need to change). With your pencil, you were to write code to define the
Athlete class. In addition to the __init__() method, you were to
define a new method called top3() that, when invoked, returns the top
three times. You were to be sure to adjust the get_coach_data()
function to return an Athlete object as opposed to a dictionary, and
you weren’t to forget to amend print(), too.

class Athlete:
def __init__(self, a_name, a_dob=None, a_times=[]):
self.name = a_name There’s nothing new here as this
code is taken straight from the
self.dob = a_dob most recent IDLE session.
self.times = a_times
Did you remember to
use “self”?
def top3(self):
return(sorted(set([sanitize(t) for t in self.times]))[0:3])

def get_coach_data(filename): Remove the dictionary creati

try: code and replace it with Athleonte
with open(filename) as f: object creation code instead.
data = f.readline()
templ = data.strip().split(',')
return({'Name' : templ.pop(0),
'DOB' : templ.pop(0), Athlete(templ.pop(0), templ.pop(0), templ)
'Times': str(sorted(set([sanitize(t) for t in templ]))[0:3])})
except IOError as ioerr:
print('File error: ' + str(ioerr))
return(None) Use the dot notation to get Invoke the “top3()”
at your data. method and convert its
james = get_coach_data('james2.txt')
results to a string prior
to its display on screen.
james.name str(james.top3())
print(james['Name'] + "'s fastest times are: " + james['Times'])

196   Chapter 6
 custom data objects

Test Drive
With these changes applied to your program, let’s ensure you continue to get the same results as
earlier. Load your code into IDLE and run it.

The code to the “sanitize()” fun

not shown here, but it is still parction is
program. t of this

Cool! There’s no
change here.

And to make objects

do more, I just add more
methods, right?

Yes, that’s correct: more functionality = more methods.

Simply add methods to encapsulate the new functionality you need within
your class. There’s no limit to how many methods a class can have, so feel
free to knock yourself out!
you are here 4   197
no dumb questions

Q: I’m not sure I see why the top3() method is coded to return Q: OK, I think I’m convinced. But tell me: how do I go about
a three-item list, as opposed to a string? Surely a string would adding more times to my existing Athlete objects?
make the print() statement in the main program easier to write?

A: It would, but it wouldn’t be as flexible. By returning a list (albeit

A: To do more, add more methods. With your Athlete class
created, it’s a breeze to extend it to do more work for you: simply add
a small one), the top3() method lets the calling code decide what more methods.
happens next, as opposed to forcing the caller to work with a string.
Granted, the current program needs to treat the list like a string, but So, if you want to add a single new timing value to your times
not all programs will want or need to. attribute, define a method called add_time() to do it for you.

Q: Why does the class even need the top3() method? Why
Additionally, you can add a list of times by defining a method called
add_times().Then all you need to do in your code is say
not store the top three times as an attribute within the class and something like this:
create it as part of the object’s creation? sarah.add_time('1.31')

A:
to add a single time to Sarah’s timing data, or say this:
Again, better not to, because doing so is less flexible. If you james.add_times(['1.21','2.22'])
compute and store the top three times at object creation, you make it to add a bunch of times to James’s data.
harder to extend the list of timing data associated with the object.

For instance, if you add more timing data after the object is created,
Q: But surely, knowing that times is a list, I could write code
like this to do the same thing?
you’ll need to arrange to recompute the top three (because the new sarah.times.append('1.31')
times might be fast) and update the attribute. However, when you james.times.append(['1.21','2.22'])
compute the top three times “on the fly” using a call to the top3()
method, you always ensure you’re using the most up-to-date data.
A: You could, but that would be a disaster.
Q: OK, I get that. But, with a little extra work, I could do it Q: What?!? Why do you say that? There’s nothing wrong
during object creation, right?
with my suggestion, is there?

A: Well, yes…but we really don’t advise that. By preserving the

original data in each object’s attributes, you are supporting the
A: Well…it does indeed work. However, the problem with writing
code like that is that it exposes (and exploits) that fact that the
extension of the class to support additional requirements in the
future (whatever they might be). If you process the data as part of timing data is stored in a list within the Athlete class. If you
the object initialization code, the assumptions you make about how later change your class implementation to use (for instance) a string
programmers will use your class might just come back to bite you. instead of a list, you may well break all of the existing code that uses
your class and that exploits the fact that the timing data is a list.
Q: But what if I’m the only programmer that’ll ever use a
By defining your own API with add_time() and add_
custom class that I write?
times(), you leave open the possibility that the way the data
A: Trust us: you’ll thank yourself for coding your class to be as
flexible as possible when you come to use it for some other purpose
is stored within your class can change in the future (obviously, only
if such a change makes sense). It is worth noting that one of the
reasons for using object orientation is to hide away the details of a
in a future project. When you are creating a class, you have no idea class’s implementation from the users of that class. Defining your
how it will be used by other programmers in their projects. And, if you own API directly supports this design ideal. Exposing the internals of
think about, you have no idea how you might use it in the future, too. your class’s implementation and expecting programmers to exploit it
breaks this fundamental ideal in a very big way.

198   Chapter 6
 custom data objects

Let’s add two methods to your class. The first, called add_time(), appends a single
additional timing value to an athlete’s timing data. The second, add_times(), extends an
athlete’s timing data with one or more timing values supplied as a list.
Here’s your current class: add the code to implement these two new methods.

class Athlete:
def __init__(self, a_name, a_dob=None, a_times=[]):
self.name = a_name
self.dob = a_dob
self.times = a_times

def top3(self):
return(sorted(set([sanitize(t) for t in self.times]))[0:3])

Add your new

methods here.

Don’t put down the pencil just yet! Provide a few lines of code to
test your new functionality:

you are here 4   199

more methods

Let’s add two methods to your class. The first, called add_time(), appends a single
additional timing value to an athlete’s timing data. The second, add_times(), extends an
athlete’s timing data with one of more timing values supplied as a list.
Here’s your current class: you were to add the code to implement these two new methods.

class Athlete:
def __init__(self, a_name, a_dob=None, a_times=[]):
self.name = a_name
self.dob = a_dob
self.times = a_times

def top3(self):
return(sorted(set([sanitize(t) for t in self.times]))[0:3])

def add_time(self, time_value):

Take the supplied argument and
self.times.append(time_value) append it to the existing list of
Don’t forget to timing values.
use “self”!!!
def add_times(self, list_of_times): Take the list of supplied argum ents
timing
self.times.extend(list_of_times) and extend the existing list of
values with them.

While still holding on firmly to your pencil, you were to provide a

few lines of code to test your new functionality:
Create a new vera = Athlete(‘Vera Vi’)
object instance
for Vera. vera.add_time(‘1.31’) Add a single timing value.
print(vera.top3()) This will display a list with only one value in it: 1.31.
vera.add_times([‘2.22’, “1-21”, ‘2:22’]) Add three more timing values.

print(vera.top3()) The top 3 timing scores are now: 1.21, 1.31 and 2.22.

200   Chapter 6
 custom data objects

Do this!

Amend your code with the updated version

of your Athlete class before proceeding
with this Test Drive.

Test Drive
After running your existing program, try out your test code in the IDLE shell to confirm that
everything is working as expected.

As expected.

Create a new athlete.

Add one timing value.
Display the top three times (there’s only one, so that’s all you see).
Add three more timing values.
Display the top three times (which, now, makes a little more sense).

Great: it worked.

You’ve packaged your code with your data and created a custom class
from which you can create objects that share behaviors. And when extra
functionality is required, add more methods to implement the required
functionality.
By encapsulating your athlete code and data within a custom class, you’ve
created a much more maintainable piece of software. You will thank
yourself for doing this when, in six months, you need to amend your code.

Well done. This is really coming along!

you are here 4   201

reinventing the wheel

Emmm...maybe I’m missing something, but

isn’t your Athlete class wasteful? I mean,
you’ve extended it with functionality that’s
already in lists, which feels a little like
reinventing the wheel to me...

Yes, your Athlete class is much like a list.

Your Athlete class does indeed behave like a list most
of the time, and you’ve added methods to expose some list
functionality to the users of your class. But it’s true: you are
reinventing the wheel here. Your add_time() method
is a thin wrapper around the list append() method and
your add_times() method is list’s extend() method in
disguise.
In fact, your Athlete class only differs from Python’s list
due to the inclusion of the name and dob object attributes.

202   Chapter 6
 custom data objects

Wouldn't it be dreamy if there were a

way to extend a built-in class with custom
attributes? But I know it's just a fantasy…

you are here 4   203

inherit class

Inherit from Python’s built-in list

Python’s class lets you create a custom class from scratch, just like you did with
your Athlete class. However, class also lets you create a class by inheriting
from any other existing class, including Python’s built-in data structure classes
that provide you with list, set, and dict. Such classes are referred to as
subclasses.
What’s really nice is that when you inherit from an existing class (such as
list), you are given all of the existing functionality for free.
As your existing class is really nothing more than a list with added attributes,
perhaps a better design is to kill off your Athlete class and replace it
with a class that inherits from the built-in list class? It’s certainly worth
considering, isn’t it?

Sorry to hear about your

Athlete class. But, according
to my files, you’re in line to inherit a
mountain of functionality from the
built-in list class. Congratulations,
you’re rich!

Slippery
lawyer-type

204   Chapter 6
 custom data objects

Tonight’s talk: Inheritance, a.k.a. He looks just like his father.

Custom Class: Inherited Class:

Programmers like me because they get to
control everything in their code…and you know
programmers: they love to code.
Yes, they do. But sometimes writing everything from
scratch is not the best design decision.
Design! Phooey! Real programmers eat, sleep,
dream, snore, and exhale code. All that design talk
is for people who can’t code!
Is it really? So, you’re saying it’s much better to do
everything from scratch and repeat the work of
others, because your way is the best way. Are you
serious?!?
No, no, no: you’re not listening. It’s all done with
control. When you build everything from the
ground up, you’re in control, as it’s all your code.
And you’re happy to reinvent the wheel, even
though someone else solved that problem eons ago?
Of course, especially when there are custom
requirements to be taken into consideration. In that
case, a brand-spanking new custom class is the only
way to go.
Not if you can extend someone else’s class to handle
your custom requirements. That way, you get the
best of both worlds: inheritied functionality (so
you’re not reinventing the wheel) together with the
custom bits. It’s a win-win situation.
Yeah, right…it’s a win-win for you, not me.
But it’s not about us: it’s to do with making the life
of the programmer easier, even the ones that live to
code, right?
I guess so, although I’m still a fan of custom code…

you are here 4   205

idle session

Let’s see what’s involved in inheriting from Python’s built-in list class. Working in IDLE’s shell, start by creating a
custom list derived from the built-in list class that also has an attribute called name:
>>> class NamedList(list): Provide the name of the class that this new
def __init__(self, a_name): class derives from.
list.__init__([])
self.name = a_name Initialize the derived from class, and then
assign the argument to the attribute.
With your NamedList class defined, use it to create an object instance, check the object’s type (using the
type() BIF), and see what it provides (using the dir() BIF):
>>> johnny = NamedList("John Paul Jones") Create a new “NamedList” object instance.
>>> type(johnny)
<class '__main__.NamedList'>
Yes, “johnny” is a “NamedList”.
>>> dir(johnny)
['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__dict__', '__doc__',
'__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__',
'__iadd__', '__imul__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__module__',
'__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__',
'__rmul__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__',
'__weakref__', 'append', 'count', 'extend', 'index', 'insert', 'name', 'pop', 'remove',
'reverse', 'sort']
“johnny” can do everything a list can, as well as
store data in the “name” attribute.
Use some of the functionality supplied by the list class to add to the data stored in johnny:

Add data to the “NamedList” using the

>>> johnny.append("Bass Player")

methods provided by the list built in.

>>> johnny.extend(['Composer', "Arranger", "Musician"])
>>> johnny
['Bass Player', 'Composer', 'Arranger', 'Musician']
>>> johnny.name
Access the list data, as well as
'John Paul Jones' the attribute data.
Because johnny is a list, it’s quite OK to do list-type things to it:

“johnny” is like any other list, so feel free to

>>> for attr in johnny:

use it wherever you’d use a list.

print(johnny.name + " is a " + attr + ".")

John Paul Jones is a Bass Player.
John Paul Jones is a Composer.
John Paul Jones is a Arranger.
John Paul Jones is a Musician.

Confirmation: John’s a busy boy. §

206   Chapter 6
 custom data objects

Here is the code for the now defunct Athlete class. In the space provided below, rewrite this
class to inherit from the built-in list class. Call your new class AthleteList. Provide a
few lines of code to exercise your new class, too:
class Athlete:
def __init__(self, a_name, a_dob=None, a_times=[]):
self.name = a_name
self.dob = a_dob
self.times = a_times

def top3(self):
return(sorted(set([sanitize(t) for t in self.times]))[0:3])

def add_time(self, time_value):

self.times.append(time_value)

def add_times(self, list_of_times):

self.times.extend(list_of_times)

Write your new

class code here.

Exercise your
code here.

you are here 4   207

new athletelist

Here is the code for the now defunct Athlete class. In the space provided below, you were
to rewrite this class to inherit from the built-in list class. You were to call your new class
AthleteList, as well as provide a few lines of code to exercise your new class:
class Athlete:
def __init__(self, a_name, a_dob=None, a_times=[]):
self.name = a_name
self.dob = a_dob
self.times = a_times

def top3(self):
return(sorted(set([sanitize(t) for t in self.times]))[0:3])

def add_time(self, time_value):

self.times.append(time_value) These
methods
def add_times(self, list_of_times): aren’t needed
anymore.
The class name self.times.extend(list_of_times)
has changed.
Inherit from the built-in list class.
class AthleteList(list):
def __init__(self, a_name, a_dob=None, a_times=[]):
Nothing new here… list.__init__([])
this code is very The data
similar to the self.name = a_name
itself is the
“NamedList” init
code. self.dob = a_dob timing data,
so the “times”
self.extend(a_times) attribute is
def top3(self): gone.
return(sorted(set([sanitize(t) for t in self]))[0:3])
Use the new
class’s name.
vera = AthleteList(‘Vera Vi’)
vera.append(‘1.31’)
Now that you’re This code does a good job of
inheriting from print(vera.top3()) exercising your new class.
the built-in list,
you can use its vera.extend([‘2.22’, “1-21”, ‘2:22’])
methods to get print(vera.top3())
your work on.

208   Chapter 6
 custom data objects

Do this!

In your code, replace your Athlete

class code with your new AthleteList
class code, and don’t forget to change
get_coach_data() to return an
AthleteList object instance as
opposed to an Athlete object instance.

Q: Sorry…but not three minutes ago you were telling me not Q: Can I inherit from my own custom classes?

A:
to expose the inner workings of my class to its users, because
that was fundamentally a bad idea. Now you’re doing the exact Of course, that’s the whole idea. You create a generic class
opposite! What gives? that can then be “subclassed” to provide more specific, targeted

A: functionality.

Q:
Well spotted. In this particular case, it’s OK to expose the fact
that the class is built on top of list. This is due to the fact that the Can I put my class in a module file?
class is deliberately called AthleteList to distinguish it from
the more generic Athlete class. When programmers see the
word “list” in a class name, they are likely to expect the class to work
A: Yes, that’s a really good idea, because it lets you share your
class with many of your own programs and with other programmers.
like a list and then some. This is the case with AthleteList. For instance, if you save your AthleteList class to a file

Q: And I can inherit from any of the built-in types?

called athletelist.py, you can import the into your code
using this line of code:

A: Yes. from athletelist import AthleteList

Q: What about inheriting from more than one class…does then use the class as if it was defined in your current program. And,
of course, if you create a really useful class, pop it into its own
Python support multiple interitance?
module and upload it to PyPI for the whole world to share.
A: Yes, but it’s kind of scary. Refer to a good Python reference text
for all the gory details.

you are here 4   209

test drive

Test Drive
One last run of your program should confirm that it’s working to specification now. Give it a go in
IDLE to confirm.

Your entire
program now
produces the
output the
coach wants.

210   Chapter 6
 custom data objects

Coach Kelly is impressed

That looks great! I can’t

wait to show this to my
young athletes and see
their reaction...

By basing your class on built-in functionality, you’ve leveraged the power of

Python’s data structures while providing the custom solution your application
needs.
You’ve engineered a much more maintainable solution to Coach Kelly’s data
processing needs.

Good job!

you are here 4   211

 python toolbox

Your Python Toolbox

You’ve got Chapter 6 under your

CHAPTER 6
belt and you’ve added some key
CHAPTER 6

Python techiques to your toolbox. ƒƒ Create a empty dictionary using the

dict() factory function or using {}.
ƒƒ To access the value associated with
the key Name in a dictionary called

g o
person, use the familiar square bracket

Py th on L i n notation: person['Name'].
ata
a built-in d
ionary” - ows you to
ƒƒ Like list and set, a Python’s dictionary
• “Dict
dynamically grows as new data is added
hat all
structure tata values with keys.
to the data structure.

associate d of
k-up part
ƒƒ Populate a dictionary as you go:
- t h e lo o new_d = {} or new_d = dict()
• “Key”
ary. and then
the diction a p art of the , d['Name'] = 'Eric Idle'
h e d at e
• “Value
”-t c a n be any valu .
or do the same thing all in the one go:
( w h ic h )
dictionary other data structure
new_d = {'Name': 'Eric

including an
Idle'}
ƒƒ The class keyword lets you define a
class.

More Python Lingo ƒƒ Class methods (your code) are defined in

much the same way as functions, that is,
with the def keyword.
• “self” - a method
argument
that always refers to the curren ƒƒ Class attributes (your data) are just like
object instance. t variables that exist within object instances.
ƒƒ The __init__() method can be
defined within a class to initialize object
instances.
ƒƒ Every method defined in a class must
provide self as its first argument.
ƒƒ Every attribute in a class must be prefixed
with self. in order to associate it data
with its instance.
ƒƒ Classes can be built from scratch or can
inherit from Python’s built-in classes or
from other custom classes.
ƒƒ Classes can be put into a Python module
and uploaded to PyPI.

212   Chapter 6
 7 web development

Putting it all together

This Web thing will
never catch on...especially
now that I have my trusty
Underwood to keep me
company...

Sooner or later, you’ll want to share your app with lots of people. 
You have many options for doing this. Pop your code on PyPI, send out lots of emails, put
your code on a CD or USB, or simply install your app manually on the computers of those
people who need it. Sounds like a lot of work…not to mention boring. Also, what happens
when you produce the next best version of your code? What happens then? How do you
manage the update? Let’s face it: it’s such a pain that you’ll think up really creative excuses
not to. Luckily, you don’t have to do any of this: just create a webapp instead. And, as this
chapter demonstrates, using Python for web development is a breeze.

this is a new chapter   213

caring is sharing

It’s good to share

The coach showed us your program

running on his laptop...any chance me and my
friends could also get access to our list of
times? I’d love to show them to my dad...

Coach Kelly’s
young athletes

You’re a victim of your own success.

The new requests come flooding in right after Coach Kelly starts showing
off your latest program. It appears that everyone wants access to the coach’s
data!

The thing is: what’s the “best way” to do this?

214   Chapter 7
 web development

You can put your program on the Web

...but you probably want

only one version of your
program “out there” that ...and you need to make sure
You’ll want to be everyone accesses... updates to your program
able to share your are easy to apply.
functionality with lots
of people...

A “webapp” is what you want.

If you develop your program as a Web-based application (or webapp, for short),
your program is:
• Available to everyone who can get to your website
• In one place on your web server
• Easy to upate as new functionality is needed

But…how do webapps actually work?

you are here 4   215

anatomy of a web request

Webapps Up Close
No matter what you do on the Web, it’s all about requests and responses. A web request is sent
from a web browser to a web server as the result of some user interaction. On the web server, a
web response (or reply) is formulated and sent back to the web browser. The entire process can
be summarized in five steps.

Step 2: The web

I just type the browser converts
web address into my the user’s action
into a web request
browser’s location bar
and sends it to a
Step 1: Your user enters and press Enter... server over the
a web address, selects
Internet.
a hyperlink, or clicks a
button in her chosen
web browser.

The Internet

a web
Here comes
request.

Hey, hello there...what’s this?

A web request just for me? How
nice...

Deciding what to do next

One of two things happen at this point. If the web request
is for static content—such as an HTML file, image, or
Web anything else stored on the web server’s hard disk—the web
Server server locates the resource and returns it to the web browser as
Step 3: The web server a web response.
receives the web request
and has to decide what If the request is for dynamic content—that is, content that
to do next. must be generated—the web server runs a program to produce
the web response.

216   Chapter 7
 web development

Here you go...a web

Step 4: The web server response generated just for
processes the web you. Enjoy!
request, creating a web
response, which is sent
back over the Internet Here co
to the waiting web response.mes a web
browser.

Web
Server

The Internet

The (potentially) many substeps of step 4

In practice, step 4 can involve multiple substeps, depending
on what the web server has to do to produce the response.
Obviously, if all the server has to do is locate static content
and sent it back to the browser, the substeps aren’t too
taxing, because it’s all just file I/O. That’s exactly what I
need. Thanks!
However, when dynamic content must be generated, the sub­
steps involve the web server locating the program to execute,
executing the located program, and then capturing the output
from the program as the web response…which is then sent
back to the waiting web browser.
This dynamic content generation process has been
standardized since the early days of the Web and is known
as the Common Gateway Interface (CGI). Programs
that conform to the standard are often referred to as CGI
scripts.
Step 5: The web
browser receives the
web response and
displays it on your
user’s screen.

you are here 4   217

webapp requirements

What does your webapp need to do?

Let’s take a moment to consider what you want your webapp to look like
and how it should behave on your user’s web browser. You can then use this
information to help you specify what your webapp needs to do.

I guess I need a nice,

friendly home page to
kick things off, eh?

...and once I’ve

Yeah...and I want to be selected mine, I want
able to get at my times them to look nice on
easily... my screen, so I can print
them for my mom.

218   Chapter 7
 web development

There’s nothing like grabbing your pencil and a few blank paper
napkins to quickly sketch a simple web design. You probably
need three web pages: a “welcome” page, a “select an athlete”
page, and a “display times” page. Go ahead and draw out a rough
design on the napkins on this page, and don’t forget to draw any
linkages between the pages (where it makes sense).

you are here 4   219

back-of-the-napkin sketch

§ There’s nothing like grabbing your pencil and a few blank paper
napkins to quickly sketch a simple web design. You probably
need three web pages: a “welcome” page, a “select an athlete”
The home page page, and a “display times” page. You were to draw out a rough
displays a friendly design on the napkins. You were to draw any linkages between
graphic and a link to the pages (where it made sense).
start the web app.
go to
Click on the home page’s linkoftoall the
a page that displays a list athlete’s
coach’s athletes. Click on an“Select”
radio button and then the
button to see the data.

Select an athlet
e from this list to
work with:
e to Co a ch K el ly’s Website Sarah
Welcom
w , al l t h at yo u’ ll find here is my James
For n o
data.
athlete’s timing Julie
rack!
See you on the t Mikey

Select

Timimg data for Sarah:

2.18 The third web page

displays the selected
2.21 athlete’s data and
provides links back to
2.22 the other two pages.

Home Select another athlete.

220   Chapter 7
 web development

Design your webapp with MVC

Now that you have an idea of the pages your webapp needs to provide, your
next question should be: what’s the best way to build this thing?
Ask 10 web developers that question and you’ll get 10 different answers; the
answer often depends on whom you ask.
Despite this, the general consensus is that great webapps conform to the
Model-View-Controller pattern, which helps you segment your webapp’s code
into easily manageable functional chunks (or components):

The Model
The code to store (and sometimes process) your webapp’s data

The View
The code to format and display your webapp’s user interface(s)

The Controller
The code to glue your webapp together and provide its business logic

By following the MVC pattern, you build your webapp in such as way as to
enable your webapp to grow as new requirements dictate. You also open up the
possibility of splitting the workload among a number of people, one for each
component.

Let’s build each of the MVC components for your webapp.

you are here 4   221

 build a model

Model your data

Your web server needs to store a single copy of your data, which in this case is
Coach Kelly’s timing values (which start out in his text files).
When your webapp starts, the data in the text files needs to be converted
to AthleteList object instances, stored within a dictionary (indexed by
athlete name), and then saved as a pickle file. Let’s put this functionality in a
new function called put_to_store().
While your webapp runs, the data in the pickle needs to be available to your
webapp as a dictionary. Let’s put this functionality in another new function
called get_from_store().

When your webapp starts: While your webapp runs:

julie.txt
The single pickle
james.txt with all of the
coach’s data
[‘Is this the right room
for an argument?’, “No
you haven’t!”, ‘When?’,
“No you didn’t!”, “You

mikey.txt
didn’t!”, ‘You did not!’,

stored in a
‘Ah! (taking out his wallet
and paying) Just the five
minutes.’, ‘You most
certainly did not!’, “Oh
no you didn’t!”, “Oh no
you didn’t!”, “Oh look,
this isn’t an argument!”,

dictionary
“No it isn’t!”, “It’s
just contradiction!”,
‘It IS!’, ‘You just
contradicted me!’, ‘You
DID!’, ‘You did just
then!’, ‘(exasperated)
Oh, this is futile!!’,
‘Yes it is!’]

sarah.txt

The
get_from_store()
The function
put_to_store()
function

[‘Is this the right room

for an argument?’, “No
{'Sarah': AthleteList... ,
you haven’t!”, ‘When?’,
“No you didn’t!”, “You

'James': AthleteList... ,
didn’t!”, ‘You did not!’,
‘Ah! (taking out his wallet
and paying) Just the five
minutes.’, ‘You most
certainly did not!’, “Oh
no you didn’t!”, “Oh no
you didn’t!”, “Oh look,
this isn’t an argument!”,
“No it isn’t!”, “It’s

'Julie': AthleteList... ,
just contradiction!”,
‘It IS!’, ‘You just
contradicted me!’, ‘You
DID!’, ‘You did just
then!’, ‘(exasperated)
Oh, this is futile!!’,
‘Yes it is!’]

'Mikey': AthleteList... }
The single pickle with
all of the data stored s returned
in a dictionary A dictionary of AthleteList)” function
from the “get_from_sto re(

222   Chapter 7
 web development

Here is the outline for a new module called athletemodel.py, which provides the
functionality described on the previous page. Some of the code is already provided for you. Your
job is to provide the rest of the code to the put_to_store() and get_from_store()
functions. Don’t forget to protect any file I/O calls.
import pickle
from athletelist import AthleteList

def get_coach_data(filename):
# Not shown here as it has not changed since the last chapter.

def put_to_store(files_list): This function is called with a

all_athletes = {} list of filenames as its sole
You need code in argument.
here to populate
the dictionary with
the data from the
files.

And don’t forget

to save the
dictionary to a
pickle (and check
for file I/O errors).

return(all_athletes)

def get_from_store():
all_athletes = {} Both functions
need to return
a dictionary of
Get the dictionary AthleteLists.
from the file, so
that it can be
returned to the
caller.

return(all_athletes)

you are here 4   223

model module

Here is the outline for a new module called athletemodel.py, which provides the
functionality described on the previous page. Some of the code is already provided for you. Your
job was to provide the rest of the code to the put_to_store() and get_from_store()
functions. You were not to forget to protect any file I/O calls.
import pickle
from athletelist import AthleteList

def get_coach_data(filename):
# Not shown here as it has not changed since the last chapter.

def put_to_store(files_list):
all_athletes = {}

Take each file, turn it for each_file in files_list:

Each athlete’s name is
into an AthleteList ath = get_coach_data(each_file) used as the “key” in the
object instance, and dictionary. The “value” is
add the athlete’s data all_athletes[ath.name] = ath the AthleteList object
to the dictionary. instance.
try:
Save the entire with open(‘athletes.pickle', ‘wb') as athf:
dictionary of pickle.dump(all_athletes, athf)
AthleteLists And don’t forget
to a pickle. a try/except to
except IOError as ioerr: protect your file
print(‘File error (put_and_store): ' + str(ioerr)) I/O code.
return(all_athletes)

def get_from_store():
all_athletes = {}

try:
Simply read the with open(‘athletes.pickle', ‘rb') as athf:
entire pickle into
the dictionary. all_athletes = pickle.load(athf)
What could be Again…don’t
easier? except IOError as ioerr: forget your
try/except.
print(‘File error (get_from_store): ' + str(ioerr))
return(all_athletes)

224   Chapter 7
 web development

Let’s test your code to ensure that it is working to specification. Type your code into an IDLE edit window and save
your code into a folder that also includes the coach’s text files. Press F5 to import your code to the IDLE shell, and
then use the dir() command to confirm that the import has been successful:
>>> dir()
['AthleteList', '__builtins__', '__doc__', '__name__', '__package__', 'get_coach_data’,
'get_from_store', 'pickle', 'put_to_store']

Create a list of files to work with, and then call the put_to_store() function to take the data in the list of files
and turn them into a dictionary stored in a pickle:

>>> the_files = ['sarah.txt', 'james.txt', 'mikey.txt', 'julie.txt'] Here’s all of the

>>> data = put_to_store(the_files)
AthleteLists.
>>> data
{'James Lee': ['2-34', '3:21', '2.34', '2.45', '3.01', '2:01', '2:01', '3:10', '2-22', '2-
01', '2.01', '2:16'], 'Sarah Sweeney': ['2:58', '2.58', '2:39', '2-25', '2-55', '2:54', '2.18',
'2:55', '2:55', '2:22', '2-21', '2.22'], 'Julie Jones': ['2.59', '2.11', '2:11', '2:23', '3-
10', '2-23', '3:10', '3.21', '3-21', '3.01', '3.02', '2:59'], 'Mikey McManus': ['2:22', '3.01',
'3:01', '3.02', '3:02', '3.02', '3:22', '2.49', '2:38', '2:40', '2.22', '2-31']}
At this point, the athletes.pickle file should appear in the same folder as your code and text files. Recall
that this file is a binary file, so trying to view it in IDLE or in your editor is not going to make much sense. To access
the data, use the dictionary returned by the put_to_store() or get_from_store() functions.

Use the existing data in the data dictionary to display each athlete’s name and date of birth:
>>> for each_athlete in data:
print(data[each_athlete].name + ' ' + data[each_athlete].dob)

b”
By accessing the “name” andth“do
James Lee 2002-3-14

e rest of
attributes, you can get at
Sarah Sweeney 2002-6-17

the AthleteList data.

Julie Jones 2002-8-17
Mikey McManus 2002-2-24

Use the get_from_store() function to load the pickled data into another dictionary, then confirm that the
results are as expected by repeating the code to display each athlete’s name and date of birth:

>>> data_copy = get_from_store()

>>> for each_athlete in data_copy:
print(data_copy[each_athlete].name + ' ' + data_copy[each_athlete].dob)

James Lee 2002-3-14

The data in the returned dictio
Sarah Sweeney 2002-6-17 is as expected, exactly the samenary
Julie Jones 2002-8-17 that produced by put_to_store( as
Mikey McManus 2002-2-24
).

you are here 4   225

interface view

View your interface

With your model code written and working, it’s time to look at your view code,
which creates your webapp’s user interface (UI).
On the Web, UIs are created with HTML, the Web’s markup technology. If
you are new to HTML, it is worth taking some time to become familiar with
this critical web development technology. There’s lots of material on the Web
and more than a few good books out there.

[Note from Marketing:

This is the book that we
recommend for quickly
getting up to speed with
HTML…not that we’re
biased or anything. § ].

(Most of) the Head First

Code Review Team
Hey, we hear you are getting into
web development? We have a small
module that we put together that might help
you generate HTML. It’s a little rough, but it
works. You’re more than welcome to use it for
your projects, if you like.

YATE: Yet Another Template Engine

Your friends over at the Head First Code Review Team heard you’re planning
to write some code to generate HTML for your webapp’s UI. They’ve sent
over some code that they swear will make your life easier. It’s a small library
of HTML-generating helper functions called yate. The code was produced
quickly and was originally designed to be “throw away,” so the team has
provided it as is. It’s somewhat raw, but it should be OK.

226   Chapter 7
 web development

from string import Template

def start_response(resp="text/html"):
return('Content-type: ' + resp + '\n\n')

def include_header(the_title):

There’s not much help here, just

with open('templates/header.html') as headf:

code. No comments, explanations the

head_text = headf.read()

documentation, or anything! ,
header = Template(head_text)
return(header.substitute(title=the_title))

def include_footer(the_links):
with open('templates/footer.html') as footf:
foot_text = footf.read()
link_string = ''
for key in the_links:
link_string += '<a href="' + the_links[key] + '">' + key + '</a>&nbsp;&nbsp;&nbsp;&nbsp;'
footer = Template(foot_text)
return(footer.substitute(links=link_string))

def start_form(the_url, form_type="POST"):

return('<form action="' + the_url + '" method="' + form_type + '">')

def end_form(submit_msg="Submit"):
return('<p></p><input type=submit value="' + submit_msg + '">')

def radio_button(rb_name, rb_value):

return('<input type="radio" name="' + rb_name +
'" value="' + rb_value + '"> ' + rb_value + '<br />')

def u_list(items):
u_string = '<ul>'
for item in items:
u_string += '<li>' + item + '</li>'
u_string += '</ul>'
return(u_string)

def header(header_text, header_level=2):

return('<h' + str(header_level) + '>' + header_text +
'</h' + str(header_level) + '>')

def para(para_text):
return('<p>' + para_text + '</p>')

you are here 4   227

template engine code

Let’s get to know the yate code before proceeding with the rest of this chapter. For each chunk
of code presented, provide a written description of what you think it does in the spaces provided:

from string import Template Write your

Take a momentem to explanations in
plate” the spaces.
look up the “Thon’s
module in Pyt set.
documentation

def start_response(resp="text/html"):
return('Content-type: ' + resp + '\n\n')
One has already This function takes a single (optional) string as its argument and uses it to
been done for you.
create a CGI “Content-type:” line, with “text/html” as the default.
def include_header(the_title):
with open('templates/header.html') as headf:
head_text = headf.read()
header = Template(head_text)
return(header.substitute(title=the_title))

def include_footer(the_links):
with open('templates/footer.html') as footf:
foot_text = footf.read()
link_string = ''
for key in the_links:
link_string += '<a href="' + the_links[key] + '">' + key +
'</a>&nbsp;&nbsp;&nbsp;&nbsp;'
footer = Template(foot_text)
return(footer.substitute(links=link_string))

228   Chapter 7
 web development

def start_form(the_url, form_type="POST"):

return('<form action="' + the_url + '" method="' + form_type + '">')

def end_form(submit_msg="Submit"):
return('<p></p><input type=submit value="' + submit_msg + '"></form>')

def radio_button(rb_name, rb_value):

return('<input type="radio" name="' + rb_name +
'" value="' + rb_value + '"> ' + rb_value + '<br />')

def u_list(items):
u_string = '<ul>'
for item in items:
u_string += '<li>' + item + '</li>'
u_string += '</ul>'
return(u_string)

def header(header_text, header_level=2):

return('<h' + str(header_level) + '>' + header_text +
'</h' + str(header_level) + '>')

def para(para_text):
return('<p>' + para_text + '</p>')

you are here 4   229

template engine described

Let’s get to know the yate code before proceeding with the rest of this chapter. For each chunk
of code presented, you were to provide a written description of what you think it does:

from string import Template

Import the “Template” class from the standard library’s “string”

Note the default module. This allows for simple string-substitution templates.
for “resp”.
def start_response(resp="text/html"):
return('Content-type: ' + resp + '\n\n')

This function takes a single (optional) string as its argument and uses it to
create a CGI “Content-type:” line, with “text/html” as the default.
Open the def include_header(the_title):

template file with open('templates/header.html') as headf:

(which is HTML), head_text = headf.read()
read it in, and header = Template(head_text)
substitute in the
provided “title”.
return(header.substitute(title=the_title))

This function takes a single string as its argument and uses at the title for
the start of a HTML page. The page itself is stored within a separate file
in “templates/header.html”, and the title is substituted in as needed.
Open the template
file (which is
HTML), read it in, def include_footer(the_links):

and substitute in the with open('templates/footer.html') as footf:

provided dictionary foot_text = footf.read()

of HTML links in link_string = ''
“the_links”. for key in the_links:
This looks a little
link_string += '<a href="' + the_links[key] + '">' + key + weird, but it’s an
'</a>&nbsp;&nbsp;&nbsp;&nbsp;'
HTML hack for
Turn the
footer = Template(foot_text)
forcing spaces
dictionary of
return(footer.substitute(links=link_string)) into a string.
links into a string, Similar to the “include_header” function, this one uses its single string as
which is then its argument to create the end of a HTML page. The page itself is stored
substituted into
the template. within a separate file in “templates/footer.html”, and the argument is used
to dynamically create a set of HTML link tags. Based on how they are used,
it looks like the argument needs to be a dictionary.

230   Chapter 7
 web development

This is typically either

“POST” or “GET”.
def start_form(the_url, form_type="POST"):
return('<form action="' + the_url + '" method="' + form_type + '">')

This function returns the HTML for the start of a form and lets the caller
specify the URL to send the form’s data to, as well as the method to use.
def end_form(submit_msg="Submit"):
return('<p></p><input type=submit value="' + submit_msg + '"></form>')

This function returns the HTML markup, which terminates the form while
allowing the caller to customize the text of the form’s “submit” button.
def radio_button(rb_name, rb_value):
return('<input type="radio" name="' + rb_name +
'" value="' + rb_value + '"> ' + rb_value + '<br />')

Given a radio-button name and value, create a HTML radio button (which is
typically included within a HTML form). Note: both arguments are required.
def u_list(items):
u_string = '<ul>'

A simple “for” for item in items:

loop does the u_string += '<li>' + item + '</li>'
trick. u_string += '</ul>'
return(u_string)

Given a list of items, this function turns the list into a HTML unnumbered
list. A simple “for” loop does all the work, adding a LI to the UL element
with each iteration.
def header(header_text, header_level=2):
return('<h' + str(header_level) + '>' + header_text +
'</h' + str(header_level) + '>')

Create and return a HTML header tag (H1, H2, H2, and so on) with level 2
as the default.. The “header_text” argument is required.
def para(para_text):
return('<p>' + para_text + '</p>')

Enclose a paragraph of text (a string) in HTML paragraph tags. Almost not

worth the effort, is it?

you are here 4   231

no dumb questions

Q: Where are the HTML templates used in the include_ Q: And you did this because you are using MVC?

A:
header() and include_footer() functions?

A: Partly, yes. The reason the MVC pattern is being followed is to

They are included with the yate module’s download. Go ensure that the model code is separate from the view code, which
ahead and grab them from the Head First Python support website, are both separate from the controller code. No matter the size of the
and put them into a folder of your choice. project, following MVC can make your life easier.

Q: Why do I need yate at all? Why not include the HTML that I Q: But surely MVC is overkill for something this small?

A:
need right in the code and generate it with print() as needed?

A: We don’t think so, because you can bet that your webapp will
You could, but it’s not as flexible as the approach shown grow, and when you need to add more features, the MVC “separation
here. And (speaking from bitter experience) using a collection of of duties” really shines.
print() statements to generate HTML works, but it turns your
code into an unholy mess.

Let’s get to know the yate module even more. With the code downloaded and tucked away in an easy-to-
find folder, load the module into IDLE and press F5 to take it for a spin. Let’s start by testing the start_
response() function. The CGI standard states that every web response must start with a header line that
indictes the type of the data included in the request, which start_response() lets you control:
>>> start_response() The default CGI response heade
'Content-type: text/html\n\n' plus variations on a theme. r,
>>> start_response("text/plain")
'Content-type: text/plain\n\n'
>>> start_response("application/json")
'Content-type: application/json\n\n'

The include_header() function generates the start of a web page and let’s you customizee its title:
>>> include_header("Welcome to my home on the web!")
'<html>\n<head>\n<title>Welcome to my home on the web!</title>\n<link type="text/css"
rel="stylesheet" href="/coach.css" />\n</head>\n<body>\n<h1>Welcome to my home on the web!</
h1>\n'

, but don’t worry; it’s meantb

This all looks a little bit mebssybrowser, NOT by you. Your we
to be processed by your we y working with this HTML. Note
browser will have no difficult file (more on this in a bit).
the inclusion of a link to a CSS

232   Chapter 7
 web development

The include_footer() function produces HTML that terminates a web page, providing links (if provided as a
dictionary). An empty dictionary switches off the inclusion of the linking HTML:

>>> include_footer({'Home': '/index.html', 'Select': '/cgi-bin/select.py'})

'<p>\n<a href="/index.html">Home</a>&nbsp;&nbsp;&nbsp;&nbsp;<a href="/cgi-bin/select.
py">Select</a>&nbsp;&nbsp;&nbsp;&nbsp;\n</p>\n</body>\n</html>\n'
>>> include_footer({})
'<p>\n\n</p>\n</body>\n</html>\n' With links included, and
without.
The start_form() and end_form() functions bookend a HTML form, with the parameter (if supplied)
adjusting the contents of the generated HTML:

>>> start_form("/cgi-bin/process-athlete.py")
The argument allows you to specif
'<form action="/cgi-bin/process-athlete.py" method="POST">' the name of the program on the y
>>> end_form() server to send the form’s data
'<p></p><input type=submit value="Submit"></form>'
to.
>>> end_form("Click to Confirm Your Order")
'<p></p><input type=submit value="Click to Confirm Your Order"></form>'

HTML radio buttons are easy to create with the radio_button() function:
>>> for fab in ['John', 'Paul', 'George', 'Ringo']:

Which one is your favorite? io

radio_button(fab, fab)

Select from the list of rad

'<input type="radio" name="John" value="John"> John<br />' buttons.
'<input type="radio" name="Paul" value="Paul"> Paul<br />'
'<input type="radio" name="George" value="George"> George<br />'
'<input type="radio" name="Ringo" value="Ringo"> Ringo<br />'

Unordered list are a breeze with the u_list() function: Again, not too easy on your eye,
u_list(['Life of Brian', 'Holy Grail']) fine as far as your web browse but
'<ul><li>Life of Brian</li><li>Holy Grail</li></ul>' concerned. r is
The header() function lets you quickly format HTML headings at a selected level (with 2 as the default):
>>> header("Welcome to my home on the web")
'<h2>Welcome to my home on the web</h2>'
Nothing too exciting here, but it works
as expected. Same goes for here.
>>> header("This is a sub-sub-sub-sub heading", 5)
'<h5>This is a sub-sub-sub-sub heading</h5>'

Last, but not least, the para() function encloses a chunk of text within HTML paragraph tags:
>>> para("Was it worth the wait? We hope it was...")
'<p>Was it worth the wait? We hope it was...</p>'

you are here 4   233

controller code

Control your code

Your model code is ready, and you have a good idea of how the yate
module can help you with your view code. It’s time to glue it all together
with some controller code.
First things first: you need to arrange your wedapp’s directory structure to
help keep things organized. To be honest, anything goes here, although by
giving it a little thought, you can enhance your ability to extend your webapp
over time. Here’s one folder structure that Head First Labs recommends.

folders, this folder

As well as containing the subx.h l” file, your
You can call your top-level contains your webapps “inde , tm and anything
folder anything you like.
webapp “favicon.ico” icon, style sheetsinto one of the
else that doesn’t fit neatly
subfolders.

Any code that you write for

your webapp needs to reside in a
cgi-bin specially named folder called “cg
bin”. i-

Let’s keep the coach’s

data files in a separate
data folder by putting all of
the TXT files in here.

If your webapp has any ima fil

(JPGs, GIFs, PNGs, and so ges
on) ,
es
Do this! images them into their own folder to helpop
keep things organized. p

Head on over to the

Head First Python support The templates that
website, download templates came with the “yate.py”
webapp.zip, and download can go in here.
unpack it to your hard
disk.

234   Chapter 7
 web development

CGI lets your web server run programs

I’m all fired up and
The Common Gateway Interface (CGI) is an Internet standard that allows for ready to go! I live to
a web server to run a server-side program, known as a CGI script. serve-up HTML and
Typically, CGI scripts are placed inside a special folder called cgi-bin, so run CGIs...
that the web server knows where to find them. On some operating systems
(most notably UNIX-styled systems), CGI scripts must be set to executable
before the web server can execute them when responding to a web request.

CGI
Web
So...to run my webapp,
More on this in
Server
I need a web server with
a little bit. CGI enabled.

All webapps need to run on web servers.

Practically every web server on the planet supports CGI. Whether
your running Apache, IIS, nginx, Lighttpd, or any of the others, they
all support running CGI scripts written in Python.
But using one of these tools here is overkill. There’s no way the
coach is going to agree to download, unpack, install, configure,
and manage one of these industry heavyweights.
As luck would have it, Python comes with its very own web server,
included in the http.server library module. Check the
contents of the webapp.zip download: it comes with a CGI-
enabled web server called simplehttpd.py.

Here are the five lines of cod

Import the HT build a web server in Python. e needed to
server and CGI TP
modules.
from http.server import HTTPServer, CGIHTTPRequestHandler
Specify a port.
Create a port = 8080
HTTP server. httpd = HTTPServer(('', port), CGIHTTPRequestHandler)
print("Starting simple_httpd on port: " + str(httpd.server_port))
ly
Display a friendart
message an d st httpd.serve_forever()

your server.

you are here 4   235

generate list

Display the list of athletes

Let’s create a program called generate_list.py which, when executed
by the web server, dynamically generates a HTML web page that looks
something like this:

It wouldn’t hurt to
add a title to this
web page, would it?
This is a paragraph.

Select an athlete from this list

to work with:
Sarah
There’s one radio
button for each James
athlete. All of this is
Julie contained within an
HTML form.
Mikey

Select

A “submit” button

When your user selects an athlete by clicking on her radio button and clicking
Select, a new web request is sent to the web server. This new web request
contains data about which radio button was pressed, as well as the name of a CGI
script to send the form’s data to.
Recall that all of your CGI scripts need to reside in the cgi-bin folder on
your web server. With this in mind, let’s make sure your generate_list.py
CGI script sends its data to another program called:
cgi-bin/generate_timing_data.py

236   Chapter 7
 web development

Pool Puzzle
Your job is to take the code from the pool and
place them into the blank lines in the CGI
script. You may not use the same line
of code more than once. Your goal is to
make a CGI script that will generate a
HTML page that matches the hand-drawn
design from the previous page.

Import the modules that you nee

already met “athletemodel” and d. You’ve
I’ve started import athletemodel
“ya
things off The “glob” module lets you query te”.
operating system for a list of fil your
import yate
for you. import glob e names.
ction
Use your “put_to_store()” funlet
data_files = glob.glob("data/*.txt")
of ath es
to create a dictionary
from the list of data files.
athletes = athletemodel.put_to_store(data_files)

Let’s add a link to

the bottom of the
generated HTML
page that takes
your user home.
print(yate.include_footer({"Home": "/index.html"}))

Note: each thing from

the pool can be used
once!
print(yat
e.start_f
print(yat orm("gene
e.para("S rate_timi
elect an ng_data.p
print(yat athlete f y"))
print(yat e.include rom the list
e.radio_b _header(" to work w
utton("wh Coach Kel ith:"))
ich_athle ly's List
te", athl of Athlet
for each_athlete in athletes: etes[each es"))
se ()) pr _athlete]
espon int(yate.e .name))
e.start_r nd_form("
print(yat Select"))

you are here 4   237

 cgi script

Pool Puzzle Solution

Your job was to take the code from the pool
and place them into the blank lines in the
CGI script. You were not to use the same
line of code more than once. Your goal
was to make a CGI script that generates a
HTML page that matches the hand-drawn
design.

import athletemodel
import yate
import glob
Start generating the
web page, providing an
Always start with a
data_files = glob.glob("data/*.txt")
appropriate title.
Content-type line.
athletes = athletemodel.put_to_store(data_files)

Start generating print(yate.start_response())

the form, A paragraph telling
your user what to
print(yate.include_header("Coach Kelly's List of Athletes"))
providing the name print(yate.start_form("generate_timing_data.py"))
of the server- do
print(yate.para("Select an athlete from the list to work with:"))
side program to
link to.
for each_athlete in athletes:

Generate a radio-
print(yate.radio_button("which_athlete", athletes[each_athlete].name))
End the form generation with a
button for each of custom
print(yate.end_form("Select"))
“Submit” button.
your athletes. print(yate.include_footer({"Home": "/index.html"}))

Cool…an
empty pool.
238   Chapter 7
 web development

What you need to do next depends on the operating

system you’re running your web server on.
If you are running on Windows, stop reading right now and
proceed to the Test Drive. However, if you are running a Unix-
based system (such as Linux, Mac OS X, or BSD) you need to
do two things to prepare your CGI script for execution:
1. Set the executable bit for your CGI using the chmod +x command.
2. Add the following line of code to the very top of your program:
#! /usr/local/bin/python3

From your terminal win_ldow, type

ge ne ra te ist.
chmod +x
py to set the exe cut abl e bit. You
need do this only once.

Test Drive
To test drive your CGI script, you need to have a web server up and running. The code to
simplehttpd.py is included as part of the webapp.zip download. After you unpack the ZIP
file, open a terminal window in the webapp folder and start your web server:

File Edit Window Help WebServerOnWindows Use this command

c:\Python31\python.exe simplehttpd.py on Windows-based
Starting simple_httpd on port: 8080 systems.
File Edit Window Help WebServerOnUNIX
$ python3 simple_httpd.py
Starting simple_httpd on port: 8080

Use this command

on Unix-based
systems.
you are here 4   239
test drive

Test Drive, continued

With your web server running, let’s load up Coach Kelly’s home page and get things going. You’ve
started your web server running on port 8080 on your computer, so you need to use the following
web address in your web browser: http://localhost:8080.

The coach’s home page

appears in your browser.
It’s called “index.html”
and it is included in the
“webapp.zip” download.

…and your web server

springs into life, logging
(to the screen) any and
all web requests that it
processes.

File Edit Window Help DisplayingHomePage

$ python3 simple_httpd.py

The “timing
Starting simple_httpd on port: 8080

data” hyperlink is
localhost - - [12/Sep/2010 14:30:03] "GET / HTTP/1.1" 200 -
localhost - - [12/Sep/2010 14:30:03] "GET /coach.css HTTP/1.1" 200 -
waiting for you localhost - - [12/Sep/2010 14:30:03] "GET /images/coach-head.jpg HTTP/1.1" 200 -
to click it. localhost - - [12/Sep/2010 14:30:03] "GET /favicon.ico HTTP/1.1" 200 -

240   Chapter 7
 web development

Sure enough, clicking on the home page’s link runs the generate_list.py program on the web
server, which displays Coach Kelly’s athletes as a list of radio buttons.

Looking good…the web

page has been generated
correctly…

…and your web server logs

the web request to run
the “generate_list.py”
CGI script.
File Edit Window Help DisplayingHomePage
$ python3 simple_httpd.py
Starting simple_httpd on port: 8080
localhost - - [12/Sep/2010 14:30:03] "GET / HTTP/1.1" 200 -
localhost - - [12/Sep/2010 14:30:03] "GET /coach.css HTTP/1.1" 200 -
localhost - - [12/Sep/2010 14:30:03] "GET /images/coach-head.jpg HTTP/1.1" 200 -
localhost - - [12/Sep/2010 14:30:03] "GET /favicon.ico HTTP/1.1" 200 -
localhost - - [12/Sep/2010 14:45:16] "GET /cgi-bin/generate_list.py HTTP/1.1" 200 -

You can click the Home hyperlink to return to the coach’s home page, or
select an athlete from the list (by clicking on their radio-button), before
pressing the Select button to continue.

Select an athlete and press Select. What happens?

you are here 4   241

no such cgi script

The dreaded 404 error!

Whoops! Your web server has responded with a “404” error code, which is
its way of telling you that something was wrong with your request. The web
server is in fact telling you that it can’t locate the resource that your web
browser requested, so it’s telling you that you made a mistake:

Yikes! The web server can’t d

“/cgi-bin/generate_timing_dfinata
the
.py
CGI script, which triggers the 40 ”
4.

Check the web server’s console window to confirm that your attempt to post your
form’s data to generate_timing_data.py resulted in failure.
Which isn’t really that surprising seeing as you have yet to write that code! So…things
aren’t as bad as they first appear. The “404” error is exactly what you would expect
to be displayed in this situation, so your generate_list.py CGI is working
fine. What’s needed is the code to the other CGI script.

If you create the required CGI script, you’ll be back on track.

242   Chapter 7
 web development

Tonight’s talk: To be CGI or not to be CGI, that is the question.

A Python Program: A Python CGI Script:

Listen: you’re really not all that different than me;
you just work on a web server, whereas I can work
anywhere.
Yes. I like to think of myself as special.
Special?!? But you only work on the Web, nowhere
else. How’s that “special”?
Because all the cool stuff works on the Web these
days and I’m designed, optimized, tailored, and
engineered for the Web. Because the Web’s a cool
place, it follows that I must be cool, too. See: special.
Nonsense! The truth is that you work only on the Web
and break pretty quickly when used elsewhere. You
don’t even have control over your own I/O.
I don’t need control over my input and output. I
have a friendly web server to take care of that for
me. My input comes from the web server and my
output goes to the web server. This arrangement
allows me to concentrate on the important stuff.
Like [sniggers] generating text in the form of
HTML? That’s really taxing…
Smirk all you want; HTML makes the World Wide
Web go around and I’m a master at generating it
dynamically, on demand, and as needed. Without me,
the Web would be a pretty static place.
Oh, get over yourself ! You’re a regular program,
just like me. I can generate HTML, too, I just
choose not to.
And if you did generate HTML, you’d want it
displayed somewhere…like in a browser?
I guess so…
And to do that you’d need to rely on the services of
a friendly web server, right?
Ummmm…I guess so.
Which would make you a CGI script. So, you’d be
special, too. Q.E.D.

you are here 4   243

yet another cgi script

Create another CGI script

Let’s take a moment to recall what is required from the generate_
timing_data.py CGI script. Based on your hand-drawn sketch from It’s probably a good
earlier, your need to generate a new HTML page that contains the top three idea to add a title
times for the selected athlete: to this page.
This looks like a
HTML heading for
the page, right?
Timing data for Sarah: Let’s include
the ahtlete’s
full name and
2.18 DOB here.

2.21
This might be best 2.22
rendered as an
unordered HTML list.
Home Select another athlete.

Two hyperlinks: one jumps to th e home

to the
page, whereas the other returns
previous selection page.

But how do you know which athlete is selected?

When you click on a radio-button and then press the Select button, a new
web request is sent to the server. The web request identifies the CGI script
to execute (in this case, that’s generate_timing_data.py), together
with the form’s data. The web server arranges to send the form’s data to your
CGI script as its input. Within your code, you can access the form data using
Python’s cgi module, which is part of the standard library:

Import the import cgi Grab all of the form data

“cgi” library. and put it in a dictionary.
form_data = cgi.FieldStorage()
athlete_name = form_data['which_athlete'].value

Access a named piece of data from the form’s data.

244   Chapter 7
 web development

It’s time to exercise your newly acquired web-coding chops. Grab

your pencil and write the code for the generate_timing_data.

Write the code

py CGI script. It’s not too different from the generate_list.py
to your new CGI
code, so you should be able to reuse a lot of your existing code.

script here.

you are here 4   245

revised script

It’s time to exercise your newly acquired web-coding chops. You

were to grab your pencil and write the code for the generate_
timing_data.py CGI script. It’s not too different from the
generate_list.py code, so you should be able to reuse a lot of
your existing code.

#! /usr/local/bin/python3 This line is needed on Unix-based systems only.

import cgi
Import the
libraries and import athletemodel
modules you
intend to use. import yate

Get the data

from the athletes = athletemodel.get_from_store()
model.
Which
athlete’s form_data = cgi.FieldStorage()
data are
you working athlete_name = form_data['which_athlete'].value
with?

Nothing print(yate.start_response())
new here
or here. print(yate.include_header("Coach Kelly's Timing Data"))
print(yate.header("Athlete: " + athlete_name + ", DOB: " + Grab the
athlete’s name
athletes[athlete_name].dob + ".")) and DOB.
print(yate.para("The top times for this athlete are:"))
Turn the top three list into
print(yate.u_list(athletes[athlete_name].top3())) an unordered HTML list.
The bottom of print(yate.include_footer({"Home”: "/index.html",
this web page
has two links. "Select another athlete": "generate_list.py"}))
A link back to the previous
CGI script.

246   Chapter 7
 web development

Note: If you are on a Unix-based system, don’t forget

to add “chmod +x generate_timing_data.py” to

Test Drive set the executable bit.

Your web server should still be running from earlier. If it isn’t, start it again. In your web browser,
return to the coach’s home page, then select the hyperlink to display the list of athletes, select Sarah,
and then press the button.

This all
looks OK.

Ah, phooey!
Something’s not
quite right here.
Where’s Sarah’s
top three times?
File Edit Window Help HoustonWeHaveAProblem
$ python3 simple_httpd.py
Starting simple_httpd on port: 8080
localhost - - [12/Sep/2010 14:30:03] "GET / HTTP/1.1" 200 -
Does the localhost - - [12/Sep/2010 14:30:03] "GET /coach.css HTTP/1.1" 200 -

web server’s localhost - - [12/Sep/2010 14:30:03] "GET /images/coach-head.jpg HTTP/1.1" 200 -

logging
localhost - - [12/Sep/2010 14:30:03] "GET /favicon.ico HTTP/1.1" 200 -

information
localhost - - [12/Sep/2010 14:45:16] "GET /cgi-bin/generate_list.py HTTP/1.1" 200 -
localhost - - [12/Sep/2010 16:12:27] “GET /cgi-bin/generate_list.py HTTP/1.1” 200 -
tell you localhost - - [12/Sep/2010 16:12:29] “POST /cgi-bin/generate_timing_data.py HTTP/1.1” 200 -
anything? Traceback (most recent call last):
File “/Users/barryp/HeadFirstPython/chapter7/cgi-bin/generate_timing_data.py”, line 21, in <module>
print(yate.u_list(athletes[athlete_name].top3()))
TypeError: ‘list’ object is not callable
localhost - - [12/Sep/2010 16:12:29] CGI script exit status 0x100

Your CGI has suffered from a TypeError exception, but other than looking at the web
server’s logging screen, it’s not clear on the web browser screen that anything has gone wrong.

What do you think is the problem here? Take a moment

to study the error message before flipping the page.

you are here 4   247

track down cgi errors

Enable CGI tracking to help with errors

The CGI standard dictates that any output generated by a server-side
program (your CGI script) should be captured by the web server and sent
to the waiting web browser. Specifically, anything sent to STDOUT (standard
output) is captured.
When your CGI script raises an exception, Python arranges for the error
message to display on STDERR (standard error). The CGI mechanism is
programmed to ignore this output because all it wants is the CGI script’s
standard output.

When your CGI works, I’ll fill

your STDOUT with lovely HTML.
When your CGI fails, it’s a case
of—POOF!—gone for good. Sorry,
but that’s the way the CGI cookie
crumbles...

Web
Server

This behavior is fine when the webapp is deployed, but not when it’s being
developed. Wouldn’t it be useful to see the details of the exception in the
browser window, as opposed to constantly having to jump to the web server’s
logging screen?
Well…guess what? Python’s standard library comes with a CGI tracking
module (called cgitb) that, when enabled, arranges for detailed error
messages to appear in your web browser. These messages can help you work
out where your CGI has gone wrong. When you’ve fixed the error and your
CGI is working well, simply switch off CGI tracking:
Add these two lines near the
start of your CGI scripts to
import cgitb enable Python’s CGI tracking
cgitb.enable() technology.

248   Chapter 7
 web development

Test Drive
Add the two CGI tracking lines of code near the top of your generate_timing_data.py CGI
script. Press the Back button on your web browser and press the Select button again. Let’s see what
Wow! Look at all happens this time.
of this detail.

What’s this? top3() is

Notice that the CGI tracking module tries to identify
undefined?!?
exactly where the problem with your code lies.

you are here 4   249

small fix, big difference

A small change can make all the difference

The CGI tracking output indicates an error with the use of the top3() method
from the AthleteList code.
A quick review of the code to the AthleteList class uncovers the source of the
error: the top3() method has been redesignated as a class property.

This decorator allows you to acc

returned by “top3()” as if it weess the data
@property
attribute. re a class
def top3(self):
return(sorted(set([self.sanitize(t) for t in self]))[0:3])

The use of the @property decorator allows the top3() method to appear
like an attribute to users of the class. So, instead of calling the top3() A method call
method like this: always needs the
parentheses…

print(yate.u_list(athletes[athlete_name].top3()))

Treat the top3() method as if it was another class attribute, and call it like
this:

print(yate.u_list(athletes[athlete_name].top3))

It’s a small change, but it’s an important one …unless the method is declared
to be an “@property”, in which
When a change is made to the way a class is used, you need to be careful to case parentheses are NOT
consider what impact the change has on existing programs, both yours and required.
those written by others.
At the moment, you are the only one using the AthleteList class, so it’s
not a big deal to fix this. But imagine if thousands of programmers were
using and relying on your code…

Let’s fix your CGI script and try again.

250   Chapter 7
 web development

Test Drive
Make the small edit to your code to remove the brackets from the call to the top3() method, press
your web browser’s Back button, and press the Select button one last time.

Whoo hoo! This

time the selected
athlete’s data is
displayed on screen.
Nice, eh?

Now that you’ve

solved that
problem, be sure
to switch off
CGI tracking.

Q: What happens if the coach recruits new athletes? Q: Shouldn’t the server’s data be in a database as opposed

A:
to a pickle? Surely that would be better, right?
All Coach Kelly needs do is create a new text file similar to the
others, and your webapp handles the rest by dynamically including
the new athlete the next time your webapp runs, which occurs when
A: In this case, it’s probably overkill to use a database, but it might
be worth considering sometime in the future.
someone clicks on the home page’s “timing data” hyperlink.

you are here 4   251

successful webapp

Your webapp’s a hit!

This is great! Now I can

share my data with the kids,
and they can access their
times without bugging me...

Hey, this is super cool

and the coach is right...
if we train hard, we can
Wait until my mom
improve on these times!
sees my times...she’ll
be so proud of me!

By moving your program to the Web, you’ve made it a no-brainer for Coach
Kelly to share his data with not only his athletes, but with anyone else that
needs to access his data.
By conforming to the MVC pattern and using CGI, you’ve built a webapp in
such a way that it’s easy to extend as new requirements are identified.

Congratulations! You’re a web developer.

252   Chapter 7
 web development

Your Python Toolbox

You’ve got Chapter 7 under your

CHAPTER 7
belt and you’ve added some key
Python techiques to your toolbox.
ƒƒ The Model-View-Controller pattern lets
you design and build a webapp in a
maintainable way.
ƒƒ The model stores your webapp’s data.

Python Lingo
ƒƒ The view displays your webapp’s user
interface.

• “@proper ƒƒ The controller glues everything together

ty” -
lets you arrange a decorator that with programmed logic.
appear as if it is for a class method to ƒƒ The standard library string module
a class attribute. includes a class called Template,
which supports simple string substitutions.
ƒƒ The standard library http.server

Web Lingo
module can be used to build a simple web
server in Python.
” - a p r ogr a m that runs on
• “webapp
ƒƒ The standard library cgi module
the Web.
provides support for writing CGI scripts.

q ue st ” - se n t from the web ƒƒ The standard library glob module is

• “web re
web server.
browser to the
great for working with lists of filenames.

sp on se ” - se n t from the web ƒƒ Set the executable bit with the chmod
• “web re
to t h e w eb b r owser in repsonse +x command on Linux and Mac OS X.
server .
to a web request
ƒƒ The standard library cgitb module,
eway when enabled, lets you see CGI coding
• “CGI” -
the Common Gateb server to errors within your browser.
allows a w
Interface, which e program. ƒƒ Use cgitb.enable() to switch on
run a server-sid a
CGI tracking in your CGI code.

G I sc ri p t ” - a nother name for ƒƒ Use cgi.FieldStorage() to

• “C
am.
server-side progr
access data sent to a web server as part
of a web request; the data arrives as a
Python dictionary.

you are here 4   253

8 mobile app development

Small devices

This had better be a

smartphone running
Honeycomb or Mr. Smooth
is history!

Putting your data on the Web opens up all types of possibilities. 

Not only can anyone from anywhere interact with your webapp, but they are increasingly
doing so from a collection of diverse computing devices: PCs, laptops, tablets, palmtops,
and even mobile phones. And it’s not just humans interacting with your webapp that
you have to support and worry about: bots are small programs that can automate web
interactions and typically want your data, not your human-friendly HTML. In this chapter,
you exploit Python on Coach Kelly’s mobile phone to write an app that interacts with your
webapp’s data.

this is a new chapter   255

going mobile

The world is getting smaller

Coach Kelly is continuing to use his webapp every day, but he’s having a
problem with his new smartphone.

I can access my timing data over WiFi on

my phone, but it’s so small it’s all but impossible
to read, let alone click on links or buttons. Can
you take a look at it for me? Gotta dash. I’ve got
another 5K to do before breakfast...

There’s more than just desktop computers out there.

Who knew that your users would try to interact with your webapp using
something other than a desktop computer or laptop?
It’s a diverse computing environment out there.

256   Chapter 8
 mobile app development

Coach Kelly is on Android

The coach has a lovely new smartphone that’s running Google’s Android
operating system. Sure enough, when you check it out, the webapp is way too
small and not much use on the coach’s three-inch screen:

And don’t go telling

me to do all that two-
Does anyone ha fingered zoom and double-
magnifying glass?ve a tapping thing. That just
drives me crazy!

Obviously, the coach needs to access his data and run his webapp
on his phone…but what’s the best way to do this if not through the
phone’s browser?

Open your web browser on your desktop computer (or phone)

and enter “Python for Android” into your favorite search engine.
Make a note in the space below of the most promising site from
your search results:

you are here 4   257

scripting layer for android

You were to open your web browser on your desktop computer

(or phone) and enter “Python for Android” into your favorite
search engine. You were then to make a note in the space below
of the most promising site from your search results:
Is this the one
you found?
http://code.google.com/p/android-scripting (the home of the SL4A project.)

Run Python on the coach’s smartphone

A quick search of the Web uncovers a pleasent surprise: Python runs on Android.
At least a version of Python runs on Android. A project called Scripting Layer
for Android (SL4A) provides technology to let you run Python on any Android
device. But there’s a catch.

Ummmm...I just checked

the SL4A website, and
it looks like it supports
Python 2.6.2, not Python 3.
Phooey!

Yes. SL4A ships with Python 2, not 3.

Python 3, this book’s preference, is the best version of
Python yet, but it achieves its greatness at the cost of a
lack of backward compatibility. There’s some stuff in
3 that will never work in 2 and vice versa.
Is this fact alone a show-stopper?

258   Chapter 8
 mobile app development

Don’t worry about Python 2

The fact that Python 2 is available for Android and you’ve learned Python
I’m quite happy
3 in this book is nothing to lose sleep over. Python 2 is still Python, and the
to run Python 3
differences between Python 2 and Python 3 are easy to manage.
all day long...
Think about your webapp for a minute. Right now, the model, view, and
controller code resides on the web server, which is running Python 3.

Your web browser

runs here.

Web
The Internet
Server

All of your webapp

code runs here.
If you move the user interaction to the smartphone, the model and some
of the controller code stay on the server (and continue to run on Python
3), whereas the view code and the rest of the controller code move to the
smartphone, where they need to be rewritten to run on Python 2. Python 3 is still
going strong...

Run Python code?

No problem...just so
long as it’s Python 2.

Web
The Internet
Server

Half your webapp’s

code runs here…

…and the other

half runs here.
you are here 4   259
android sdk

Set up your development environment

Understandably, the coach won’t let your have his phone to work on until
you have something that works. Thankfully, Google provides a cross-platform Do this!
Android emulator that lets you develop for the phone as needed, even
though you don’t own any hardware.

Follow along with

Download the Software Development Kit (SDK) these instructions to
ensure you have your
Let’s get started developing for Android. Visit this website and download the
Android development
SDK for your computer and operating system:
environment correctly set
up on your computer.
http://developer.android.com/sdk/index.html

The Android
SDK website

Note: This is how the Android SDK

Despite what this website might look like it’s telling you, you do not need to
download page looks at the tim e of
this writing. It might look aes:little
install Eclipse to run the Android emulator. However, you do need to have a
t
different for you. No worri of jus
Java Runtime Environment installed. If you are unsure about this, don’t worry:
sion th e
download the latest ver
the Android emulator will advise your best course of action if it spots that
Java is missing.
SDK.
260   Chapter 8
 mobile app development

Configure the SDK and emulator

You need to do two things to configure the SDK and emulator: add an Android
Platform and create an Android Virtual Device (known as an AVD).

Add an Android platform The Android download contath ins

a folder called “tools”. Ru n e
The coach is running Android 2.2 on his phone, so let’s add a 2.2 platform
“android” program wit hin th is
to mimic this setup. Open up the Android SDK and AVD Manager tool,
select Available Packages, and pick 2.2 for installation. folder.

This is the only version of

the SDK that you need.

This might take a minute or tw

depending on the speed of your o,
network connection.

Create a new Android Virtual Device (AVD)

With the 2.2 platform downloaded and installed, create a new Android
Virtual Device.
Give your AVD a name,
and select a target.

Set the size of the

virtual SDcard: 512 is
more than enough.

Your AVD is
a simulated
Click on
“Create AVD”. Android phone.
you are here 4   261
emulate sl4a

Install and configure Android Scripting

These instructions work on a “re
With the emulator ready, use the AVD Manager to start your 2.2 device. Click phone, too. Just be sure to enableal”
on the emulator’s browser (the little globe), surf to this web address: “Unknown sources” to allow for
http://code.google.com/p/android-scripting
Market application downloads. non-
and tap on the “boxed” bar code near the bottom of the page:
Don’t worry if it takes you em
minute or two to start. Ther em ulator a
ula
slower than the actual phone… tor is

On the emulator,
tap on the
“boxed” bar code
to start the
SL4A download.

When the download completes, select the emulator’s Menu button ➝ More ➝
Downloads, and then tap on the sl4a_r2.apk file to install the SL4A The version available to you : mig ht be
rry do wn load the
package on the emulator. When the install completes, tap Done. different, but don’t wo
latest release.

ferent than
The version you see might bethdif st recent.
this. Don’t worry; yours in e mo

262   Chapter 8
 mobile app development

Add Python to your SL4A installation

Return to the emulator’s web browser, double-tap on the screen to zoom in,
and select the Downloads tab. Double-tap again and tap the following link:
Again, the version you see migthtthe
be different than this. Selec
python_for_android_r1.apk
Tap the download link, and tap on the package name to download it. Select most recent file.
Menu ➝More ➝ Downloads, and tap on the newly downloaded package.

The Python for Android app runs. When you are ready, tap Open -> Install This last bit is really
to complete the installation. This downloads, extracts, and installs the Python important.
support files for Android, which can take a few minutes to complete. When
it does, Python 2.6.2 and Python for Android are installed on your emulator
and ready for action.

Let’s confirm everything is working with a quick test.

you are here 4   263

 your script on android

Test Python on Android

Return to your emulator’s main screen and find an app called SL4A added to your list
of app icons. Tap this app to display the list of Python scripts preinstalled with Python
for Android. Simplty tap on any script name to execute it:

The “menu”
button.

Be sure to set
the SL4A
rotation mode
to automatic.
Your screen might
switch to landscape by default
the first time you run a script.
To fix this, choose Menu ➝
Preferences, scroll down to
Take your Android emulator for a spin Rotation mode, and set its
Here’s a four-line Python script that you can create to test your installation. value to Automatic.
Let’s call this script mydroidtest.py:

Import the “android”

library and create a new import android
app object instance.
app = android.Android()
msg = "Hello from Head First Python on Android"
Create an appropriate
message and display it on app.makeToast(msg)
screen.
Issue this command at your min
window to transfer your scrter al
To transfer your script to the emulator, you need to copy it to the emulator’s
virtual SD card. Another program within the tools folder called adb helps
the emulator. ipt to
with this:

tools/adb push mydroidtest.py /sdcard/sl4a/scripts

Your script should now appear on the list of scripts available to SL4A.

264   Chapter 8
 mobile app development

Test Drive
Let’s confirm that your Android setup is working. With the SL4A app open, simply tap on your script’s
name to run it, and then click the run wheel from the menu.

Click your app’s

name…

…then click the

“run wheel.”

Your Android
emulator with SL4A
is working, and it’s
running your Python
And there’s your message. It code.
works!
you are here 4   265
what to do?

Define your app’s requirements

Let’s think a little bit about what your Android app needs to do.

Nothing’s really
changed...you just
have to get the web
data onto the phone.

Frank: Well…first off, the view code no longer has to generate HTML,
so that makes things interesting.
Jill: In fact, you need the web server only to supply your data on
request, not all that generated HTML.
Joe: Ah ha! I’ve solved it. Just send the pickle with all the data from the
server to the Android phone. It can’t be all that hard, can it?
Jill: Sorry, guys, that’ll cause problems. The pickle format used by
Python 3 is incompatible with Python 2. You’ll certainly be able to send
the pickle to the phone, but the phone’s Python won’t be able to work
with the data in the pickle.
Frank: Darn…what are our options, then? Plain data?
Frank Joe
Jill Joe: Hey, good idea: just send the data as one big string and parse it on
the phone. Sounds like a workable solution, right?
Jill: No, that’s a potential disaster, because you never know in what
format that stringed data will arrive. You need an data interchange format,
something like XML or JSON.
Frank: Hmm…I’ve heard XML is a hound to work with…and it’s
probably overkill for this simple app. What’s the deal with JSON?
Joe: Yes, of course, I keep hearing about JSON. I think they use it in
lots of different places on the Web, especially with AJAX.
Frank: Oh, dear…pickle, XML, JSON, and now AJAX…I think my
brain might just explode here.
Jill: Never worry, you only need to know JSON. In fact, you don’t even
need to worry about understanding JSON at all; you just need to know
how to use it. And, guess what? JSON comes standard with Python
2 and with Python 3…and the format is compatible. So, we can use
JSON on the web server and on the phone.
Frank & Joe: Bonus! That’s the type of technology we like!

266   Chapter 8
 mobile app development

JSON Exposed
This week’s interview:
The Data Interchange Lowdown

Head First: Hello, JSON. Thanks for agreeing to Python, use JSON to convert it to JSON’s object
talk to us today. notation, and then send the converted data to
another computer running a program written in C#?
JSON: No problem. Always willing to play my part
in whatever way I can. JSON: And as long as C# has a JSON library, you
can recreate the Python data as C# data. Neat, eh?
Head First: And what is that, exactly?
Head First: Yes, that sounds interesting…only
JSON: Oh, I’m just one of the most widely used
[winks] why would anyone in their right mind want
data interchange formats on the Web. When you
to program in C#?
need to transfer data over the Internet, you can rely
on me. And, of course, you’ll find me everywhere. JSON: [laughs] Oh, come on now: be nice. There’s
plenty of reasons to use different programming
Head First: Why’s that?
languages for different reasons.
JSON: Well…it’s really to do with my name. The
Head First: Which goes some of the way to explain
“JS” in JSON stands for “JavaScript” and the “ON”
why we have so many great programming titles, like
stands for “Object Notation.” See?
Head First C#, Head First Java, Head First PHP and
Head First: Uh…I’m not quite with you. MySQL, Head First Rails, and Head First JavaScript.
JSON: I’m JavaScript’s object notation, which JSON: Was that a shameless, self-serving plug?
means I’m everywhere.
Head First: You know something…I think it might
Head First: Sorry, but you’ve completely lost me. well have been! [laughs].
JSON: The first two letters are the key ones: I’m JSON: [laughs] Yes, it pays to advertise.
a JavaScript standard, which means you’ll find me
Head First: And to share data, right?
everywhere JavaScript is…which means I’m in every
major web browser on the planet. JSON: Yes! And that’s exactly my point: when you
need a language-neutral data interchange format that is
Head First: What’s that got to do with Python?
easy to work with, it’s hard to pass me by.
JSON: That’s where the other two letters come
Head First: But how can you be “language neutral”
into play. Because I was initially designed to allow
when you have JavaScript in your name?
JavaScript data objects to be transferred from one
JavaScript program to another, I’ve been extended JSON: Oh, that’s just my name. It’s what they
to allow objects to be transferred regardless of what called me when the only language I supported was
programming language is used to create the data. JavaScript, and it kinda stuck.
By using the JSON library provided by your favorite
Head First: So they should really call you
programming language, you can create data that
something else, then?
is interchangeable. If you can read a JSON data
stream, you can recreate data as you see fit. JSON: Yes, but “WorksWithEveryProgramming
LanguageUnderTheSunIncludingPythonObject
Head First: So I could take an object in, say,
Notation” doesn’t have quite the same ring to it!

you are here 4   267

leaving pickle on the plate

This is NOT cool... I spent all that time

learning to use pickles and now you’re
abandoning them in favor of this “JSON”
thing. You’ve got to be joking...?

You are not exactly “abandoning” pickle.

The JSON technology is a better fit here for a number
of reasons. First of all, it’s a text-based format, so
it fits better with the way the Web works. Second, it’s
a standard that works the same on Python 2 and
Python 3, so there are no compatibility issues. And
third, because JSON is language-neutral, you open
up the possibility of other web tools written in other
programming languages interacting with your server.
If you use pickle here, you lose all this.

268   Chapter 8
 mobile app development

JSON is an established web standard that comes preinstalled with Python 2 and Python 3. The JSON API is not that
much different to the one used by pickle:
Import the JSON library.
>>> import json
>>> names = ['John', ['Johnny', 'Jack'], 'Michael', ['Mike', 'Mikey', 'Mick']]
>>> names Create a list of lists.
['John', ['Johnny', 'Jack'], 'Michael', ['Mike', 'Mikey', 'Mick']]

Transform the Python list-of-lists into a JSON list of lists.

>>> to_transfer = json.dumps(names)
>>> to_transfer The format is similar,
'["John", ["Johnny", "Jack"], "Michael", ["Mike", "Mikey", "Mick"]]' but different.
Transform the JSON list of list
>>> from_transfer = json.loads(to_transfer) into one that Python understands back
>>> from_transfer
s.
['John', ['Johnny', 'Jack'], 'Michael', ['Mike', 'Mikey', 'Mick']]
e
The new data is exactly the sam
>>> names as the original list of lists.
['John', ['Johnny', 'Jack'], 'Michael', ['Mike', 'Mikey', 'Mick']]

Add a new function to the athletemodel module that, when

called, returns the list of athlete names as a string.

Call the new function get_names_from_store().

you are here 4   269

athletemodel function

You were to add a new function to the athletemodel module

that, when called, returns the list of athlete names as a string.

You were to all the new function get_names_from_store().

def get_names_from_store(): Get all the data from the pickle.

athletes = get_from_store()
Extract a list
of athlete names response = [athletes[each_ath].name for each_ath in athletes]
from the data.
return(response)
Return the list to the caller.

So...rather than running

a CGI script to create a HTML
web page, you want me to deliver
just the data, right? That’s OK. Not
a problem—just be sure to tell me
which script to run...

Web
Server

270   Chapter 8
 mobile app development

With your new function written and added to the athletemodel module, create a new CGI
script that, when called, returns the data from the get_names_from_store() function to
the web requester as a JSON data stream.
Call your new script cgi-bin/generate_names.py.
Hint: Use application/json as your Content-type.

I may be small, but I’m

mighty capable. Whether
you need a web page or just
your data, you can count on me
to get the job done.

you are here 4   271

json-generating cgi script

With your new function written and added to the athletemodel module, you were to create
a new CGI script that, when called, returns the data from the get_names_from_store()
function to the web requester as a JSON data stream.
You were to call your new script cgi-bin/generate_names.py.

Don’t forget this #! /usr/local/bin/python3

“magic” first line
if you’re running
on Linux or Mac import json
OS X.
import athletemodel Do your imports.

import yate

Get the data from

names = athletemodel.get_names_from_store() the model.
Start with the
appropriate
“Content-type”:
line. print(yate.start_response('application/json')) Sort “names”, then convert
print(json.dumps(sorted(names))) to JSON and send to
STDOUT.

Take care testing

your JSON-generating
CGI code.
The behavior you see
when testing your JSON-
generating CGI script will differ
depending on the web browser you
are using. For instance, Firefox might
attempt to download the generated
data as opposed to display it on screen.

272   Chapter 8
 mobile app development

Test Drive
If it is not already running, start your web server and be sure to set the executable bit with the
chmod +x cgi-bin/generate_names.py command (if on Linux or Mac OS X). When you’re
ready, grab your favorite web browser and take your new CGI for a spin.
I in your
Enter the web address of the CG
browser’s location bar.

Hey! It looks like the coach

added two new athletes. has

File Edit Window Help GeneratingJSON

$ python3 simple_httpd.py
Starting simple_httpd on port: 8080

The web server’s localhost - - [18/Sep/2010 06:31:29] "GET /cgi-bin/generate_names.py HTTP/1.1" 200 -

logging information
localhost - - [18/Sep/2010 06:35:29] "GET /cgi-bin/generate_list.py HTTP/1.1" 200 -

confirms that the

localhost - - [18/Sep/2010 06:35:35] "POST /cgi-bin/generate_timing_data.py HTTP/1.1" 200 -
localhost - - [18/Sep/2010 06:35:38] "GET /cgi-bin/generate_list.py HTTP/1.1" 200 -
CGI executed. localhost - - [18/Sep/2010 06:35:40] "GET /index.html HTTP/1.1" 200 -
localhost - - [18/Sep/2010 06:35:49] "GET /cgi-bin/generate_names.py HTTP/1.1" 200 -

That worked!

Now all you have to do is arrange for the Android emulator to request the
data within a Python script and display the list of names on the smartphone’s
screen. How hard can that be?

you are here 4   273

two apis

The SL4A Android API

The SL4A technology provides a high-level API to the low-level Android API,
and SL4A’s API is documented in the online API reference:
http://code.google.com/p/android-scripting/wiki/ApiReference

Recall the code from earlier, which demonstrated a minimal Android SL4A
app:
Import the “android”
library and create a new import android
app object instance.
app = android.Android()
msg = "Hello from Head First Python on Android"
Create an appropriate
message and display it on app.makeToast(msg)
screen.

Six calls to the Android API let you create a list of selectable items in a dialog,
together with positive and negative buttons, which are used to indicate the
selection your user made. Note how each of the calls to the Android “dialog”
API results in something appearing on screen.

Always start with an import.

import android Create an Android

app object.
app = android.Android()

app.dialogCreateAlert("Select an athlete:")
app.dialogSetSingleChoiceItems(['Mikey', 'Sarah', 'James', 'Julie'])
app.dialogSetPositiveButtonText("Select")
app.dialogSetNegativeButtonText("Quit")
app.dialogShow()
resp = app.dialogGetResponse().result

Display your dialog se

on the phone. Wait for a resp.on
from your user

274   Chapter 8
 mobile app development

Android Code Magnets

Here is the code to a program that queries your web server for the list of names as a JSON array
and then displays the list on the smartphone. The only trouble is, the second half of the program
is a bunch of mixed-up code magnets at the bottom of the screen. Your job is to rearrange the
magnets to complete the program.
import android
import json
import time Do the usual imports…these
ones pull in web client
from urllib import urlencode
functionality.
All of this from urllib2 import urlopen
program’s
messages are in
one place.
hello_msg = "Welcome to Coach Kelly's Timing App"
list_title = 'Here is your list of athletes:'
quit_msg = "Quitting Coach Kelly’s App."

The name web_server = 'http://192.168.1.33:8080' Change this to the

of the CGI get_names_cgi = '/cgi-bin/generate_names.py' web address that’s
script to run running your web
on the web def send_to_server(url, post_data=None): server.
server if post_data:
This function takes both a
page = urlopen(url, urlencode(post_data)) web address (url) and some
else: optional data (post_data)
page = urlopen(url) and sends a web request to
return(page.read().decode("utf8")) your web server. The web
response is returned to the
caller.
This code’s a
mess…can you
fix it?

get_names_cgi)))
athlete_names = sorted(json.loads(send_to_server(web_server +
status_update(qu resp = app
it_msg) .dialogGet
Response()
.result
app.dialogShow() app.dialogCreateAlert(list_title) status_update(he
llo_msg)
):
e(msg, how_long=2
def status_updat app.dialogSetPositiveBut
g) tonText('Select')
app.makeToast(ms
ong) app = android.An
time.sleep(how_l droid()
app.dialogSetNegativeButtonText('Quit')
app.dialogSetSingleChoiceItems(athlete_names)
you are here 4   275
android query

Android Code Magnets Solution

Here is the code to a program that queries your web server for the list of names as a JSON array
and then displays the list on the smartphone. The only trouble is, the second half of the program
is a bunch of mixed-up code magnets at the bottom of the screen. Your job was to rearrange the
magnets to complete the program.
import android
import json
import time
from urllib import urlencode
from urllib2 import urlopen

hello_msg = "Welcome to Coach Kelly's Timing App"

list_title = 'Here is your list of athletes:'
quit_msg = "Quitting Coach Kelly’s App."
web_server = 'http://192.168.1.33:8080'
get_names_cgi = '/cgi-bin/generate_names.py'

def send_to_server(url, post_data=None):

if post_data:
page = urlopen(url, urlencode(post_data))

Create an Android else:

app object. page = urlopen(url) This is a little

return(page.read().decode("utf8")) function for Send the web request
displaying short to your server, then
app = android.Android() messages on the turn the JSON response
def status_update(msg, how_long=2):
phone. into a sorted list.
app.makeToast(msg)
time.sleep(how_long)
Say status_update(hello_msg)
“hello”. get_names_cgi)))
athlete_names = sorted(json.loads(send_to_server(web_server +

app.dialogCreateAlert(list_title)
app.dialogSetSingleChoiceItems(athlete_names) Create a two-buttoned
dialog from the list of
app.dialogSetPositiveButtonText('Select')
athlete names.
app.dialogSetNegativeButtonText('Quit')
app.dialogShow()
Wait for the user to tap a ton
resp = app.dialogGetResponse().result then assign the result to “rebut
sp”.
,

Say “bye bye.”

status_update(quit_msg)

276   Chapter 8
 mobile app development

Test Drive
Recall that (for now) your Android Python scripts run within the emulator, not within IDLE. So use the
tools/adb program to copy your program to the emulator. Call your program coachapp.py. When
the code is copied over, start SL4A on your emulator, and then tap your script’s name.

Tap your app’s

name, and
then tap the
“run wheel."

And there they are…Coach

Kelly’s athletes.

This is looking really good! Your app has communicated with your web server,
requested and received the list of athlete names, and displayed the list on your
emulator.

If you app doesn’t run, don’t panic. Check your code for typos.
Run your app again in the Python terminal by tapping on the little terminal icon to the
left of the “run wheel” within SL4A. If your code raises an error, you’ll see any messages
on the emulator’s screen, which should give you a good idea of what went wrong.

you are here 4   277

positive or negative

Select from a list on Android

When your user taps on a button, the “result” of the call to
dialogGetResponse() is set to positive if the first button is tapped
or negative if the second button is tapped. In your code, you can check
the value of resp, which is a dictionary, and the which key is set to either
positive or negative.
A subsequent call to dialogGetSelectedItems() returns the index
value of the selected list item.

Index item 0

Index item 1

Index item 2

Index item 3

Index item 4

The “positive” button The “negative” button

So…if the positive button is tapped, you can index into the list of athlete names
to see which athlete was selected from the displayed list. The selected name can then
be sent to the web server to request the rest of the athlete’s data using the send_
to_server() function.

You can use this behavior in the next version of your code.

278   Chapter 8
 mobile app development

1 Assume that you have a CGI script called cgi-bin/

generate_data.py, which, when called, requests the data
for a named athlete from the server.

Provide the code (which includes a call to thensend_to_

server() function) to implement this functionality:

Additionally, write the code required to display the list of times returned from the server within an Android
2 dialog.

Hints: Use the dialogSetItems() method from the Android API to add a list of items to a dialog. Also,
remember that the data arriving over the Internet will be formatted using JSON.

you are here 4   279

ask for an athlete

You were to assume that you have a CGI script called cgi-bin/
1
generate_data.py, which, when called requests the data for
a named athlete from the server.

Provide the name of You were to provide the code (which includes a call to
the CGI to run. the send_to_server() function) to implement this
functionality:

get_data_cgi = '/cgi-bin/generate_data.py'
Send the request send_to_server(web_server + get_data_cgi, {'which_athlete': which_athlete})
to the web server,
together with the
athlete name. Include the data.
Additionally, you were to write the code required to display the list of times returned from the server within
2 an Android dialog: Wh ich button was pressed?

When your user if resp['which'] in ('positive'): The index value

taps th e “po sit ive ” is in the first
button…work out selected_athlete = app.dialogGetSelectedItems().result[0]
element of the
the index value list of results
chosen. returned from the
Look up the which_athlete = athlete_names[selected_athlete] dialog.
athlete’s name using
the index value. Send a new
athlete = json.loads(send_to_server(web_server + get_data_cgi, web request
to the server
{'which_athlete': which_athlete})) to fetch the
Dynamically
create the athlete’s data.
dialog’s title.
athlete_title = which_athlete + ' top 3 times:'
app.dialogCreateAlert(athlete_title)
The user needs app.dialogSetItems(athlete['Top3'])
to see only the
data this time, so app.dialogSetPositiveButtonText('OK’)
you need to use Set the single
“dialogSetItems()”. app.dialogShow()
button’s text.

Wait for a tap

from the user. resp = app.dialogGetResponse().result

280   Chapter 8
 mobile app development

The athlete’s data CGI script

Here’s the code for the cgi-bin/generate_data.py CGI script, which
takes a web request and returns the indicated athlete’s data from the model:

#! /usr/local/bin/python3

import cgi
import json
import athletemodel Get all the data
import yate from the model.
Process the
data sent with athletes = athletemodel.get_from_store()
the request and
extract the
form_data = cgi.FieldStorage() Start a web
athlete’s name. athlete_name = form_data['which_athlete'].value response, with JSON
print(yate.start_response('application/json')) as the data type.
print(json.dumps(athletes[athlete_name]))

Include the indicated

athlete’s data in the web .
The complete Android app, so far response, formatted by JSON
You’ve made quite a few changes to your program at this stage. Before you test it
on the Android emulator, take a moment to look at your code in its entirety:

import android
import json
import time

from urllib import urlencode

from urllib2 import urlopen

hello_msg = "Welcome to Coach Kelly's Timing App"

list_title = 'Here is your list of athletes:'
quit_msg = "Quitting Coach Kelly's App."

web_server = 'http://192.168.1.34:8080'
The rest of your
code is on the
get_names_cgi = '/cgi-bin/generate_names.py' following page.
get_data_cgi = '/cgi-bin/generate_data.py'

you are here 4   281

app code, continued

def send_to_server(url, post_data=None):

if post_data:
page = urlopen(url, urlencode(post_data))
else:
page = urlopen(url)
return(page.read().decode("utf8"))

app = android.Android()

def status_update(msg, how_long=2):

app.makeToast(msg)
time.sleep(how_long)

status_update(hello_msg)

athlete_names = sorted(json.loads(send_to_server(web_server + get_names_cgi)))

app.dialogCreateAlert(list_title)
app.dialogSetSingleChoiceItems(athlete_names)
app.dialogSetPositiveButtonText('Select')
app.dialogSetNegativeButtonText('Quit')
app.dialogShow()
resp = app.dialogGetResponse().result

if resp['which'] in ('positive'):
selected_athlete = app.dialogGetSelectedItems().result[0]
which_athlete = athlete_names[selected_athlete]
athlete = json.loads(send_to_server(web_server + get_data_cgi,
{'which_athlete': which_athlete}))

athlete_title = athlete['Name'] + ' (' + athlete['DOB'] + '), top 3 times:'

app.dialogCreateAlert(athlete_title)
app.dialogSetItems(athlete['Top3'])
app.dialogSetPositiveButtonText('OK')
app.dialogShow()
resp = app.dialogGetResponse().result

status_update(quit_msg)

282   Chapter 8
 mobile app development

Test Drive
Let’s give the latest version of your app a go. Copy the app to your emulator, and put the new CGI
script in your cgi-bin folder on your web server (remember to set the executable bit, if needed).
What happens when you run your latest app using the emulator’s Python shell as opposed to the
“run wheel”?

You are dumped into the

Python shell with a rather
nasty error message.

You’re getting a
“TypeError”.

After reading the error

message, click “Yes” to return
to the SL4A script listing.

Yikes! Your code has a TypeError, which is crashing your app when you try
to display the selected athlete’s timing data. Why do you think this is happening?

you are here 4   283

debugging data

The data appears to have changed type

...ummm, I see. But
Look at the CGI somehow, the data
code...it gets the data that arrives isn’t an
from the model and AthleteList.
sends it to the web
browser...

Let’s add a debugging line of code to your CGI script to try and determine what’s
going on. Recall that the CGI mechanism captures any output your script sends
to standard output by default, so let’s use code like this to send your debugging
messgage to the web server’s console, which is displaying on standard error: Redirect the output
from “print()” to
“stderr”, rather
Import “sys” import sys
than the default,
from the which is “stdout”.
standard library. print(json.dumps(athletes[athlete_name]), file=sys.stderr)

Run your app again and, of course, it’s still crashes with a TypeError.
However, if you check your web server’s console screen, you’ll see that the
data being sent as the JSON web response is clearly visible. Notice anything?
File Edit Window Help JustWhatsInTheData
$ python3 simple_httpd.py
Starting simple_httpd on port: 8080
This is a list of 192.168.1.33 - - [18/Sep/2010 17:40:04] "GET /cgi-bin/generate_names.py HTTP/1.1" 200 -
athlete timing 192.168.1.33 - - [18/Sep/2010 17:40:08] "POST /cgi-bin/generate_data.py HTTP/1.1" 200 -

values…but where’s ["2-44", "3:01", "2.44", "2.55", "2.51", "2:41", "2:41", "3:00", "2-32", "2.11", "2:26"]

the name and

DOB values?

284   Chapter 8
 mobile app development

JSON can’t handle your custom datatypes

Unlike pickle, which is smart enough to pickle your custom classes, the
JSON library that comes with Python isn’t. This means that the standard
library’s JSON library can work with Python’s built-in types, but not with
your AthleteList objects.
The solution to this problem is straightforward: add a method to your
AthleteList class to convert your data into a dictionary, and send that
back to the app. Because JSON supports Python’s dictionary, this should work.

Let’s create a new method in your AthleteList class. Called to_dict(), your new
method needs to convert the class’s attribute data (name, DOB, and top3) into a dictionary. Be
sure to decorate your new method with @property, so that it appears to be a new attribute to
users of your class.

Q: What’s the purpose of this @property thing again?

A: The @property decorator lets you specify that a method is to be presented to users of your class as if it were an attribute. If you
think about things, your to_dict() method doesn’t change the state of your object’s data in any way: it merely exists to return the object’s
attribute data as a dictionary. So, although to_dict() is a method, it behaves more like an attribute, and using the @property
decorator let’s you indicate this. Users of your class (that is, other programmers) don’t need to know that when they access the to_dict
attribute they are in fact running a method. All they see is a unified interface: attributes access your class’s data, while methods manipulate it.

you are here 4   285

data to dictionary

Let’s create a new method in your AthleteList class. Called to_dict(), your new
method needs to convert the class’s attribute data (name, DOB, and top3) into a dictionary. Be
sure to decorate your new method with @property, so that it appears to be a new attribute to
users of your class.

Decorate your @property

new method with Create a new method.
“@property”. def as_dict(self):
return({‘Name’: self.name,
‘DOB’: self.dob, Return a dictionary of the object’s
data attributes.
‘Top3’: self.top3})

Did you remember to use “self”?

Do this!

As well as updating your AthleteList

class code, be sure to change cgi-bin/
generate-data.py to return a
dictionary, rather than the object instance,
when servicing its web request.
While you’re making changes, adjust the
coachapp.py app code to include the
athlete’s name and DOB values in the
second dialog’s title.

286   Chapter 8
 mobile app development

Test Drive
With your changes applied to AthleteList.py, cgi-bin/generate_data.py and
coachapp.py, use the adb tool to copy the latest version of your app to the emulator. Let’s see
how things work now.

Here’s the code

that your app uses
in response to an
athlete selection.

Tap!

Success.

Your app displays the selected

athlete’s top three times on
screen. How cool is that?

you are here 4   287

file transfer over wifi

Run your app on a real phone

Now that your app is running successfully on your emulator, it’s time to try it
on a real phone. This is where things get interesting.
These
There are many options when it comes to copying your code to a real device:
instructions
• Use file transfer over Bluetooth. do not work
on the
• Use file transfer with a USB connection.
emulator.
• Use the Android SDK’s adb tool with USB.
The Android emulator
• Use a file transfer tool over WiFi. does not currently support
Google’s Android Market,
Unfortunately, which technique to use (and which work) depends very much
which you’ll need access to
on your phone.
use when following along
At Head First Labs, we’ve had the greatest and most consistent success with with the instructions on
the last option: use a file transfer tool over WiFi. these pages.

Step 1: Prepare your computer

To transfer files securely between your Android phone and your computer,
enable SSH file transfers by running an SSH server on your computer. How
you do this depends on the operating system you are running:
• Windows: download one of the many free SSH servers.
• Mac OS X: enable remote logins.
• Linux: install and enable OpenSSH Server.

Step 2: Install AndFTP on your Android phone

Use the Android Market on your phone to find and install the AndFTP app.
This excellent tool lets you transfer files to and from your Android phone over
FTP, SFTP, and FTPS.
To use it with the SSH server running on your computer, you’ll want to select
SFTP as the file transfer protocol within the app, because AndFTP defaults
to using the FTP protocol.

The AndFTP app is

Let’s take a look at what’s involved. one of our faves.

288   Chapter 8
 mobile app development

Configure AndFTP
With AndFTP running on your phone, configure it to connect to your
computer (Hostname) using SFTP as the transfer protocol (Type). Leave the Port,
Username, Password, and Remote dir entries as they are, but change the Local dir
entry to /sdcard/sl4a/scripts.

Change this entry to

be the web name or
address of your SSH Be sure to set
server. this to “SFTP”.
The value for
“Port” should
change to 22.

Set this to “/sdcard/sl4a/sc

which ensures files transferredripts”
from your server are added to
SL4A.
Be sure to tap “Save”.

With the connection set up, tap AndFTP’s Connect button to establish a
connection to your SSH server, entering your Username and Password when
prompted.
With the connection to the server established, navigate to the server folder
containing the file(s) you want to transfer to the phone, mark the files for
download, and tap the Download button.
Your app
When the download completes, click Disconnect to terminate the connection is ready!
between the phone and your computer. If you transferred a Python program,
it should now be added to the list of scripts within SL4A.

It’s time to let Coach Kelly take a look.

you are here 4   289

app complete (almost)

The coach is thrilled with his app

That’s looking great! I knew you could do it...

now all I need is a way to add a new timing
value directly from my phone. That would
be awesome!

The coach’s app

running on the
coach’s phone.

Welcome to the future!

You’ve delivered a solution that automates interaction with your website while
providing a modern interface on an Android phone. Your app allows your
users to access web data directly on their mobile device.
The fact that your server code runs on Python 3 and your Android client code
runs on Python 2 makes very little difference: it’s all just Python code, after all.
All that’s left to do is write some code to satisfy Coach Kelly’s latest request,
and you’ll get to that in the next chapter.

This is great work.

290   Chapter 8
 mobile app development

Your Python Toolbox

You’ve got Chapter 8 under your

CHAPTER 8
belt and you’ve added some key
Python techiques to your toolbox.

th o n L i n g o
Py
ƒƒ The json library module lets you
s release
the previou
convert Python’s built-in types to the text-

• “Pyth
on 2” - has compatibility based JSON data interchange format.
which ot
of Python, Python 3 (and are n ƒƒ Use json.dumps() to create a
h
“issues” wit ting worked up over).
stringed version of a Python type.

worth get ƒƒ Use json.loads() to create a

Python type from a JSON string.
ƒƒ Data sent using JSON needs to
have its Content-Type: set to
Android Lingo application/json.
ƒƒ The urllib and urllib2 library
• “SL4A” - the Scripting Layer modules (both available in Python 2)
for Android lets you run Python can be used to send encoded data from
your Android device. on a program to a web server (using the
urlencode() and urlopen()
• “AVD” - an Android
Virtual functions).
Device which lets you emulate yo
Android device on your computer.ur ƒƒ The sys module provides the sys.
stdin, sys.stdout and sys.
stderr input streams.

you are here 4   291

9 manage your data

Handling input

Input this, input that...that’s

all I ever hear...input, input,
input, input...all day long. It’s
enough to drive me mad!

The Web and your phone are not just great ways to display data. 
They are also great tools to for accepting input from your users. Of course, once your
webapp accepts data, it needs to put it somewhere, and the choices you make when
deciding what and where this “somewhere” is are often the difference between a webapp
that’s easy to grow and extend and one that isn’t. In this chapter, you’ll extend your webapp
to accept data from the Web (via a browser or from an Android phone), as well as look at
and enhance your back-end data-management services.

this is a new chapter   293

add data anywhere

Your athlete times app has gone national

We love what you

did for Coach Kelly, but it
would be great if we could
add times for an athlete
no matter where we are. Is
this possible?

The National Underage Athletics Committee (NUAC) took one look at your
Android app and realized it’s just what they need…almost.
There are many ways to improve your webapp, but for now, let’s concentrate
on the committee’s most pressing need: adding a new time value to an existing
athlete’s data set.
Adding new data to text files isn’t going to work: there are just too many
coaches around the country adding data. The committee wants something
that’s user friendly from any web browser or Android phone.

Can you help?

294   Chapter 9
 manage your data

Use a form or dialog to accept input

...or if you are on

Simply use the standard your phone, a call to
<FORM> and <INPUT> tags the “dialogGetInput()”
within your HTML web page to function will do the trick.
get input from your users...

On the Web, your user interacts with your web form and enters data. When
she presses the submit button, the web browser gathers up all of the form’s
data and sends it to the web server as part of the web request.
On your Android phone, you can use the dialogGetInput() method to
get input from the user, then mimic the behavior of the web form’s submit Here’s where the
button in code. data is included
In fact, you’ve done this already: check out this line of code from your
with the web
coachapp.py app, which sends the selected athlete name to your web
request.
server:

athlete = json.loads(send_to_server(web_server + get_data_cgi, {'which_athlete': which_athlete}))

you are here 4   295

form action

Create an HTML form template

Let’s extend yate.py to support the creation of a HTML form. Take a look
a this simple form, together with the HTML markup used to produce it.

The name of the CGI

script to send the
form’s data to.

<form action="cgi-bin/process-time.py" method="POST">

Enter a timing value:
<input type="Text" name="TimeValue" size=40>
<br />

Click the “Send” <input type="Submit" value="Send">

button to submit the </form>

form’s data to your
web server.

When your user clicks on the Send button, any data in the input area is sent
to the web server as part of the web request.
On your web server, you can access the CGI data using the facilities provided
by the standard library’s cgi module:

import cgi

Get the data sent form = cgi.FieldStorage()

from the form as timing_value = form["TimeValue"].value
part of the web
request. the “TimeValue” key from
Access the value associated with
the form’s data.

The cgi module converts the data associated with the web request into a
dictionary-like object that you can then query to extract what you need.

296   Chapter 9
 manage your data

Let’s turn the HTML form from the previous page into a template within the yate.py module.
1 Start by creating a new template called templates/form.html that allows you to
parameterize the form’s CGI script name, method, input tags, and submit button text:

With the template ready, write the code for two functions you intend to add to yate.py.
2
The first, called create_inputs(), takes a list of one of more strings and creates HTML <INPUT> tags
for each string, similar to the one that accepts TimeValue on the previous page.
The second, called do_form(), uses the template from Part 1 of this exercise together with the create_
inputs() function to generate a HTML form. Given a list of <INPUT>
def create_inputs(inputs_list): tag names.

Return the
generated tags to The name of the CGI script
the caller. return(html_inputs) <INPUT> tag names are requirand a list of
ed arguments.
def do_form(name, the_inputs, method="POST", text="Submit"):

The HTTP method and texte

to the “Submit” button hav
sensible default values.

Substitute the
arguments and
generated <INPUT>
tags into the return(form.substitute(cgi_name=name, http_method=method,
template to create
the form. list_of_inputs=inputs, submit_text=text))

you are here 4   297

html form template

You were to turn the HTML form into a template within the yate.py module.
1 You were to start by creating a new template called templates/form.html that allows you
to parameterize the form’s CGI script name, method, input tags, and submit button text.

The CGI script’s

<form action=$cgi_name method=$http_method> The list of <INPUT>
name and associated tags and the submit
Enter a timing value: $list_of_inputs<br /> button’s text is also
HTTP method are parameterized.
parameterized. <input type="Submit" value=$submit_text></form>
With the template ready, you were to write the code for two functions to add to yate.py.
2
The first, called create_inputs(), takes a list of one of more strings and creates HTML <INPUT> tags
for each string, similar to the one that accepts TimeValue.
The second, called do_form(), uses the template from Part 1 of this exercise together with the create_
inputs() function to generate a HTML form:

def create_inputs(inputs_list): This “continuation”

character lets you split
html_inputs = '' a long line of code over
multiple lines.
Take each name for each_input in inputs_list:
and create an
<INPUT> tag. html_inputs = html_inputs + ‘<input type= “Text" name="' + \
each_input + '" size=40>'

return(html_inputs)

def do_form(name, the_inputs, method="POST", text="Submit"):

Grab the with open(‘templates/form.html') as formf:

template from form_text = formf.read()
your disk.
inputs = create_inputs(the_inputs)
Create the list of <INPUT> ta
Create a form = Template(form_text) gs.
template form.
return(form.substitute(cgi_name=name, http_method=method,
list_of_inputs=inputs, submit_text=text))

298   Chapter 9
 manage your data

Test Drive
Here’s the code to a CGI script called cgi-bin/test-form.py, which generates the HTML form
from earlier. As you can see, there’s nothing to it.

#! /usr/local/bin/python3
Always start with a
CGI response. Dynamically create the form, ed.
import yate supplying any arguments as requir
print(yate.start_response('text/html'))
print(yate.do_form('add_timing_data.py', ['TimeValue'], text='Send'))

Set the executable bit (if required on your OS) using chmod + x test_form.py , and then use
your browser to confirm that your HTML form-generating code is working.

Enter the URL for the CGbarI script into

your web browser’s location .

Use your browser’s

“View Source” menu
option to confirm that
the generated form is
exactly what you need.

The generated HTML

form appears within
the browser’s window.

Great. You’ve extended yate.py to support the creation of a simple data

entry form. Now all you need to do is to decide what happens once the data
arrives on your server.

you are here 4   299

data delivery

The data is delivered to your CGI script

In addition to running your webapp, the web server also arranges to deliver
any submitted form data to your waiting CGI script. Python’s cgi library
converts the data into a dictionary and, as you already know, provides you
with convenient access to the submitted data:
All of your form’s data
import cgi has been added to the
“form” dictionary.
form = cgi.FieldStorage()

Additional information about the web request is also available to you via the
web server’s environment. Typically, you won’t need to access or use this data
directly. However, occasionally, it can be useful to report on some of it.
Here is some code that takes advantage of Python’s built-in support for
querying your CGI script’s environment using the os library, assuming the
environment values have been set by a friendly web server. Note that the data
in the enviroment is available to your code as a dictionary.

import os Be sure to include the

import time “os” library in your list of
import sys imports.

addr = os.environ['REMOTE_ADDR']
Query three environment variab les
les.
host = os.environ['REMOTE_HOST']
and assign their values to variab
method = os.environ['REQUEST_METHOD']

cur_time = time.asctime(time.localtime()) Get the current time.

Display the
queried data
print(host + ", " + addr + ", " + cur_time + ": " + method, file=sys.stderr)

on standard
error.

Let’s exploit both code snippets on this page to log the data sent from a form
to your web server’s console. When you are convinced that the data is arriving
at your web server intact, you can extend your code to store the received data
in your model.

Let’s write a CGI to display your form’s data.

300   Chapter 9
 manage your data

CGI Magnets
You need a new CGI script called add_timing_data.py,
which processes the data from a form and displays the data on
your web server’s console screen. The CGI needs to query the
environment, arranging to display the logged data on one line.
The code exists, but most of it is all over the floor. Rearrange the
magnets to produce a working program.

Don’t forget
this line if you #! /usr/local/bin/python3
are running on
Max OS X or import cgi
There’s not much of a
Linux. import os response for now…so
import time just send back plain
import sys text to the waiting
import yate
web browser.

print(yate.start_response('text/plain'))
There’s really addr = os.environ['REMOTE_ADDR']
nothing new host = os.environ['REMOTE_HOST']
here.
method = os.environ['REQUEST_METHOD']
cur_time = time.asctime(time.localtime())
print(host + ", " + addr + ", " + cur_time + ": " + method + ": ",
end='', file=sys.stderr)

print('OK.')
form = cgi.FieldStorage()
end=' ',
print(each_form_item + '->' + form[each_form_item].value,
print(file=sys.stderr)
file=sys.s
for each_form_item in form.keys(): tderr)

you are here 4   301

add timing data

CGI Magnets Solution

You need a new CGI script called add_timing_data.py,
which processes the data from a form and displays the data on
your web server’s console screen. The CGI needs to query the
environment, arranging to display the logged data on one line.
The code exists, but most of it is all over the floor. You were to
rearrange the magnets to produce a working program.

#! /usr/local/bin/python3

import cgi
import os
import time
import sys
import yate

print(yate.start_response('text/plain'))
addr = os.environ['REMOTE_ADDR']
host = os.environ['REMOTE_HOST']
method = os.environ['REQUEST_METHOD']
cur_time = time.asctime(time.localtime())
print(host + ", " + addr + ", " + cur_time + ": " + method + ": ",
end='', file=sys.stderr) Ensure that this “print()”
form = cgi.FieldStorage() function does NOT take a
newline.
for each_form_item in form.keys():

print(each_form_item + '->' + form[each_form_item].value,

end=' ', file=sys.stderr)

print(file=sys.stderr)

print('OK.')
Take a newline on standard error.

302   Chapter 9
 manage your data

Test Drive
Let’s use your form-generating CGI script from earlier to try out add_timing_data.py. As you
enter data in the form and press the Send button, watch what happens on the web server’s console.

Enter some
data into your
web form.

The web browser

displays a very File Edit Window Help DisplayingPOSTs

basic response. $ python3 simple_httpd.py

All is “OK”.
Starting simple_httpd on port: 8080
localhost - - [21/Sep/2010 17:34:54] "GET /cgi-bin/test_form.py HTTP/1.1" 200 -
localhost - - [21/Sep/2010 17:34:54] "GET /favicon.ico HTTP/1.1" 200 -
localhost - - [21/Sep/2010 17:35:24] "POST /cgi-bin/add_timing_data.py HTTP/1.1" 200 -
localhost, 127.0.0.1, Tue Sep 21 17:35:24 2010: POST: TimeValue->2.22

The web server’s

logging screen
displays the data
that arrived, as
well as the name
associated with it.

That worked perfectly. The data entered into the form is delivered to your
CGI script on the your server. Your next challenge is to provide the same user
input experience on an Android phone.

you are here 4   303

android data entry

Ask for input on your Android phone

When you ask for user input on Android, the dialog that your user sees looks
something like this example, which asks your user to confirm or change the
web address and port for your server.

The input dialog has

a title. A space for data
entry, when tapped,
bring ups the “soft”
There’s some keyboard.
additional descriptive
text (or message).

A “Cancel” button
An “OK” button lets you change your
confirms the entry. mind.

A single Android call creates this interface for you using the
dialogGetInput() method:

title = 'Which server should I use?'

message = "Please confirm the server address/name to use for your athlete's timing data:"
data = 'http://192.168.1.33:8080'
resp = app.dialogGetInput(title, message, data).result

Pressing the Ok button sets resp to the data entered into the input area. The result of your
Pressing the Cancel button sets resp to None, which is Python’s internal
user’s interaction
null-value.
with the dialog is
assigned to “resp”.
Let’s create some Android data-entry dialogs.

304   Chapter 9
 manage your data

Let’s create a small Android app that interacts with your user twice. The first dialog asks the
user to confirm the web address and port to use for the web server. Assuming your user taps
the OK button on your dialog, a second dialog pops up to request the timing value to send to the
server. As with the first dialog, tapping the OK button continues execution by sending the newly
acquired timing value to the web server. Tapping Cancel at any time causes your app to exit.
Some of the code is provided for you. Your job is to complete the program. Write the code you
think you need under this code, and call your program get2inputsapp.py:

import android
from urllib import urlencode
from urllib2 import urlopen

There’s
nothing new server_title = 'Which server should I use?’

here…you’ve server_msg = "Please confirm the server address/name to use for your athlete's timing data:"

seen all of timing_title = 'Enter data'

this code timing_msg = 'Provide a new timing value:'

before. web_server = 'http://192.168.1.33:8080'

add_time_cgi = '/cgi-bin/add_timing_data.py'

app = android.Android()

def send_to_server(url, post_data=None):

if post_data:
page = urlopen(url, urlencode(post_data))
else:
page = urlopen(url)
return(page.read().decode("utf8"))

you are here 4   305

user interaction

You were to create a small Android app that interacts with your user twice. The first dialog asks
the user to confirm the web address and port to use for the web server. Assuming your user taps
the OK button on your dialog, a second dialog pops up to request the timing value to send to the
server. As with the first dialog, tapping the OK button continues execution by sending the newly
acquired timing value to the web server. Tapping Cancel at any time causes your app to exit.
Some of the code was provided for you. Your job was to complete the program by writing the
code you think you need under this code and call your program get2inputsapp.py.

import android
from urllib import urlencode
from urllib2 import urlopen

server_title = 'Which server should I use?’

server_msg = "Please confirm the server address/name to use for your athlete's timing data:"
timing_title = 'Enter data'
timing_msg = 'Provide a new timing value:'
web_server = 'http://192.168.1.33:8080'
add_time_cgi = '/cgi-bin/add_timing_data.py'

app = android.Android()

The first dialog asks your

def send_to_server(url, post_data=None):

user to confirm the web

if post_data:

address and port to use.

page = urlopen(url, urlencode(post_data))
else:
page = urlopen(url)
return(page.read().decode("utf8"))

resp = app.dialogGetInput(server_title, server_msg, web_server).result

If your user did NOT tap

if resp is not None: on the Cancel button… …the second
dialog asks for
a new timing value.
web_server = resp
resp = app.dialogGetInput(timing_title, timing_msg).result
if resp is not None: Again, if your user did NOT
tap on the Cancel button… …the app sends the
new_time = resp data to the web server.
send_to_server(web_server + add_time_cgi, {‘TimingValue’: new_time})

306   Chapter 9
 manage your data

Test Drive
Let’s copy get2inputsapp.py to the emulator using the adb tool:

tools/adb push get2inputsapp.py /sdcard/sl4a/scripts

The get2inputsapp.py app appears on the list of scripts within SL4A. Go ahead and give it a tap:

Your new app starts,

and you can edit
the web server
address and port.

When you click

on the input
area, Android’s
“soft” keyboard
pops up.
Enter a new
timing value,
and then tap
“Ok”.

File Edit Window Help InputsFromAndroid

$ python3 simple_httpd.py
Starting simple_httpd on port: 8080
localhost - - [21/Sep/2010 17:34:54] "GET /cgi-bin/test_form.py HTTP/1.1" 200 -
localhost - - [21/Sep/2010 17:34:54] "GET /favicon.ico HTTP/1.1" 200 - The web server’s
localhost - - [21/Sep/2010 17:35:24] "POST /cgi-bin/add_timing_data.py HTTP/1.1" 200 -
log confirms
localhost, 127.0.0.1, Tue Sep 21 17:35:24 2010: POST: TimeValue->2.22
192.168.1.33 - - [21/Sep/2010 20:50:30] "POST /cgi-bin/add_timing_data.py HTTP/1.1" 200 -
the data was
localhost, 192.168.1.33, Tue Sep 21 20:50:30 2010: POST: TimingValue->2:56 sent from your
emulator.

Perfect. That’s working, too. Regardless of where your data originates—on

the Web or a phone—your app can send it to your web server.

you are here 4   307

update which dataset?

It’s time to update your server data

Yikes! I think there’s a problem here...

your server data is in two places: within
your pickle and in the NUAC’s text files.
The question is: which one do you update?

Which of your two datasets should you update?

If you update the pickle, the next time the put_to_store()
function runs, your most recent update will vanish as put_
to_store() recreates the pickle from the data in the text files.
That’s not good.
If you update the appropriate athlete’s text file, the data in the
pickle will be stale until put_to_store() runs again. If
another process calls the get_from_store() function in the
meantime, the update to the pickle might not have been applied
and will appear to be missing for anyone reading your data. That’s
not good, either.

Oh, look, how lovely: I

have a new timing value to add to
the system. Who’s going first?

No, no, no,

Update me, then he’s busy. Just
I’ll update him. update me!

Web
Server
[‘Is this the right room
for an argument?’, “No
you haven’t!”, ‘When?’,
“No you didn’t!”, “You
didn’t!”, ‘You did not!’,
‘Ah! (taking out his wallet
and paying) Just the five
minutes.’, ‘You most
certainly did not!’, “Oh
no you didn’t!”, “Oh no
you didn’t!”, “Oh look,
this isn’t an argument!”,
“No it isn’t!”, “It’s
just contradiction!”,
‘It IS!’, ‘You just
contradicted me!’, ‘You
DID!’, ‘You did just
then!’, ‘(exasperated)
Oh, this is futile!!’,
‘Yes it is!’]

Your text file Your pickle file

308   Chapter 9
 manage your data

Avoid race conditions

Of course...I could
write to the text file
and then immediately call
“put_to_store()” to update
the pickle, right?

Yes, that’s one possible solution, but it’s a poor one.

You might think it highly unlikely…but it is possible for another
process to call the get_from_store() function between the text file
update and the pickle recreation, resulting in a short period of data
inconsistency. These types of situations are known as race conditions
and are hard to debug when they occur.
It’s best to keep them from ever happening if you can.
The basic problem here is that you have one update with one piece of data
that results in two file interactions. If nothing else, that’s just wasteful.

Hey, thanks for What update?!?

the update! It’s been all quiet
over here...

[‘Is this the right room

for an argument?’, “No
you haven’t!”, ‘When?’,
“No you didn’t!”, “You
didn’t!”, ‘You did not!’,
‘Ah! (taking out his wallet
and paying) Just the five
minutes.’, ‘You most
certainly did not!’, “Oh
no you didn’t!”, “Oh no
you didn’t!”, “Oh look,
this isn’t an argument!”,
“No it isn’t!”, “It’s
just contradiction!”,
‘It IS!’, ‘You just
contradicted me!’, ‘You
DID!’, ‘You did just

Your up-to-
then!’, ‘(exasperated)
Oh, this is futile!!’,
‘Yes it is!’]

date text file Your temporarily

inconsistent pickle file

you are here 4   309

avoid race conditions

You need a better data storage mechanism

Your initial text files and pickle design is fine when only one user is accessing
the data. However, now that more than one person can access the data at any
time, and from anywhere, your design is in need of improvement. Above all,
you need to avoid that race condition.

This is very Listen, bud, it’s not my fault...until

upsetting...I appear someone, somewhere runs the “put_to_store()”
to be missing an update. function without someone, somewhere else running the
“get_from_store()” function, you’ll have to do without that
data update. I’m not a miracle worker...I just do what
I’m told.

[‘Is this the right room

for an argument?’, “No
you haven’t!”, ‘When?’,
“No you didn’t!”, “You
didn’t!”, ‘You did not!’,
‘Ah! (taking out his wallet
and paying) Just the five
minutes.’, ‘You most
certainly did not!’, “Oh
no you didn’t!”, “Oh no
you didn’t!”, “Oh look,
this isn’t an argument!”,
“No it isn’t!”, “It’s
just contradiction!”,
‘It IS!’, ‘You just
contradicted me!’, ‘You
DID!’, ‘You did just
then!’, ‘(exasperated)
Oh, this is futile!!’,
‘Yes it is!’]

Your inconsistent
and upset pickle file
Web
Server

Q: Surely you should have thought about this problem long Q: So I’m facing a rewrite of large chunks of my code?

A:
ago and designed this “properly” from the start?

A: That’s certainly one way to look at things, and hindsight is

always a wonderful thing! However, programs have a tendency
Let’s see. You did build your program using the MVC pattern,
and you are using Python, so those two facts should take the sting
out of any potential rewrite, assuming a rewrite is what’s required
to start out small, then grow to provide more features, which can here.
introduce complexity. Recall that the coach’s app started life as a
simple “standalone” text-based program, which was then moved
to the Web to support multiple users. Part of the app was then
redeveloped for use on an Android phone. And yes, if we’d known all
of this ahead of time, we might have been designed it differently.

310   Chapter 9
 manage your data

Wouldn't it be dreamy if I could put my

data in only one place and support all my
app’s requirements? But I know it's just a
fantasy...

you are here 4   311

which database management system?

Use a database management system

You need to move away from your text file and pickle combination and use a
real database management system. You have plenty of choices here…

If you want rock-

Back ends based on solid without the
Oracle’s technology corporate bloat, it has
MS SQL Server is
are rock solid. to be PostgreSQL.
used everywhere!
I really like
MySQL and
MariaDB.

All of these fine technologies will work, but they are overkill for your app’s
data requirements. And besides some of these are way beyond the NUAC’s
budget, let alone their ability to set up, run, and maintain such a system.
What you need is something that’s effectively hidden from the NUAC yet lets
you take advantage of what a database management system has to offer.

If only such a technology existed…

312   Chapter 9
 manage your data

Python includes SQLite

Python 3 comes preinstalled with Release 3 of SQLite, a full-featured, zero-
config, SQL-based data management system.
To use SQLite, simply import the sqlite3 library and use Python’s
standardized database API to program it.There’s really nothing to it: no
database setup, no config, and no ongoing maintenance.
With your data stored in SQLite, rewrite your webapp’s model code to use
SQL to access, manipulate, and query your data. You can plan to move
to one of the bigger database systems if and when your application needs
dictate such a move.
SQLite sounds perfect for the NUAC’s data, doesn’t it?

Geek Bits

The material in this chapter assumes

you are comfortable with SQL
database technology. If you are
new to SQL (or just need a quick
refresher), take a look at Head
First SQL, which comes highly
recommended.

[Note from Marketing: Availa ble anywhere

with a
good books are sold and to anyone
valid credit card.]

you are here 4   313

database connection process

Exploit Python’s database API

The Python Database API provides a standard mechanism for
programming a wide variery of database management systems, including
SQLite. The process you follow in your code is the same regardless of which
back-end database you’re using.

Connect
Establish a connection to your
chosen database back end.

Create
Create a cursor to communicate through the
connecton to your data.

Interact
Using the cursor, manipulate your
data using SQL.

Commit Rollback
Tell your connection to apply Tell your connection to abort your
all of your SQL manipulations Poof! SQL manipulations, returning your
to your data and make them data to the state it was in before your
permanent. interactions started.

Close
Destroy the connection to the
When you close your
database back end.
connection, your cursor
is destroyed, too.

314   Chapter 9
 manage your data

The database API as Python code

Here’s how to implement an interaction with a database using the sqlite3
module:

This disk file is used to hold

the database and its tables.
As always, import
the library you
need. import sqlite3

Establish a connection
to a database. connection = sqlite3.connect('test.sqlite')

cursor = connection.cursor()
Create a cursor to
the data.
Execute some SQL. cursor.execute("""SELECT DATE('NOW')""")

Commit any changes,

making them permanent. connection.commit()

Close your connection connection.close()

when you’re finished.

Depending on what happens during the Interact phase of the process, you
either make any changes to your data permanent (commit) or decide to
abort your changes (rollback).
You can include code like this in your program. It is also possible to interact
with you SQLite data from within IDLE’s shell. Whichever option you choose,
you are interacting with your database using Python.
It’s great that you can use a database to hold your data. But what schema
should you use? Should you use one table, or do you need more? What data
items go where? How will you design your database?

Let’s start working on the answers to these questions.

you are here 4   315

design your database

A little database design goes a long way

Let’s consider how the NUAC’s data is currently stored within your pickle.
Each athlete’s data is an AthleteList object instance, which is associated
with the athlete’s name in a dictionary. The entire dictionary is pickled.

{ Sarah: AthleteList
James: AthleteList
Julie: AthleteList
Mikey: AthleteLi
st ... }

The pickled dictionary has any num

AthleteLists within it. ber of

Each AthleteList has the following attributes:

Sarah: AthleteList

The athlete’s name

The athlete’s DOB

The athlete’s list of times

With this arrangement, it is pretty obvious which name, date of birth, and list
of times is associated with which individual athlete. But how do you model
these relationships within a SQL-compliant database system like SQLite?

You need to define your schema and create some tables.

316   Chapter 9
 manage your data

Define your database schema

Here is a suggested SQL schema for the NUAC’s data. The database is called
coachdata.sqlite, and it has two related tables.
The first table, called athletes, contains rows of data with a unique
ID value, the athlete’s name, and a date-of-birth. The second table, called
timing_data, contains rows of data with an athlete’s unique ID and the
actual time value.

This is a new attribute that

coachdata.sqlite should make it easy to guarantee
uniqueness.

CREATE TABLE athletes (

id INTEGER PRIMARY KEY AUTOINCREMENT UNIQUE NOT NULL,
name TEXT NOT NULL,
dob DATE NOT NULL )

CREATE TABLE timing_data (

athlete_id INTEGER NOT NULL,
value TEXT NOT NULL,
FOREIGN KEY (athlete_id) REFERENCES athletes)

e two
Note how this schema “links” th
tables using a foreign key.
There can be one and only one row of data for each athlete in the athletes
table. For each athlete, the value of id is guaranteed to be unique, which
ensures that two (or more) athletes with the same name are kept separate
within the system, because that have different ID values.
Within the timing_data table, each athlete can have any number of time
values associated with their unique athlete_id, with an individual row of
data for each recorded time.

Let’s look at some sample data.

you are here 4   317

athletes and values

What does the data look like?

If the two tables were created and then populated with the data from the
NUAC’s text files, the data in the tables might look something like this.

This is what the data in the

“timing_data” table might s
look like, with multiple row
of data for each athlete and
one row for each timing value.

This is what the data in the “at

hletes”
table might look like, with one row
data for each athlete. of

If you create these two tables then arrange for your data to be inserted into
them, the NUAC’s data would be in a format that should make it easier to
work with.
Looking at the tables, it is easy to see how to add a new timing value for an
athlete. Simply add another row of data to the timing_data table.
Need to add an athlete? Add a row of data to the athletes table.
Want to know the fastest time? Extract the smallest value from the
timing_data table’s value column?

Let’s create and populate these database tables.

There’s more data in this

318   Chapter 9
table than shown here.
 manage your data

SQLite Magnets
Let’s create a small Python program that creates the coachdata.
sqlite database with the empty athletes and timing_data
tables. Call your program createDBtables.py. The code you
need is almost ready. Rearrange the magnets at the bottom of the
page to complete it.

import sqlite3

cursor.execute("""CREATE TABLE athletes (

athlete_id INTEGER NOT NULL,

value TEXT NOT NULL,
FOREIGN KEY (athlete_id) REFERENCES athletes)""")

connection.commit()
connection.close()

cursor = connec
tion.cursor()
dob DATE NOT NU cursor.execute("
LL )""")
""CREATE TABLE
timing_data (

connection = sqlite3.connect('coachdata.sqlite') name TEXT

NOT NULL,

id INTEGER PRIMARY KEY AUTOINCREMENT UNIQUE NOT NULL,

you are here 4   319

create database tables

SQLite Magnets Solution

Your job was to create a small Python program that creates the
coachdata.sqlite database with the empty athletes
and timing_data tables. You were to call your program
createDBtables.py. The code you needed was almost ready,
and you were to rearrange the magnets at the bottom of the page to
complete it.

import sqlite3

connection = sqlite3.connect('coachdata.sqlite')

cursor = connection.cursor()

cursor.execute("""CREATE TABLE athletes (

id INTEGER PRIMARY KEY AUTOINCREMENT UNIQUE NOT NULL,

name TEXT NOT NULL,

dob DATE NOT NULL )""")

cursor.execute("""CREATE TABLE timing_data (

athlete_id INTEGER NOT NULL,

value TEXT NOT NULL,
FOREIGN KEY (athlete_id) REFERENCES athletes)""")

connection.commit()
The commit isn’t always required
connection.close()
most other database systems, butwith
is with SQLite. it

320   Chapter 9
 manage your data

Transfer the data from your pickle to SQLite

As well as writing the code to create the tables that you need, you also need
to arrange to transfer the data from your existing model (your text files and
pickle combination) to your new database model. Let’s write some code to do
that, too.
The data in these variables
You can add data to an existing table with the SQL INSERT statement. is substituted in place of
Assuming you have data in variables called name and dob, use code like this to the “?” placeholders.
add a new row of data to the athletes table:

cursor.execute("INSERT INTO athletes (name, dob) VALUES (?, ?)",(name, dob))

ut supplying a value for

You don’t need to worry aboLit provides one for you
the “id” column, because SQ e
automatically.

Ready Bake Here’s a program, called initDBathletes.py, which takes

your athlete data from your existing model and loads it into your
Python Code newly created SQLite database.

Connect
import sqlite3

to the new
database. connection = sqlite3.connect('coachdata.sqlite')
cursor = connection.cursor()

Grab the
data from import glob
the existing import athletemodel
model. data_files = glob.glob("../data/*.txt")
athletes = athletemodel.put_to_store(data_files)

Get the athlete’s

for each_ath in athletes:

name and DOB

from the pickled name = athletes[each_ath].name
data. dob = athletes[each_ath].dob

Use the INSERT

statement to add cursor.execute("INSERT INTO athletes (name, dob) VALUES (?, ?)", (name, dob))

a new row to the connection.commit()

Make the change(s) permanent.
“athletes” table.
connection.close()

you are here 4   321

names and numbers

What ID is assigned to which athlete?

You need to query the data in your database table to work out which ID value
is automatically assigned to an athlete.
With SQL, the SELECT statement is the query king. Here’s a small snippet of Again, the placeholders indica
code to show you how to use it with Python, assuming the name and dob where the data values are te
variables have values: substituted into the query.

cursor.execute("SELECT id from athletes WHERE name=? AND dob=?", (name, dob))

If the query succeeds and returns data, it gets added to your cursor. You can
call a number of methods on your cursor to access the results:
• cursor.fetchone() returns the next row of data. Each of these cursor
• cursor.fetchmany() returns multiple rows of data. methods return a list
of rows.
• cursor.fetchall() returns all of the data.

Names alone are not enough

anymore...if you want to uniquely
identify your athletes, I need to
know their IDs.

Web
Server

322   Chapter 9
 manage your data

Insert your timing data

You’re on a roll, so let’s keep coding for now and produce the code to take
an athlete’s timing values out of the pickle and add them to your database.
Specifically, you’ll want to arrange to add a new row of data to the
timing_data table for each time value that is associated with each athlete
in your pickle.
Those friendly coders over at the Head First Code Review Team have just
announced they’ve added a clean_data attribute to your AthleteList
class. When you access clean_data, you get back a list of timing values
that are sanitized, sorted, and free from duplicates.The Head First Code
Review Team has excellent timing; that attribute should come in handy with
your current coding efforts.

Grab your pencil and write the lines of code needed to query the
athletes table for an athlete’s name and DOB, assigning the
result to a variable called the_current_id. Write another
your
Again, it’s OK to assume in “do query to extract the athlete’s times from the pickle and add them
code that the “name” and b” to the timing_data table.
variables exist and have values
assigned to them.

you are here 4   323

database queries

You were to grab your pencil and write the lines of code needed
to query the athletes table for an athlete’s name and DOB,
assigning the result to a variable called the_current_id. You
were then to write another query to extract the athlete’s times
from the pickle and add them to the timing_data table.
Query the “athletes”
table for the ID.
cursor.execute(“SELECT id from athletes WHERE name=? AND dob=?”,
Remember: (name, dob))
“fetchone()” It often makes sense
returns a list. to split your execute
statement over multiple
the_current_id = cursor.fetchone()[0] lines.
Take each of
the “clean” for each_time in athletes[each_ath].clean_data:
times and use cursor.execute("INSERT INTO timing_data (athlete_id, value) VALUES (?, ?)”,
it, together
with the ID, (the_current_id, each_time))
within the
SQL “INSERT” Add the ID and the time
statement. As always, make the value to the “timing_
connection.commit() change(s) permanent. data” table.

Do this!

Add the code to your initDBathletes.py code from earlier,

just after the connection.commit()call. Rename your
program initDBtables.py, now that both the athletes
and timing_data tables are populated with data by a single
program.

That’s enough coding (for now). Let’s transfer your pickled data.

324   Chapter 9
 manage your data

Test Drive
You’ve got two programs to run now: createDBtables.py creates an empty database, defining
the two tables, and initDBtables.py extracts the data from your pickle and populates the tables.
Rather than running these programs within IDLE, let’s use the Python command-line tool instead.

If you are running Windows,

replace “python3” with this:
“C:\Python31\python.exe”.
File Edit Window Help PopulateTheTables
$ python3 createDBtables.py
$ python3 initDBtables.py
Be careful $
to run both
programs ONLY
once.

Hello? Something happened there,

didn’t it? I ran the programs but nothing
appeared on screen...how do I know if
anything worked?

you are here 4   325

sqlite manager

SQLite data management tools

When it comes to checking if your manipulations of the data in your
database worked, you have a number of options: Life really is
too short.
a Write more code to check that the database is in the
state that you expect it.
Which can certainly work, but is error-prone, tedious, and way too
much work.

b Use the supplied “sqlite3” command-line tool.

That’s a period,
followed by the
Simply type sqlite3 within a terminal window to enter the SQLite
word “help”.
“shell.” To find out which commands are available to you, type .help
and start reading. The tool is a little basic (and cryptic), but it works.

c Use a graphical database browser.

There are lots of these; just Google “sqlite database browser” for Works great,
more choices than you have time to review. Our favorite is the SQLite but only on
This is what Manager, which installs into the Firefox web browser as an extension. Firefox.
SQLite Manager
looks like.

Great, all of the athletes are in

the “athletes” table.

But how do you integrate your new database into your webapp?

326   Chapter 9
 manage your data

Integrate SQLite with your existing webapp

So...we just need to

change our model code
to use SQLite...but
Joe: This should be easy. We just have to rewrite the code in
what’s involved?
athletemodel.py to use the database, while keeping the API the
same.
Frank: What do you mean by keeping the API the same?
Joe: Well…take the get_from_store() function, for instance. It
returns an AthleteList dictionary, so we need to make sure that
when we update get_from_store() to use our database that it
continues to return a dictionary, just as it’s always done.
Frank: Ah, now I get it: we can query the database, grab all the data,
turn it into a big dictionary containing all of our AthleteList
objects and then return that to the caller, right?
Joe: Yes, exactly! And the best of it is that the calling code doesn’t need
to change at all. Don’t you just love the beauty of MVC?
Frank: Ummm…I guess so.
Joe
Jim: [cough, cough]
Frank: What’s up, Jim?
Jim
Frank Jim: Are you guys crazy?
Joe & Frank: What?!?
Jim: You are bending over backward to maintain compatibility with an
API that exists only because of the way your data model was initially
designed. Now that you’ve reimplemented how your data is stored in
your model, you need to consider if you need to change your API, too.
Joe & Frank: Change our API? Are you crazy?!?
Jim: No, not crazy, just pragmatic. If we can simplify the API by
redesigning it to better fit with our database, then we should.
Joe: OK, but we haven’t got all day, y’know.
Jim: Don’t worry: it’ll be worth the effort.

you are here 4   327

get out of a pickle

Let’s spend some time amending your model code to use your SQLite database as opposed
to your pickle. Start with the code to your athletemodel.py module. Take a pencil and
strike out the lines of code you no longer need.

import pickle

from athletelist import AthleteList

def get_coach_data(filename):
try:
with open(filename) as f:
data = f.readline()
templ = data.strip().split(',')
return(AthleteList(templ.pop(0), templ.pop(0), templ))
except IOError as ioerr:
print('File error (get_coach_data): ' + str(ioerr))
return(None)

def put_to_store(files_list):
all_athletes = {}
for each_file in files_list:
ath = get_coach_data(each_file)
all_athletes[ath.name] = ath
try:
with open('athletes.pickle', 'wb') as athf:
pickle.dump(all_athletes, athf)
except IOError as ioerr:
print('File error (put_and_store): ' + str(ioerr))
return(all_athletes)

328   Chapter 9
 manage your data

def get_from_store():
all_athletes = {}
try:
with open('athletes.pickle', 'rb') as athf:
all_athletes = pickle.load(athf)
except IOError as ioerr:
print('File error (get_from_store): ' + str(ioerr))
return(all_athletes)

def get_names_from_store():
athletes = get_from_store()
response = [athletes[each_ath].name for each_ath in athletes]
return(response)

Remember: there’s no
requirement to maintain
the existing API.

you are here 4   329

out of a pickle

Let’s spend some time amending your model code to use your SQLite database as opposed
to your pickle. Start with the code to your athletemodel.py module. You were to take a
pencil and strike out the lines of code you no longer need.

import pickle

from athletelist import AthleteList

def get_coach_data(filename):
try:
with open(filename) as f:
data = f.readline()
templ = data.strip().split(',') None of this code
return(AthleteList(templ.pop(0), templ.pop(0), templ)) is needed anymore,
because SQLite
provides the data
except IOError as ioerr:
print('File error (get_coach_data): ' + str(ioerr)) model for you.
return(None)

def put_to_store(files_list):
all_athletes = {}
for each_file in files_list:
ath = get_coach_data(each_file)
all_athletes[ath.name] = ath
try:
with open('athletes.pickle', 'wb') as athf:
pickle.dump(all_athletes, athf)
except IOError as ioerr:
print('File error (put_and_store): ' + str(ioerr))
return(all_athletes)

330   Chapter 9
 manage your data

def get_from_store():
all_athletes = {}
try:
with open('athletes.pickle', 'rb') as athf:
all_athletes = pickle.load(athf)
except IOError as ioerr:
print('File error (get_from_store): ' + str(ioerr))
return(all_athletes)

def get_names_from_store():
athletes = get_from_store()
response = [athletes[each_ath].name for each_ath in athletes]
return(response)

This might seem a little

drastic...but sometimes a
redesign requires you to throw
away obsolete code.

you are here 4   331

get names from store

You still need the list of names

Throwing away all of your “old” model code makes sense, but you still need
to generate a list of names from the model. Your decision to use SQLite is
about to pay off: all you need is a simple SQL SELECT statement.

Ready Bake Here’s the code for your new get_names_from_store()

Python Code function:

import sqlite3

db_name = 'coachdata.sqlite'
Connect to the
database.
def get_names_from_store():
connection = sqlite3.connect(db_name)
Extract the cursor = connection.cursor()
data you need. results = cursor.execute("""SELECT name FROM athletes""")
response = [row[0] for row in results.fetchall()]
Formulate a connection.close()
response.
return(response)

Return the list of

names to the caller.
I guess in this case it
actually makes perfect
sense to maintain the
API for this call.

332   Chapter 9
 manage your data

Get an athlete’s details based on ID

In addition to the list of names, you need to be able to extract an athlete’s
details from the athletes table based on ID.

Ready Bake Here’s the code for another new function called
Python Code get_athlete_from_id():

Note the use of the placeh er

A new function def get_athlete_from_id(athlete_id): to indicate where the “athleold
gets the data connection = sqlite3.connect(db_name) id” argument is inserted intote_
associated with cursor = connection.cursor() the SQL SELECT query.
a specific ID.
Get the “name” results = cursor.execute("""SELECT name, dob FROM athletes WHERE id=?""",
and “DOB” values (athlete_id,))
from the athletes (name, dob) = results.fetchone()
table.
results = cursor.execute("""SELECT value FROM timing_data WHERE athlete_id=?""",
Get the list of (athlete_id,))
times from the data = [row[0] for row in results.fetchall()]
“timing_data”
table. response = { 'Name': name,
'DOB': dob, Take the data from both
'data': data, query results and turn it into
Return the 'top3': data[0:3]} a dictionary.
athlete’s data connection.close()
to the caller. return(response)

This function is a more involved than get_names_from_store(), but

not by much. It still follows the API used with working with data stored in
SQLite. This is coming along. nicely.
With the model code converted, you can revisit your CGI scripts to use your
new model API.

Let’s see what’s involved with converting the CGIs.

you are here 4   333

use ids internally

Isn’t there a problem here? The

“get_names_from_store()” function returns a list
of names, while the “get_athlete_from_id()” function
expects to be provided with an ID. But how does the
web browser or the phone know which ID to use when
all it has to work with are the athletes’ names?

That’s a good point: which ID do you use?

Your current CGIs all operate on the athlete name, not
the ID. In order to ensure each athlete is unique, you
designed your database schema to include a unique ID
that allows for your system to properly identify two (or
more) athletes with the same name, but at the moment,
your model code doesn’t provide the ID value to either
your web browser or your phone.
One solution to this problem is to ensure that the athlete
names are displayed to the user within the view, while the
IDs are used internally by your system to unique identify
a specific athlete. For this to work, you need to change
get_names_from_store().

334   Chapter 9
 manage your data

Here is the current code for your get_names_from_store() function. Rather than
amending this code, create a new function, called get_namesID_from_store(),
based on this code but including the ID values as well as the athlete names in its response.
Write your new function in the space provided.

import sqlite3

db_name = 'coachdata.sqlite'

def get_names_from_store():
connection = sqlite3.connect(db_name)
cursor = connection.cursor()
results = cursor.execute("""SELECT name FROM athletes""")
response = [row[0] for row in results.fetchall()]
connection.close()
return(response)

you are here 4   335

get name’s id

Here is your current code for your get_names_from_store() function. Rather than
amending this code, you were to create a new function, called get_namesID_from_
store(), based on this code but including the ID values as well as the athlete names in its
response. You were to write your new function in the space provided.

import sqlite3

db_name = 'coachdata.sqlite'

def get_names_from_store():
connection = sqlite3.connect(db_name)
cursor = connection.cursor()
results = cursor.execute("""SELECT name FROM athletes""")
response = [row[0] for row in results.fetchall()]
connection.close()
return(response)

Arrange to include the

def get_namesID_from_store(): value of “id” in the
SQL “SELECT” query.
connection = sqlite3.connect(db_name)
cursor = connection.cursor()
results = cursor.execute(“““SELECT name, id FROM athletes""")
response = results.fetchall()
connection.close() There’s no need to process
“results” in any way…assign the
return(response) everything returned from
query to “response”.

Remember: when you close your con

also destroyed, so you’ll generate nection, your cursor is
and use “return(results.fetchall( an exception if you try
))”.

336   Chapter 9
 manage your data

Part 1: With your model code ready, let’s revisit each of your
CGI scripts to change them to support your new model. At the

This is the “generate_list.py”

moment, all of your code assumes that a list of athlete names or an
AthleteList is returned from your model. Grab your pencil and
CGI script. amend each CGI to work with athlete IDs where necessary.
#! /usr/local/bin/python3

import glob
import athletemodel
import yate

data_files = glob.glob("data/*.txt")
athletes = athletemodel.put_to_store(data_files)

Note the change to

the title.
print(yate.start_response())
print(yate.include_header("NUAC's List of Athletes"))
print(yate.start_form("generate_timing_data.py"))
print(yate.para("Select an athlete from the list to work with:"))
for each_athlete in sorted(athletes):
print(yate.radio_button("which_athlete", athletes[each_athlete].name))
print(yate.end_form("Select"))
print(yate.include_footer({"Home": "/index.html"}))

”. This “Sharpen” is continued

This is “generate_timing_data.py on the next page, but no
#! /usr/local/bin/python3
peeking! Don’t flip over until
you’ve amended the code on
import cgi
this page.
import athletemodel
import yate

athletes = athletemodel.get_from_store()
form_data = cgi.FieldStorage()
athlete_name = form_data['which_athlete'].value
an ge.
Another title ch
print(yate.start_response())
print(yate.include_header("NUAC's Timing Data"))
print(yate.header("Athlete: " + athlete_name + ", DOB: " + athletes[athlete_name].dob + "."))
print(yate.para("The top times for this athlete are:"))
print(yate.u_list(athletes[athlete_name].top3))
print(yate.para("The entire set of timing data is: " + str(athletes[athlete_name].clean_data) +
" (duplicates removed)."))
print(yate.include_footer({"Home": "/index.html", "Select another athlete": "generate_list.py"}))

you are here 4   337

not done yet

Part 2: You’re not done with that pencil just yet! In addition to
amending the code to the CGIs that support your web browser’s
UI, you also need to change the CGIs that provide your webapp
data to your Android app. Amend these CGIs, too.
This is the
“generate_names.py”
CGI.
#! /usr/local/bin/python3

import json

import athletemodel
import yate

names = athletemodel.get_names_from_store()

print(yate.start_response('application/json'))
print(json.dumps(sorted(names)))

And here is the

“generate_data.py”
CGI. #! /usr/local/bin/python3

import cgi
import json
import sys

import athletemodel
import yate

athletes = athletemodel.get_from_store()

form_data = cgi.FieldStorage()
athlete_name = form_data['which_athlete'].value

print(yate.start_response('application/json'))
print(json.dumps(athletes[athlete_name].as_dict))

338   Chapter 9
 manage your data

Part 1: With your model code ready, you were to revisit each of
your CGI scripts to change them to support your new model. At the

This is the “generate_list.py”

moment, all of your code assumes that a list of athlete names or an
AthleteList is returned from your model. You were to grab your
CGI script. pencil and amend each CGI to work with athlete IDs where necessary.
#! /usr/local/bin/python3
You no longer need the “glob” mo
as “get_nameID_from_store()” dule,
import glob
all this work for you. does
import athletemodel
import yate

data_files = glob.glob("data/*.txt")
get_namesID_from_store()
athletes = athletemodel.put_to_store(data_files)

The “athletes” are now a listgetof

print(yate.start_response()) lists, so amend the code to
print(yate.include_header("NUAC's List of Athletes")) at the data you need.
print(yate.start_form("generate_timing_data.py"))
print(yate.para("Select an athlete from the list to work with:"))
for each_athlete in sorted(athletes):
each_athlete[0], each_athlete[1])
print(yate.radio_button("which_athlete", athletes[each_athlete].name))
print(yate.end_form("Select"))
print(yate.include_footer({"Home": "/index.html"}))

”. The rest of this “Sharpen

This is “generate_timing_data.py radio_button_id() ?!?
Solution” is on the next page.
It looks like you might need
#! /usr/local/bin/python3

a slightly different “radio_

button()” function?!?
import cgi
import athletemodel

Get the athlete’s data

import yate
Use the retur
from the model, which needed, accessinnegd data as
returns a dictionary. the dictionary ke each of
athletes = athletemodel.get_from_store()

get at the athlety/values to

form_data = cgi.FieldStorage()
athlete_name = form_data['which_athlete'].value
e’s data.
athlete = athletemodel.get_athlete_from_id(athlete_id)
print(yate.start_response())
print(yate.include_header("NUAC's Timing Data")) athlete[‘Name'] + “, DOB: " + athlete[‘DOB']
print(yate.header("Athlete: " + athlete_name + ", DOB: " + athletes[athlete_name].dob + "."))
print(yate.para("The top times for this athlete are:"))
print(yate.u_list(athletes[athlete_name].top3)) athlete[‘top3'] str(athlete[‘data'])
print(yate.para("The entire set of timing data is: " + str(athletes[athlete_name].clean_data) +
" (duplicates removed)."))
print(yate.include_footer({"Home": "/index.html", "Select another athlete": "generate_list.py"}))

you are here 4   339

 cgis for android

Part 2: You weren’t done with that pencil just yet! In addition to
amending the code to the CGIs that support your web browser’s
This is the UI, you also needed to change the CGIs that provide your webapp
“generate_names.py” data to your Android app. You were to amend these CGIs, too.
CGI.
#! /usr/local/bin/python3

import json

import athletemodel
import yate

get_namesID_from_store()
names = athletemodel.get_names_from_store()

print(yate.start_response(‘application/json’))
print(json.dumps(sorted(names)))

And here is the

“generate_data.py” The tiniest of changes need to
CGI. #! /usr/local/bin/python3 to these CGIs, because your Andrbe made
is only interested in your webapp’soid app
NOT all of that generated HTML data,
import cgi
.
import json
import sys

import athletemodel
import yate

athletes = athletemodel.get_from_store()

form_data = cgi.FieldStorage()
athlete_name = form_data[‘which_athlete’].value
athlete = athletemodel.get_athlete_from_id(athlete_id)
print(yate.start_response(‘application/json’))
print(json.dumps(athletes[athlete_name].as_dict))

Add this code to A third argument lets you specif

an ID to go with the radio buttony
“yate.py” to support .
the creation of radio
buttons that provide def radio_button_id(rb_name, rb_value, rb_id):
a value for the return('<input type="radio" name="' + rb_name +
button that differs '" value="' + str(rb_id) + '"> ' + rb_value + '<br />')
from the button
text.

340   Chapter 9
 manage your data

Test Drive
Before you run your amended webapp, be sure to move you SQLite database into the top-level
directory of your webapp (that is, into the same folder your index.html file). That way, your
model code can find it, so move it into your webapp’s root folder now. When you are ready, take your
Start (or SQL-powered webapp for a spin.
restart) your
web server. File Edit Window Help StartYourWebEngine
$ python3 simple_httpd.py
Starting simple_httpd on port: 8080

Click on the
link on the home
page.
Display the list
of athlete names
And there’s as radio buttons.
Sally’s timing
data.

That worked well. But what about your Android app?

you are here 4   341

amend for android

You need to amend your Android app, too

Unlike your HTML-based webapp, where all of your code resides and is
executed on your web server, your Android app runs on your phone and it is
programmed to work with a list of names, not a list of names and athlete IDs.
When you run coachapp.py on your emulator, weirdness ensues…

Here’s your current Android app

on the emulator. running

This is weird…instead of
the names, your app is
displaying a list of lists!

And if you tap “Select”, your app

crashes with a “ValueError”. Bu
mmer.

Just like with the CGI scripts, you need to amend you Android app to work
with the data that’s now arriving from your web server—that is, a list of lists
as opposed to a list.

That shouldn’t take too long, should it?

342   Chapter 9
 manage your data

Here is your current coachapp.py code, which you need to amend to support the way your
webapp’s model now works. Grab a pencil and make the necessary changes to this code.

import android, json, time

from urllib import urlencode

from urllib2 import urlopen

hello_msg = "Welcome to NUAC's Timing App"

list_title = 'Here is your list of athletes:'
quit_msg = "Quitting NUAC's App."
web_server = 'http://192.168.1.34:8080'
get_names_cgi = 'cgi-bin/generate_names.py'
get_data_cgi = '/cgi-bin/generate_data.py'

def send_to_server(url, post_data=None):

# There is no change to this code from the previous chapter.

app = android.Android()

def status_update(msg, how_long=2):

# There is no change to this code from the previous chapter.

status_update(hello_msg)
athlete_names = sorted(json.loads(send_to_server(web_server + get_names_cgi)))
app.dialogCreateAlert(list_title)
app.dialogSetSingleChoiceItems(athlete_names)
app.dialogSetPositiveButtonText('Select')
app.dialogSetNegativeButtonText('Quit')
app.dialogShow()
resp = app.dialogGetResponse().result

if resp['which'] in ('positive'):
selected_athlete = app.dialogGetSelectedItems().result[0]
which_athlete = athlete_names[selected_athlete]
athlete = json.loads(send_to_server(web_server + get_data_cgi,{'which_athlete': which_athlete}))
athlete_title = athlete['Name'] + ' (' + athlete['DOB'] + '), top 3 times:'
app.dialogCreateAlert(athlete_title)
app.dialogSetItems(athlete['Top3'])
app.dialogSetPositiveButtonText('OK')
app.dialogShow()
resp = app.dialogGetResponse().result
status_update(quit_msg)

you are here 4   343

support the new model

Here is your current coachapp.py code, which you need to amend to support the way your
webapp’s model now works. You were to grab a pencil and make the necessary changes to this
code.

import android, json, time

from urllib import urlencode
from urllib2 import urlopen

hello_msg = "Welcome to NUAC's Timing App"

list_title = 'Here is your list of athletes:'
quit_msg = "Quitting NUAC's App."
web_server = 'http://192.168.1.34:8080'
get_names_cgi = 'cgi-bin/generate_names.py'
get_data_cgi = '/cgi-bin/generate_data.py'

def send_to_server(url, post_data=None):

# There is no change to this code from the previous chapter.

app = android.Android()
Extract the athlete
names ONLY from the
list of lists.
def status_update(msg, how_long=2):
# There is no change to this code from the previous chapter.

status_update(hello_msg) athletes =
athlete_names = sorted(json.loads(send_to_server(web_server + get_names_cgi)))
app.dialogCreateAlert(list_title) athlete_names = [ath[0] for ath in athletes]
app.dialogSetSingleChoiceItems(athlete_names)
app.dialogSetPositiveButtonText('Select') This is a cool use of a
app.dialogSetNegativeButtonText('Quit') comprehension.
app.dialogShow()
resp = app.dialogGetResponse().result
Determine the ID associated
if resp['which'] in ('positive'): with the selected athlete.
selected_athlete = app.dialogGetSelectedItems().result[0]
which_athlete = athlete_names[selected_athlete] athletes[selected_athlete][1]
athlete = json.loads(send_to_server(web_server + get_data_cgi,{'which_athlete': which_athlete}))
athlete_title = athlete['Name'] + ' (' + athlete['DOB'] + '), top 3 times:'
app.dialogCreateAlert(athlete_title)
app.dialogSetItems(athlete['Top3']) athlete[‘top3']
app.dialogSetPositiveButtonText('OK') A small adjustment to next
app.dialogShow() line is needed to access the
resp = app.dialogGetResponse().result “top3” attribute.
status_update(quit_msg)

344   Chapter 9
 manage your data

Android Pool Puzzle

Your job is to take the code from the pool and place it into
the blank lines in the program. Your goal is to write
the code to have your app provide the user with a
mechanism to add a timing value to the server for
the currently selected athlete. For now, send your
data to the cgi-bin/add_timing_data.py
CGI script.
Hint: the code from get2inputsapp.py (from earlier in
this chapter) should come in handy here.

Add another button to thenexi sting

of your app.
dialog in the current versio
app.dialogSetNegativeButtonText('Add Time')

...
d,
ton that’s tappe
Based on the hibutng (“pass”) or t
st ar a
if resp['which'] in ('positive'):
eit he r d o no t
the user.
new dialog with
pass
elif resp['which'] in ('negative'):

send_to_server(web_server + add_time_cgi,{'Time': new_time, 'Athlete': which_athlete})

If some input is
supplied, send it
to the web server
together with the new_time
= resp
athlete’s ID.
'/cgi-bin/add_timing_data.py'
resp = app.dialog
GetInput(timing_ti
tle, timing_msg).
result
add_time_cgi = 'Provide a new ti
ming value ' + at
hlete['Name'] + ':
timing_msg = '
new time'
te r a
tle = 'En s not Non
e:
timing_ti if resp i

you are here 4   345

allow android input

Android Pool Puzzle Solution

Your job was to take the code from the pool and place it
into the blank lines in the program. Your goal was
to write the code to have your app provide the user
with a mechanism to add a timing value to the server
for the currently selected athlete. For now, you
were to send your data to the cgi-bin/add_
timing_data.py CGI script.
Hint: the code from get2inputsapp.py (from earlier in
this chapter) should come in handy here.

app.dialogSetNegativeButtonText('Add Time')

...

if resp['which'] in ('positive'):
pass
elif resp['which'] in ('negative'): Define the dialog’s
timing_title = 'Enter a new time' titles and specify
timing_msg = 'Provide a new timing value ' + athlete['Name'] + ': ' the CGI to send
add_time_cgi = '/cgi-bin/add_timing_data.py' the data to.

g and wait for

resp = app.dialogGetInput(timing_title, timing_msg).result
Display the diatlo.
if resp is not None: some user inpu
new_time = resp
send_to_server(web_server + add_time_cgi,{'Time': new_time, 'Athlete': which_athlete})

346   Chapter 9
 manage your data

Test Drive
Use the tools/adb command to copy your latest app to the emulator, and give your app a go.

Select “Vera”
from the list
of athletes…
…to see Vera’s top 3
times, then tap the
“Add Time” button…
…to enter a new
time, which is
then sent to
your web server.

File Edit Window Help DataFromAndroid

$ python3 simple_httpd.py
Starting simple_httpd on port: 8080
198.162.1.34 - - [27/Sep/2010 14:51:47] "GET /cgi-bin/generate_names.py HTTP/1.1" 200 -
198.162.1.34 - - [27/Sep/2010 14:52:01] "POST /cgi-bin/generate_data.py HTTP/1.1" 200 -
Great: your data 198.162.1.34 - - [27/Sep/2010 14:52:19] "POST /cgi-bin/add_timing_data.py HTTP/1.1" 200 -
is sent from your localhost, 198.162.1.34, Mon Sep 27 14:52:19 2010: POST: Athlete->3 Time->1.33

Android app to
your web server.

you are here 4   347

database update

Update your SQLite-based athlete data

All that’s left is to change the cgi-bin/add_timing_data.py CGI
script to write your submitted data to your database, as opposed to the web
server’s console screen.
At this point, it’s a trivial exercise, because a single SQL INSERT statement
will do the heavy lifting.

Get the data sent to your b

browser from your Android we
app.

INSERT the data into your

“timing_data” table.

With this version of your CGI script running on your web server, any new
times entered by anyone on an Android phone are added to the data in the
database.
The NUAC no longer has to worry about adding data to text files, because
the files are effectively obsoleted by the use of SQLite.
You’ve produced a robust solution that is more manageable, scalable,
programmable, and extendable. And it’s all thanks to the power of Python,
it’s database API and the inclusion of sqlite3 in the standard library.

All that’s left to do is sit back, relax and bask in the

glory of your latest programming creation…

348   Chapter 9
 manage your data

The NUAC is over the moon!

Of course, your use of SQLite gives you more than just easy insertions
of data. With the NUAC’s data in tables, it’s easy to answer some of the
questions that have been on their mind.

With our data in a

database, it’s a breeze to
work out the fastest time And if we need to
among all our athletes. know who had the
fastest time, that’s
easy, too.

This is just great! I can get

instant answers to my many
questions in the blink of an
eye. All thanks to Python
and SQLite.

To answer these and other queries on the data in the NUAC’s database, you’ll
have to bone up on your SQL. Then it’s up to you to take it from there.
You’ve converted your webapp to use an SQL database. As your data
management needs increase, you can consider alternative heavy-duty data
management technologies as needed.

This is great work. Your webapp is ready for the big time.

you are here 4   349

 python toolbox

Your Python Toolbox

You’ve got Chapter 9 under your
belt and you’ve added some
CHAPTER 9

key Python tools to your evey

expanding Python toolbox.
ƒƒ The fieldStorage() method from
the standard library’s cgi module lets

th on L i n g o you access data sent to your web server

Py -a
API”
from within your CGI script.

“Database mechanism for

ƒƒ The standard os library includes
•
ed se the environ dictionary providing
standardiz n SQL-based databa convenient access to your program’s
accessing a m within a Python environment settings.
system fro ƒƒ The SQLite database system is included
program. within Python as the sqlite3 standard
library.
Database Lingo ƒƒ The connect() method establishes a
connection to your database file.
• “Database” - a collection of one or
more tables. ƒƒ The cursor() method lets you
communicate with your database via an
• “Table” - a collect
ion of one or existing connection.
rows or data, arranged as one or more
columns. more ƒƒ The execute() method lets you send
an SQL query to your database via an
existing cursor.
• “SQL” - the “Struct
ured Query
Language” is the language of th ƒƒ The commit() method makes changes
database world and it lets you wo e to your database permanent.
with your data in your database rk ƒƒ The rollback() method cancels
statements such as CREATE, IN using any pending changes to your data.
and SELECT. SERT,
ƒƒ The close() method closes an
existing connection to your database.
ƒƒ The “?” placeholder lets you parameterize
SQL statements within your Python code.

350   Chapter 9
10 scaling your webapp

Getting real

It all started with the internal

combustion engine, then it was
the electric engine, and now
there’s App Engine. Will this
torture never end?

The Web is a great place to host your app…until things get real. 
Sooner or later, you’ll hit the jackpot and your webapp will be wildly successful. When that
happens, your webapp goes from a handful of hits a day to thousands, possibly ten of
thousands, or even more. Will you be ready? Will your web server handle the load? How
will you know? What will it cost? Who will pay? Can your data model scale to millions
upon millions of data items without slowing to a crawl? Getting a webapp up and running
is easy with Python and now, thanks to Google App Engine, scaling a Python webapp is
achievable, too. So…flip the page and find out how.

this is a new chapter   351

a whale of data

There are whale sightings everywhere

The Head First Whale Watching Group (HFWWG) coordinates the live
cetacean sightings for the entire country. To date, they’ve provided a PDF
form on their website that members of the public can download, fill in, and
mail to the HFWWG central office.
The form contains the essential data needed to record the sighting:

After a busy sightings weekend, the central office is swamped with completed
forms for thousands of sightings…which is a data-entry nightmare as all those
forms can take an age to process manually. There’s nothing worse than being
stuck in front of your computer entering data when all you want to do is be
out on the water looking for humpbacks…

352   Chapter 10
 scaling your webapp

The HFWWG needs to automate

We need to somehow
automate the recording Ideally, a solution that works on
Yeah, but we aren’t
of our sightings... the Web would be great. That
technical, and we don’t
way, anyone from anywhere
have much money.
could record a sighting. Look!
There’s one...

Suggesting to the HFWWG that they invest in an expensive web hosting

solution isn’t going to make you any friends. It’s way too expensive to buy
the capacity they’ll need for the busy weekends and a total waste of capacity
when sightings are infrequent.
Suggesting that they invest in a large, state-of-the-art web server that can be
hosted in the central office is also a nonstarter: there’s no one to look after a
setup like that, and the broadband link required to handle the anticipated
traffic would blow the their budget right out of the water.

Is there another option?

you are here 4   353

enter app engine

Build your webapp with Google App Engine

Google App Engine (GAE) is a set of technologies that lets you host your
webapp on Google’s cloud computing infrastructure.
GAE constantly monitors your running webapp and, based on your webapp’s
current activity, adjusts the resources needed to serve up your webapp’s pages.
When things are busy, GAE increases the resources available to your webapp,
and when things are quiet, GAE reduces the resources until such time as
extra activity warrants increasing them again.
On top of this, GAE provides access to Google’s BigTable technology: a set of
database technologies that make storing your webapp’s data a breeze. Google
also backs up your webapp’s data on a regular basis, replicates your webapp
over multiple, geographically dispersed web servers, and keeps App Engine
running smoothly 24/7.
And the best part? GAE can be programmed with Python.
And the even better part? You can start running your webapp on GAE for free.

That sounds perfect for

the HFWWG. What’s the
catch?

Initially, there isn’t one.

Google provides this webapp hosting service at no
charge and will continue to do so until your webapp
processes five million page views per month. Once it
exceeds this threshold, you’ll need to pay Google for
the extra capacity used. If you never reach the limit,
your use of GAE is not charged.
Five million page views? That’s a lot of sightings…

354   Chapter 10
 scaling your webapp

Download and install App Engine

When your webapp is ready for deployment, you’ll upload it to the Google
cloud and run it from there. However, during development, you can run a test
version of your webapp locally on your computer. All you need is a copy of
the GAE SDK, which is available from here:

http://code.google.com/appengine/ After installation, Windows and

X users will find a nice, graphi Mac OS
end added to their system. cal front
Download the GAE Python SKD for your operating system. Windows, Mac
OS X, and Linux are all supported, and installation is straightforward.
On Linux, a new folder called ter a
“google_appengine” is created af
successful install.
GAE uses Python 2.5
The version of Python built into GAE is a modified version of the Python 2.5
release. As when you worked with Python for Android, the fact that you aren’t
running Python 3 isn’t such a big deal with GAE, although you do need to
ensure Python 2.5 is installed on your computer. Open up a terminal window
and type:

python2.5 -V

If this command gives an error, pop on over to the Python website and grab
the 2.5 release for your operating system.

Q: Aren’t things going backward here? First, there was Python 3, then it was Python 2.6 for Android, and now we are dropping
down to 2.5 for App Engine? What gives?

A: That’s a great question. It’s important to remember to always code to the restrictions placed on you.You might think that it sucks that GAE
runs on Python 2.5, but you shouldn’t. Think of it as just another restriction placed on the code you write—that is, it must target Release 2.5
of Python. As with the Android code you created in the previous chapters, the GAE code you are about to write is not all that different than the
Python code for 3. In fact, you will be hard pressed to spot the difference.

you are here 4   355

testing app engine

Make sure App Engine is working

The environment supported by GAE within the Google cloud supports
standard CGI or Python’s WSGI. To build a GAE-compatible webapp, you
need three things: a folder to hold your webapp’s files, some code to execute,
and a configuration file.
To test your setup, create a folder called mygaetest. Within the folder,
create a small CGI you can use to test GAE. Call this CGI sayhello.py.
Use this code:

print('Content-type: text/plain\n') It doesn’t get much

easier than this…a
print('Hello from Head First Python on GAE!') plain-text message is
displayed within your
browser whenever
The configuration file must be called app.yaml and it, too, must be in
this CGI runs.
your webapp’s folder. This file tells the Google cloud a little bit about your
webapp’s runtime environment. Here’s a basic configuration file:

The “version” line identifies the

The “application” line identifies current version of your webapp
e
your webapp and is the same nam
application: mygaetest
(and usually starts at 1).
as your folder. version: 1
runtime: python
“runtime” tells GAE that your webapp api_version: 1 The “api_version”
is written in and will run on Python. indicates the release of
handlers: GAE you are targeting.
Think of the “handlers” section
the configuration file as a top-leof
- url: /.*

vel script: sayhello.py This entry tells GAE to route

webapp routing mechanism. all requests to your webapp to
your “sayhello.py” program.

Do this! sayhello.py
mygaetest

Go ahead and create the folder called

mygaetest and the two files shown here. app.yaml

356   Chapter 10
 scaling your webapp

Test Drive
The GAE SDK includes a test web server, so let’s use it to take your test GAE webapp for a spin. If you
are running on Windows or Mac OS X, fire up the Google App Engine Launcher front end. This tool
Click this makes it easy to start, stop, and monitor your webapp. On Linux, you’ll need to invoke a command
button to to kick things off. If you are using the GAE Launcher, choose File -> Add Existing Application from
the menu system to browse and select your webapp’s folder. Also: be sure to edit the Launcher’s
start your Preferences to select Python 2.5 as your preferred Python Path.
webapp.

There is no graphical front

end for Linux, so start
your GAE webapp from the
command line.

File Edit Window Help GAEonLinux

$ python2.5 google_appengine/dev_appserver.py mygaetest/

INFO 2010-10-02 12:41:16,547 appengine_rpc.py:149] Server: appengine.google.com

INFO 2010-10-02 12:41:16,555 appcfg.py:393] Checking for updates to the SDK.
INFO 2010-10-02 12:41:17,006 appcfg.py:407] The SDK is up to date.
WARNING 2010-10-02 12:41:17,007 datastore_file_stub.py:657] Could not read datastore data
from /tmp/dev_appserver.datastore
INFO 2010-10-02 12:41:17,104 dev_appserver_main.py:431] Running application mygaetest
on port 8080: http://localhost:8080
This is how the
GAE Launcher
looks on Mac OS
X…it looks similar
on Windows.
With your webapp running and waiting on port 8080, open your favorite web browser and surf on
over to the http://localhost:8080/ web address.

And there it is…

the message from
your test webapp!

you are here 4   357

more work?

I don’t believe it. This is

actually more work than
plain old CGI...and you’re
claiming this is better?!?

Yes, it is more work. But that’s about to change.

For now, this is more work than you’re used to, but remember that
this is just a quick test to make sure your GAE test environment is
up and running (and it is). When you start to work with some of
GAE’s web development features, you’ll initially see that there’s a
lot more going on behind the scenes than meets the eye.

358   Chapter 10
 scaling your webapp

App Engine uses the MVC pattern

Google has built GAE to conform to the familiar Model-View-Controller
The Model
(MVC) pattern.
The View
Like your webapp from the previous chapter, the model component of a
GAE-enabled webapp uses a back-end data storage facility that’s known as
the datastore. This is based on Google’s BigTable technology, which provides
a “NoSQL” API to your data, as well as a SQL-like API using Google’s
Query Language (GQL).
GAE’s views use templates, but unlike the simple string templates from the
previous chapter, GAE uses the templating system from the Django
Project, which is one of Python’s leading web framework technologies. In
addition to templates, GAE includes Django’s forms-building technology.
And, of course, any controller code is written in Python and can use the The Controller
CGI or WSGI standards. Unfortunately, you can’t use your yate module
with GAE, because it is a Python 3 library (and would need to be extensively
rewritten to support Python 2). Not to worry: the facilities provided by GAE
“out of the box” are more than enough to build great webapps.

So...like any other webapp that I build,

with App Engine I define a model for my data,
create some templates for my view, and then
control it all with code, right?

Yes, it’s the same process as any other webapp.

Google has worked hard to ensure that the move to App Engine
is as painless as possible. If you understand MVC (as you now
do), you are well on your way to creating with GAE. It’s just a
matter of working out how GAE implements each of the MVC
components.

you are here 4   359

model data

Model your data with App Engine

App Engine refers to data items stored within its datastore as properties, which
are defined within your model code.
Think of properties as a way to define the name and types of data within
your database schema: each property is like the column type associated piece
of data stored in a row, which App Engine refers to as an entity.

When you think “row,” I

think “entity.” And when
your think “column,” I
think “property.” Get it?

As with traditional SQL-based databases, your GAE datastore properties are

For the full list of propertyr
of a specific, predeclared type. There are lots to choose from, for instance:
• db.StringProperty: a string of up to 500 characters types supported, pop on ove
to http://code.google.com/ store/
appengine/docs/python/datal and
• db.Blob: a byte string (binary data)
• db.DateProperty: a date typesandpropertyclasses.htm
• db.TimeProperty: a time,
take a look.
• db.IntegerProperty: a 64-bit integer
• db.UserProperty: a Google account
Store this data as
ple data from
Here’s some samap “db.StringProperty”.
ter.
the prevoius ch

This data is stored as

This data is stored as a “db.DateProperty”.
a “db.IntegerProperty”.

360   Chapter 10
 scaling your webapp

Pool Puzzle
Your job is to take the properties from the
pool and place them in the correct
place in the class code, which is in a
file called hfwwgDB.py. Your goal
is to assign the correct property type
to each of the attributes within your
Sighting class.

Import the “db”

module from the GAE
extensions. from google.appengine.ext import db

Create a class called “Sighting”

class Sighting(db.Model): from the GAE “db.Model” cla that inherits
ss.
name =
email =
date =
Each property time =
is assigned to a
name. location =
fin_type =
whale_type =
blow_type =
wave_type =

)
db.StringProperty(
db.StringProperty(
)
db.DateProperty()
db.TimeProperty()
db.StringProperty() )
db.StringProperty(
)
db.StringProperty(
) db.StringProperty(
db.StringProperty(
)

you are here 4   361

property types

Pool Puzzle Solution

Your job was to take the properties from
the pool and place them in the correct
place in the class code, which is in a
file called hfwwgDB.py. Your goal
was to assign the correct property
type to each of the attributes within
your Sighting class.

from google.appengine.ext import db

class Sighting(db.Model):
name = db.StringProperty()
email = db.StringProperty()
date = db.DateProperty() Everything is a
db.TimeProperty()
“StringProperty”,
time =
except the “date” and
location = db.StringProperty() “time” fields.
fin_type = db.StringProperty()
whale_type = db.StringProperty()
blow_type = db.StringProperty()
wave_type = db.StringProperty()

362   Chapter 10
 scaling your webapp

What good is a model without a view?

GAE not only lets you define the schema for your data, but it also creates
the entities in the datastore. The first time you go to put your data in the
datastore, GAE springs to life and makes room for your data. There’s no
extra work required by you, other than defining your model in code. It’s
useful to think of GAE as executing something similar to a SQL CREATE
command on the fly and as needed. But how do you get data into the GAE
datastore?
The short answer is that you put it there, but you first need to get some data
from your webapp’s user…and to do that, you need a view. And views are
easy when you use templates.

App Engine templates in an instant

Recall that the templating technology built into GAE is based on technology
from the Django Project. Django’s templating system is more sophisticated
than the simple string-based templates used in the previous chapter. Like your
templates, Django’s templates can substitute data into HTML, but they can
also execute conditional and looping code.
Here are four templates you’ll need for your HTWWG webapp. Two of them
should be familiar to you: they are adaptions of those used in the previous
chapter. The other two are new. Go ahead and grab them from this book’s
support website. As you can see, rather that using the $name syntax for
variable substitution in the template, Django uses the {{name}} syntax:
footer.html <p>

header.html <html>
{{ links }}
</p>
<head> </body>
<title>{{ title </html>
}}</title>
</head>
<body>
<h1>{{ title }}</
h1> <form method="PO
ST" action="/">
<table>

form_start.html
form_end.html sub_title }}"></td>
</tr>
><td><input type="submit" value="{{
<tr><th>&nbsp;</th
</table>
</form>

you are here 4   363

 use a template

Use templates in App Engine

To use a template, import the template module from google.
appengine.ext.webapp and call the template.render() function.
It is useful to assign the output from template.render() to a variable,
which is called html in this code snippet:

As usual, start with your import

from google.appengine.ext.webapp import template
.
html = template.render('templates/header.html', {'title': 'Report a Possible Sighting'})

Call “template. ps values to

render()”… …supplying the …as well as a dictionary thatle. ma
template name… the named template variab
This is similar to the mechanism your yate.py module uses to parameterize
the data displayed within your HTML pages.

And I can use a bunch of calls

like this to create the view
that I need for the HTWWG
sightings form, right?

Yes, create your view with templates.

Just like the other webapps that you’ve built, you
can create your view in much the same way using
Python code. It’s a bummer that you can’t use your
yate.py module, but Django’s templates provide
most of the functionality you need here.

Q: Should I create one big template for my entire web page?

A: You could, if you want. However, if you build up your view from snippets of HTML in templates, you open up the possibility of reusing those
HTML snippets in lots of places. For instance, to maintain a consistent look and feel, you can use the same header and footer template on all of
your web pages, assuming of course that your header and footer aren’t already embedded in an entire web page (which can’t be reused).

364   Chapter 10
 scaling your webapp

1 Let’s write the rest of the code needed to create a view that
displays a data entry form for your HFWWG webapp.
In addition to your web page header code (which already exists
and is provided for you), you need to write code that starts a new
form, displays the form fields, terminates the form with a submit
button, and then finishes off the web page. Make use of the
templates you’ve been given and (here’s the rub) do it all in no

This code goes into a

more than four additional lines of code.

new program called

“hfwwg.py”.
from google.appengine.ext.webapp import template

html = template.render('templates/header.html', {'title': 'Report a Possible Sighting'})

Extend the contents of “html”

the rest of the HTML you nee with
html = html + d.

Remember: no
more than 4
lines of code!

2 Now that you have attempted to write the code required in no more than four lines of code, what
problem(s) have you encountered. In the space below, note down any issue(s) you are having.

you are here 4   365

data-entry display

1 You were to write the rest of the code needed to create a view
that displays a data entry form for your HFWWG webapp.
In addition to your webpage header code (which already exists
and is provided for you), you were to write code with starts a
new form, displays the form fields, terminates the form which a
submit button, then finishes off the webpage. You were to make
use of the templates you’ve been given and (here’s the rub) you
had to do it all in no more than four more lines of code.

from google.appengine.ext.webapp import template

html = template.render('templates/header.html', {'title': 'Report a Possible Sighting'})

The “render()” function always exp

don’t need the second one, be sur ects two arguments. If you
e to pass an empty dictionary.
html = html + template.render('templates/form_start.html’, {})

This is an issue,
isn’t it? # We need to generate the FORM fields in here…but how?!?

html = html + template.render(‘templates/form_end.html’, {‘sub_title’: ‘Submit Sighting’})

html = html + template.render(‘templates/footer.html’, {‘links’: ''})

2 Having attempted to write the code required in no more than four lines of code, you were to make a
note of any issue(s) you encountered.

This is IMPOSSIBLE to do in just four lines of code, because there’s no way

to generate the FORM fields that I need. I can’t even use the “do_form()”
function from “yate.py”, because that code is not compatible with Python 2.5…
this just sucks! You may have written som
ething like this…assuming, of course, you haven’t
thrown your copy of this book out the near est window in frustration. §

366   Chapter 10
 scaling your webapp

Wouldn't it be dreamy if I could avoid

hand-coding a <FORM> and generate the
HTML markup I need from an existing data
model? But I know it's just a fantasy…

you are here 4   367

more borrowing from django

Django’s form validation framework

Templates aren’t the only things that App Engine “borrows” from Django.
It also uses its form-generating technology known as the Form Validation
Framework. Given a data model, GAE can use the framework to generate the
HTML needed to display the form’s fields within a HTML table. Here’s an
example GAE model that records a person’s essential birth details:

This code is in a file

from google.appengine.ext import db

called “birthDB.py”.
class BirthDetails(db.Model):
name = db.StringProperty()
date_of_birth = db.DateProperty()
time_of_birth = db.TimeProperty()

This model is used with Django’s framework to generate the HTML markup
needed to render the data-entry form. All you need to do is inherit from a
GAE-included class called djangoforms.ModelForm:

ion
Import the forms library in addit
from google.appengine.ext.webapp import template

to your GAE data model.

from google.appengine.ext.db import djangoforms
import birthDB

Create a new class by inheriting

class BirthDetailsForm(djangoforms.ModelForm): “djangoforms.Model” class, and from the
class Meta: class to your data model. th en link your new
model = birthDB.BirthDetails
...
html = template.render('templates/header.html', {'title': 'Provide your birth details'})

Use your new class to generate

html = html + template.render('templates/form_start.html', {})
html = html + str(BirthDetailsForm(auto_id=False))
your form.
html = html + template.render('templates/form_end.html', {'sub_title': 'Submit Details'})
html = html + template.render('templates/footer.html', {'links': ''})

in just
som e cod e mis sin g fro m here… but don’t worry: you’ll get toweiten the
There is trate on understanding the link s bet
a moment. For now, just concen ango form validation fra me work.
model, the view code, and the Dj

368   Chapter 10
 scaling your webapp

Check your form

The framework generates the HTML you need and produces the following
output within your browser.

It’s not the

prettiest web page
ever made, but it
works.

Use the View Source menu option within your web browser to inspect the
HTML markup generated.

By setting “auto_id” to “False”

form generator uses your model in your code, the
to identify your form’s fields. property names

enough to create
The Django framework is smyouart input fields (based on
sensible labels for each of l).r
the names used in your mode

It’s time to tie things all together with your controller code.

you are here 4   369

controller code

Controlling your App Engine webapp

Like your other webapps, it makes sense to arrange your webapp controller
code within a specific folder structure. Here’s one suggestion:

to be
Your top-level folder needs ati line
named to match the “applic file.on” Put all of your webapp’s
in your webapp’s “app.yaml” hfwwgapp
controller code and
configuration files in here.

If you have static content, nt,

static put it in here (at the mome
this folder is empty).

Put your HTML

templates templates in here.

As you’ve seen, any CGI can run on GAE, but to get the most out of Google’s
technology, you need to code to the WSGI standard. Here’s some boilerplate Import a utility that
code that every WSGI-compatible GAE webapp starts with: runs your webapp.

Import App from google.appengine.ext import webapp

Engine’s
“webapp” class.
from google.appengine.ext.webapp.util import run_wsgi_app
This class
responds to a
class IndexPage(webapp.RequestHandler):
web request
b request from your web
This method runs when a GET we
def get(self):
Create an browser.
is received by your webapp.
pass
new “webapp”
object This is
for your
app = webapp.WSGIApplication([('/.*', IndexPage)], debug=True)
not unlike
application. switching on
def main(): CGI tracking.
run_wsgi_app(app)
Start your webapp.
if __name__ == '__main__':
Just use these two lines of code
main() as-is.

370   Chapter 10
 scaling your webapp

App Engine Code Magnets

Let’s put everything together. Your model code is already in your hfwwgDB.py
file. All you need to do is move that file into your webapp’s top-level folder. Copy
your templates folder in there, too.Your webapp’s controller code, in a file called
hfwwg.py, also needs to exist in your top-level folder. The only problem is that
some of the code’s all over the floor. Rearrange the magnets to fix things.

from google.appengine.ext import webapp All of the imports

from google.appengine.ext.webapp.util import run_wsgi_app have survived…so
from google.appengine.ext import db there’s no need to
rearrange them.
Let’s test how well
from google.appengine.ext.webapp import template

you’ve been paying

from google.appengine.ext.db import djangoforms

attention. There’s no
guiding lines on the
fridge door.

What’s mis sing html = template.render('templates/header.html', {'title': 'Report a Possible Sighting'})

from in here? html = html + template.render('templates/form_start.html', {})

html = html + template.render('templates/form_end.html’, {'sub_title': 'Submit Sighting'})

html = html + template.render('templates/footer.html', {'links': ''})

app = webapp.WSGIApplication([(‘/.*’, SightingInputPage)], debug=True)

There’s only one small

def main(): change from the boilerplate
run_wsgi_app(app) code in that “IndexPage” is
not being linked to.
if __name__ == '__main__':
main()

class SightingForm(djangoforms.ModelForm): class SightingInputPage(web

app.RequestHandler):
def get(self):
orm())
l + str(SightingF
m
html = ht
ghting class Meta
fwwgDB.Si :
model = h
self.response.ou
t.write(html)
import hfwwgDB

you are here 4   371

everything together

App Engine Code Magnets Solution

Let’s put everything together. Your model code is already in your hfwwgDB.py
file. You were to move that file into your webapp’s top-level folder, as well as copy
your templates folder in there, too.Your webapp’s controller code, in a file called
hfwwg.py, also needs to exist in your top-level folder. The only problem is that
some of the code’s all over the floor. You were to rearrange the magnets to fix
things:

from google.appengine.ext import webapp

from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.ext import db

data
from google.appengine.ext.webapp import template
from google.appengine.ext.db import djangoforms
Import your GAE
model code.
import hfwwgDB

class SightingForm(djangoforms.ModelForm): Use your model to create a

class Meta: form that inherits from thesighting
“django.ModelForm” class.
model = hfwwgDB.Sighting
called
The connected handler class isvid es a
class SightingInputPage(webapp.RequestHandler)
: “SightingInputPage” and it pro ds to a
method called “get” which respon
def get(self):
GET web request.
html = template.render('templates/header.html', {'title': 'Report a Possible Sighting'})
html = html + template.render('templates/form_start.html', {})

html = html + str(SightingForm()) Include the generated form in th

e HTML response.
html = html + template.render('templates/form_end.html’, {'sub_title': 'Submit Sighting'})
html = html + template.render('templates/footer.html', {'links': ''})

self.response.out.write(html) You need to send a

Did you guess this correctlyg? we browser and this line
response back to the waitin b
of code does just that.
app = webapp.WSGIApplication([('/.*', SightingInputPage)], debug=True)

def main():
run_wsgi_app(app)

if __name__ == '__main__':
main()

372   Chapter 10
 scaling your webapp

Test Drive
It’s been a long time coming, but you are now ready to test the first version of your sightings form.
If you haven’t done so already, create an app.yaml file, too. Set the application line to hfwwg
and the script line to hfwwg.py. One final step is to use the Add Existing Application menu option
within the GAE Launcher to select your top-level folder as the location of your webapp.

The launcher adds your

webapp into its list
and assigns it the next
available protocol port—
in this case, 8081.

And here’s your

generated HTML form
in all its glory.

This is looking good. Let’s get a quick opinion from the

folks over at the HFWWG.

you are here 4   373

make it pretty

I know what you’re thinking: “With a shirt like

*that*, how can this guy possibly know anything about
style?”... But let me just say that your form could do
with a bit of, well...color, couldn’t it? Any chance it
could look nicer?

OK, we get it. Web design is not your thing.

Not to worry, you know all about code reuse, right? So, let’s reuse
someone else’s cascading style sheets (CSS) to help with the “look”
of your generated HTML form.
But who can you “borrow” from and not lose sleep feeling guilty over
it?
As luck would have it, the authors of Head First HTML with CSS &
XHTML created a bunch of stylesheets for their web pages and have
made them available to you. Grab a slightly amended copy of some
of their great stylesheets from this book’s support website. When you
unzip the archive, a folder called static appears: pop this entire
folder into your webapp’s top-level folder.
There’s a file in static called favicon.ico. Move it into your
top-level folder.

Improve the look of your form

To integrate the stylesheets into your webapp, add two link tags to your Add these two lines to the
header.html template within your templates folder. Here’s what the top of your “header.html”
tags need to look like: template.

<link type="text/css" rel="stylesheet" href="/static/hfwwg.css" />

<link type="text/css" rel="stylesheet" href="/static/styledform.css" />

GAE is smart enough to optimize the delivery of static content—that is,

content that does not need to be generated by code. Your CSS files are static
and are in your static folder. All you need to do is tell GAE about them to
enable optimization. Do this by adding the following lines to the handers
section of your app.yaml file:

Provide the URL location

for your static content. - url: /static
static_dir: static Switch on the
optimization.
374   Chapter 10
 scaling your webapp

Test Drive
With your stylesheets in place and your app.yaml file amended, ask your browser to reload your form.

Looking good.

A little style goes

a long way...that’s
looking great!

you are here 4   375

 list of choices

Restrict input by providing options

At the moment, your form accepts anything in the Fin, Whale, Blow, and
Wave input areas. The paper form restricts the data that can be provided for
each of these values. Your HTML form should, too.

Anything you can do to cut down on

input errors is a good thing. As the youngest
member of the group, I was “volunteered” to
work on data clean-up duties...

Providing a list of choices restricts what users can input.

Instead of using HTML’s INPUT tag for all of your form fields, you can use the
SELECT/OPTION tag pairing to restrict what’s accepted as valid data for any of the
fields on your form. To do this, you’ll need more HTML markup. That’s the bad
news.
The good news is that the form validation framework can generate the HTML
markup you need for you. All you have to provide is the list of data items to use as
an argument called choices when defining your property in your model code. You
can also indicate when multiple lines of input are acceptable using the multiline
argument to a property.
Apply these changes to your model code in the hfwwgDB.py file.

Define your lists of values

near the top of your code.

This naming _FINS = ['Falcate', 'Triangular', 'Rounded']

convention _WHALES = ['Humpback', 'Orca', 'Blue', 'Killer', 'Beluga', 'Fin', 'Gray', 'Sperm']
helps identify _BLOWS = ['Tall', 'Bushy', 'Dense']
these lists as _WAVES = ['Flat', 'Small', 'Moderate', 'Large', 'Breaking', 'High']
containing Switch on multiple-
constant values.
...
line input.
location = db.StringProperty(multiline=True)
fin_type = db.StringProperty(choices=_FINS)
whale_type = db.StringProperty(choices=_WHALES) Use your lists
of values when
defining your
blow_type = db.StringProperty(choices=_BLOWS)
wave_type = db.StringProperty(choices=_WAVES) properties.

376   Chapter 10
 scaling your webapp

Test Drive
With these changes applied to your model code, refresh your web browser once more.

Your form is not only

looking good, but it’s
more functional, too.

The “location” field

is now displayed over
multiple lines.

Each of the “type”

fields now have
drop-down selection
menus associated
with them.

Your form now looks great! Go ahead and enter some test data, and then
press the Submit Sighting button.

What happens?

you are here 4   377

checking log console

Meet the “blank screen of death”

Submitting your form’s data to the GAE web server produces a blank screen.

Whoops…that’s not
exactly user-friendly.

To work out what happened (or what didn’t

happen), you need to look at the logging
information for your GAE webapp.
If you are running GAE on Linux, your logging
messages are displayed on screen. If you are on
Click!
Windows or Mac OS X, click the Logs button
within the Launcher to open up the Log Console
for your webapp.

Your last web request

Your request resulted in a 405 status code from the web server. According to resulted in a 405.
the official HTTP RFC standards document, 405 stands for:
“Method Not Allowed. The method specified in the Request-Line is not allowed
for the resource identified by the Request-URI. The response MUST include an Allow
Ummm…that’s as clear
header containing a list of valid methods for the requested resource”.
as mud, isn’t it?

378   Chapter 10
 scaling your webapp

Process the POST within your webapp

Listen, bud, I’ll happily
What the 405 status code actually tells you is that posted data arrived at your process your web requests all
webapp intact, but that your webapp does not have any way of processing it. day long...just as long as you
There’s a method missing. give me the methods I need!
Take a quick look back at your code: the only method currently defined is
called get(). This method is invoked whenever a GET web request arrives
at your webapp and, as you know, it displays your sightings form.
In order to process posted data, you need to define another method.
Specifically, you need to add a new method called post() to your
SightingInputPage class.

App Engine handles requests as well as responses App

Your get() method produces your HTML form and returns a web response Engine
to the waiting web browser using the self.response object and by Web
Server
invoking the out.write() method on it.
In additon to helping you with your web responses, GAE also helps you
process your web requests using the self.request object. Here are a few
lines of code that displays all of the data posted to your web server:

Don’t forget to use “self” with The “arguments()” method return

Define a new
all your methods. of the field names used on your s a list
form.
method called
“post”. def post(self):
for field in self.request.arguments():
self.response.out.write(field)
self.response.out.write(': ')
self.response.out.write(self.request.get(field))
self.response.out.write('<br />')

e value associated
The “get()” method returnsldthnam e.
with the provided form fie
So…if you know the name of your form field, you can access its value from
within your webapp using the self.request.get() method.

But what do you do with the data once you have it?

you are here 4   379

 storing data

Put your data in the datastore

Your data is sent to your webapp by GAE and you can use the self.
request.get() method to access each input field value by name. Recall
the BirthDetails model from earlier in this chapter:

This code is in a file

from google.appengine.ext import db

called “birthDB.py”.
class BirthDetails(db.Model):
name = db.StringProperty()
date_of_birth = db.DateProperty()
time_of_birth = db.TimeProperty()

Assume that an HTML form has sent data to your webapp. The data is
destined to be stored in the GAE datastore. Here’s some code to do the heavy
lifting:

def post(self): Create a new “BirthDetails”

new_birth = birthDB.BirthDetails() object to hold your data.

new_birth.name = self.request.get('name') Get each of the

new_birth.date = self.request.get('date_of_birth') form’s data values
new_birth.time = self.request.get('time_of_birth')) and assign them to
your new object’s
Put (save) your data to the GA attributes.
new_birth.put()
datastore. E
Generate a
HTML response html = template.render('templates/header.html', {'title': 'Thank you!'})
to say “thanks.” html = html + "<p>Thank you for providing your birth details.</p>"
html = html + template.render('templates/footer.html',
Send your {'links': 'Enter <a href="/">another birth</a>.'})
response to the
waiting web self.response.out.write(html)
browser.

There’s nothing to it: create a new object from your data model, get the
data from your HTML form, assign it to the object’s attributes, and then use
the put() method to save your data in the datastore.

380   Chapter 10
 scaling your webapp

Based on what you know about how to put your HTML form’s data into the GAE datastore,
create the code for the post() method that your webapp now needs. Some of the code has
been done for you already. You are to provide the rest.

def post(self):

Put your code

here.

html = template.render('templates/header.html',
{'title': 'Thank you!'})
html = html + "<p>Thank you for providing your sighting data.</p>"
html = html + template.render('templates/footer.html',
{'links': 'Enter <a href="/">another sighting</a>.'})

self.response.out.write(html)

you are here 4   381

post to datastore

Based on what you know about how to put your HTML form’s data into the GAE datastore, you
were to create the code for the post() method that your webapp now needs. Some of the
code has been done for you already. You were to provide the rest.

Create a new “Sighting” object.

def post(self):

new_sighting = hfwwgDB.Sighting()

new_sighting.name = self.request.get(‘name’)
new_sighting.email = self.request.get(‘email’)
new_sighting.date = self.request.get(‘date’)
For each of the data
new_sighting.time = self.request.get(‘time’) values received from
the HTML form,
new_sighting.location = self.request.get(‘location’) assign them to the
attributes of the
new_sighting.fin_type = self.request.get(‘fin_type’) newly created object.
new_sighting.whale_type = self.request.get(‘whale_type’)
new_sighting.blow_type =self.request.get(‘blow_type’)
new_sighting.wave_type = self.request.get(‘wave_type’)

e GAE
new_sighting.put() Store your populated object in th
datastore.
html = template.render('templates/header.html',
{'title': 'Thank you!'})
html = html + "<p>Thank you for providing your sighting data.</p>"
html = html + template.render('templates/footer.html',
{'links': 'Enter <a href="/">another sighting</a>.'})

self.response.out.write(html)

382   Chapter 10
 scaling your webapp

Test Drive
Add your post() code to your webapp (within the hfwwg.py file) and press the Back button on
your web browser. Click the Submit Sighting button once more and see what happens this time.
Here’s your
form with the
data waiting to
be submitted.

But when you click

the button, something
bad has happened…your
webapp has crashed.

blem with the

It looks like you might have atypro
doesn’t it?
format of your date proper ,

Phooey…that’s disappointing, isn’t it?

At the very least, you were expecting the data from the form to make it into
the datastore…but something has stopped this from happening. What do you
think is the problem?

you are here 4   383

conservative responses to liberal requests

Don’t break the “robustness principle”

The Robustness Principle states: “Be conservative in what you send; be liberal
in what you accept.” In other words, don’t be too picky when requesting data of
a certain type from your users, but when providing data, give ’em exactly what
they need.
If you make it too hard for your users to enter data into your system, things
will likely things break. For instance, within your model code, consider how
date and time are defined:

A date, and NOTHING ...

but a date will do. date = db.DateProperty() You must provide a
valid value for time.
time = db.TimeProperty() Anything else is simply
... UNACCEPTABLE.

The trouble is, when it comes to dates and times, there are lots of ways to
specify values.

Get the low-down on

the hoedown: quarter I say, old boy, tea is
Oh, la, la.. c’est temps
after six on 6/17/2011. at noon on the first
to toot mon flute! It’s
of each month.
14:00hr on 24/04/2011.

384   Chapter 10
 scaling your webapp

Accept almost any date and time

If you are going to insist on asking your users to provide a properly formatted
date and time, you’ll need to do one of two things:
• Specify in detail the format in which you expect the data.
• Convert the entered data into a format with which you can work.
Both appoaches have problems.
For example, if you are too picky in requesting a date in a particular format,
you’ll slow down your user and might end up picking a date format that is
foreign to them, resulting in confusion.
If you try to convert any date or time entered into a common format that
the datastore understands, you’ll be biting off more than you can chew. As
an example of the complexity that can occur, how do you know if your user
entered a date in mm/dd/yyyy or dd/mm/yyyy format? (You don’t.)

There is a third option

Of course, other webapps
If your application doesn’t require exact dates and times, don’t require them of might not be as fast and
your user. loose with dates and times.
With your sightings webapp, the date and time can be free-format fields that When that’s the case, you’ll
accept any value (in any format). What’s important is the recording of the sighting, need to revert one of the
not the exact date/time it occurred. options discussed earlier on
this page and do the best
you can.
Use “db.StringProperty()” for dates and times
If you relax the datatype restrictions on the date and time fields, not only
do you make is easier on your user, but you also make it easier on you.
For the sightings webapp, the solution is to change the property type for
date and time within the hfwwgDB.py file from what they currently are
to db.StringProperty().

...
date = db.StringProperty() It’s a small change,
time = db.StringProperty() but it’ll make all the
difference.
...

Let’s see what difference this change makes.

you are here 4   385

test drive

Test Drive
Change the types of date and time within htwwgDB.py to db.StringProperty(), being
sure to save the file once you’ve made your edit. Click Back in your web brwoser and submit your
OK, folks… sightings data once more.
let’s try
this again.
Success! It appears to
have worked this time.

Let’s enter another

sighting, just to be
sure.

By relaxing the restrictions you placed on the types of data

you’ll accept, your webapp now appears to be working fine. Go
ahead and enter a few sightings by clicking on the link on your
thank-you page and entering more data.

386   Chapter 10
 scaling your webapp

With a few sightings entered, let’s use App Engine’s included developer console to confirm that the
sightings are in the datastore.

To access the console, enter http://localhost:8081/_ah/admin into your web browser’s

location bar and click on the List Entities button to see your data.

In addition to viewing your existi

datastore, you can use the consolng data in the
test data. e to enter new

There’s all the data your entere

and an “ID” to is in a slightly different order thd, which
App Engine has assigned a “Key” es in handy you might expect. But it’s all in an what
each of your entities, which com a sighting. (App Engine stores your propertiethere.
when you need to uniquely identify alphabetical order, by name.) s in

Your GAE webapp is now ready for prime time.

Before you deploy it to Google’s cloud infrastructure, let’s run it by the folk at
HFWWG to see if they are happy for their webapp to “go live.”

you are here 4   387

restrict to registered users

It looks like you’re not quite done yet

Man, that’s looking good!

There’s just one thing we forgot
to tell you... we are worried
about spam and need to be sure
only registered users can enter a
sighting. Is that a big change?

Is this a big change?

You would imagine that it would be. You’ll have to create an new entity to
hold your registered user login information, and you’ll also need another form
to ask users to provide their registration data (which you’ll need to store in the
datastore). With that in place, you’ll need yet another form to ask your users to log
in, and then you’ll have to come up with a mechanism to restrict only registered
and logged-in users to view your webapp’s pages, assuming you can come up
with something robust that will work…?

Or…as this is GAE, you could just switch on authorization.

388   Chapter 10
 scaling your webapp

Sometimes, the tiniest change can make

all the difference…
The engineers at Google designed App Engine to deploy on Google’s cloud
infrastructure. As such, they decided to allow webapps running on GAE to
access the Google Accounts system.
By switching on authorization, you can require users of your webapp to
log into their Google account before they see your webapp’s pages. If a user
tries to access your webapp and he isn’t not logged in, GAE redirects to the
Google Accounts login and registration page. Then, after a successful login,
GAE returns the user to your waiting webapp. How cool is that?
To switch on authorization, make one small change to your app.yaml file:

application: hfwwgapp
version: 1
runtime: python
api_version: 1

handlers:
- url: /static
static_dir: static

- url: /.*
script: hfwwg.py
That’s all there
login: required
is to it.

Now, when you try to access your webapp, you are asked to log in before proceeding.

This is how the login

screen looks within the
GAE test environment
running on your computer.

you are here 4   389

log login info

Capture your user’s Google ID, too

Now that your webapp requires your users to log in, let’s arrange to capture
the user login information as part of the sighting.
Start by adding the following property to your entity’s list of attributes in your
hfwwgDB.py file. Add it right after the wave_type property.
Create a new
attribute in your which_user = db.UserProperty()
…and set its
“Sighting” class… property type.

Let’s ensure that Django’s form validation framework excludes this new attribute
when generating your HTML form. Within your hfwwg.py file, change your
SightingForm class to look like this:

class SightingForm(djangoforms.ModelForm):

Make sure Django class Meta:

doesn’t include model = hfwwgDB.Sighting

the new attribute exclude = ['which_user']
in your generated
form.

Staying within your hfwwg.py file, add another import statement near the
top of your program: Import GAE’s
from google.appengine.api import users
Google Accounts
API.

In your post() method, right before you put your new sighting to the
datastore, add this line of code:

new_sighting.which_user = users.get_current_user()

When you put your

data to the datastore,
Every time a user adds a sighting to the datastore, GAE ensures that the user’s this code includes
Google Account ID is saved, too. This extra identification information allows the the Google ID of the
HFWWG to track exactly who reported which sighting, and should (hopefully) cut currently logged-in user.
down on the amount of spam your webapp might attract.

All that’s left to do is to deploy your webapp to Google’s cloud.

390   Chapter 10
 scaling your webapp

Deploy your webapp to Google’s cloud

With your webapp developed and tested locally, you are now ready to deploy The “Dashboard” button opens
to the Google cloud. This is a two-step process: register and upload. your web browser and takes you
To register your webapp on the Google cloud, click the Dashboard button to the GAE “My Applications”
on the GAE Launcher. page (after you sign in with
your Google ID).

On Linux, use the

“appcfg.py” script to
deploy.

Enter your webapp’s name in

the box, then click on the
“Check Availability” button.

Don’t use “hfwwgapp” as

that’s already taken. §

r webapp, and then

Optionally, enter a title foron”youbutton.
click the “Create Applicati

Assuming all went according to plan and GAE confirmed that your
application has been created, all that’s left to do is to deploy. Return to the
GAE Launcher and click on the Deploy button. The console displays a
bunch of status message while the deployment progresses. If all is well, you’ll
be told that “appcfg.py has finished with exit code 0”.

Your GAE webapp is now ready to run on Google’s cloud.

you are here 4   391

test drive

Test Drive, on Google

Let’s take your webapp for a spin on Google’s cloud. Open your web browser and surf to a web
address that starts with your webapp’s name and ends in .appspot.com. For the HFWWG webapp, the
web address is http://hfwwgapp.appspot.com. When you first attempt to go to their webapp, App
Engine redirects you to the Google login page.

Provide your Google ID and

password, or sign up for a new
Google account (if you don’t
have one already).

392   Chapter 10
 scaling your webapp

After a successful login, your sighting form appears. Go ahead and enter some test data:

The sighting form served

from the Google cloud is e
EXACTLY the same as th
form served by your test
server.
Return to the http://appeng
com site to log into the consolinee..google.
is a little different than the tesThe UI
but you can use the Datastore Viet console,
confirm that your data has bee wer to
correctly. n stored

Click on this link to see

your data as stored in
the Google cloud.

you are here 4   393

a winning webapp

Your HFWWG webapp is deployed!

I’m all packed up and

ready for a weekend of This is waaaay cool. Look
whale watching. I can’t wait at that fabulous webapp.
to enter my sightings This is exactly what we
data online! need. Super work!

Lovely... I finally have

time to relax, now that I
have no more marathon
data-entry sessions.

This is professional work.

You’ve built a great data-entry webapp and deployed it on Google’s cloud. No
matter how busy things get, whether there’s a handful of sightings per day

Did you even notice that

or tens of thousands, your webapp can handle the load, thanks to Google’s App

you wrote all of your

Engine. And, best of all, the cash-strapped HFWWG doesn’t pay a penny

code using Python 2.5?

until their sightings activity reaches the level of millions of sightings per
month!

394   Chapter 10
 scaling your webapp

Your Python Toolbox

You’ve got Chapter 10 under your

CHAPTER 10
belt and you’ve added more great
Python technology to your ever-
expanding Python toolbox.

ƒƒ Every App Engine webapp must have a

configuration file called app.yaml.
ƒƒ Use the GAE Launcher to start, stop,
monitor, test, upload, and deploy your

App Engine Lingo

webapps.
ƒƒ App Engine’s templating technology

• “Datastore” - the
data repository used is based on the one use in the Django
tly Project.
by Google App Engine to permanen
store your data.
ƒƒ App Engine can also use Django’s Form
Validation Framework.
• “Entity” - the name
used for a “row of ƒƒ Use the self.response object to
data”. construct a GAE web response.

• “Property” - the na
me used for a “data ƒƒ Use the self.request object to
value”. access form data within a GAE webapp.
ƒƒ When responding to a GET request,
implement the required functionality in a
get() method.
ƒƒ When responding to a POST request,
implement the required functionality in a
post() method.
ƒƒ Store data in the App Engine datastore
using the put() method.

you are here 4   395

11 dealing with complexity

Data wrangling
Once I build up a head of
steam, it’s not all that hard
to keep on running, and
running, and running...

It’s great when you can apply Python to a specific domain area. 
Whether it’s web development, database management, or mobile apps, Python helps you
get the job done by not getting in the way of you coding your solution. And then there’s
the other types of problems: the ones you can’t categorize or attach to a domain. Problems
that are in themselves so unique you have to look at them in a different, highly specific way.
Creating bespoke software solutions to these type of problems is an area where Python
excels. In this, your final chapter, you’ll stretch your Python skills to the limit and solve
problems along the way.

this is a new chapter   397

 pace data

What’s a good time goal for the

next race?
The Head First Marathon Club has spent years collecting and
The runner selects a
collating data on their distance runners. Over time, this
distance, say 15K, and
data has helped the club produce a large spreadsheet of pace
we time her over the
data that helps their runners predict their performance over
length of her run.
various distances. The spreadsheet is huge and runs to 50
columns of tightly packed data.
Let’s take a look at the club’s data, as well as the way the
runners and their coach use it.

Here’s a portion of the Marathon

Club’s spreadsheet data.

The timed
distance is 15km.
The predicted
marathon goal.

398   Chapter 11
 dealing with complexity

Yes, she’s wicked fast!

This benchmark allows me

to look up or down the column
to select a target time for
I run the 15K in 45:01. My any other distance, such as a
coach looks up the closest marathon, for example.
match for my time along the
15K row.

This spreadsheet is a little

intimidating…but don’t worry.
You’ll sort it out soon.

you are here 4   399

rain delay

So…what’s the problem?

At the moment, we print our data onto
multiple sheets of paper, which we carry with us.
Most of the time, this works fine. But when it rains
or gets really windy, our pages are either soaked
through or they end up all over the place.

All these sheets are

a pain…especially in
the rain.

Not to mention: forgetting the sheets, keeping the sheets up to date, and
having to flip back and forth through the sheets looking for a closest match.
Of course, word of your newly acquired Python programming skills is
getting around, especially among the running crowd. Ideally, the Marathon
Club needs an Android app that can be loaded onto a bunch of phones and
carried in each coach’s pocket. The app needs to automate the lookup and
distance predictions.

Are you up to the challenge? Do you think you can help?

400   Chapter 11
 dealing with complexity

Start with the data

For now, let’s not worry about creating the Android app; you’ll get to that
soon enough. Instead, let’s solve the central data wrangling problem
and then, when you have a working solution, we’ll worry about porting your Do this!
solution to Android. We’ll begin by getting the data into a format you can
easily process with Python.
Most spreadsheet programs can export data to the widely used CSV format.
The club has done this for you and created a file called PaceData.csv,
Grab a copy of
which includes the data for each of the rows from the original the spreadsheet.
PaceData.csv from this
Here’s a sample of the raw data from the start of the CSV: book’s support website.

is th e col um n hea din gs from th e spreadsheet. They look

The first line of data ly headings that represent estimated maximal oxygen
like numbers but are actual in ml/kg-min) for the race times in each column.
consumption (or VO2 Max, on the timing data, we’ll just treat them as headings.
Because the have no affect

V02,84.8,82.9,81.1,79.3,77.5,75.8,74.2,72.5,70.9,69.4,67.9,66.4,64.9,63.5,62.1,60.7,59.4,58.1,56.8,55.
2mi,8:00,8:10,8:21,8:33,8:44,8:56,9:08,9:20,9:33,9:46,9:59,10:13,10:26,10:41,10:55,11:10,11:25,11:40,1
5k,12:49,13:06,13:24,13:42,14:00,14:19,14:38,14:58,15:18,15:39,16:00,16:22,16:44,17:06,17:30,17:53,18:

The first value on each of the res

of the lines is the timed distance t recorded
The rest of each line is a list of
or row label. run times.

You somehow have to model the data from the CSV file in your
Python program. Can you think of a data structure that might
help here? Justify your selection.

you are here 4   401

multiple associations

You somehow have to model the data from the CSV file in your
Python program. You were to think of a data structure that might
help here? You were also to justify your selection.

The list of headings can be stored in a LIST.

The list of times from each row can also be stored in a LIST, but they
Ummm…
there’s lots also need to be associated with the headings in the very first row of
to think
about here. data, so maybe a DICTIONARY is what’s needed here?

Maybe it’s some COMBINATION of the two?!?

Take another look at the data

The first row of data in the CSV file is the column headings, with the
very first value on the line, the V02 string, being redundant (it won’t ever be
used in this version of the app). The rest of the first line’s data are headings
associated with the time values in each of the columns.
Of course, the data in the columns is also associated with each row, which is
identified by a row label in the first column, such as 2mi, 5k, and so on.
Let’s look at the data in the CSV file again, which has been reformatted to
help highlight the associations.
The column headings are
This string is on the first line.
redundant.
V02, 84.8, 82.9, 81.1, 79.3, 77.5, 75.8, 74.2
2mi, 8:00, 8:10, 8:21, 8:33, 8:44, 8:56, 9:08,
5k, 12:49, 13:06, 13:24, 13:42, 14:00, 14:19, 14:38,

The row labels are

unique to each row The times on each row are associ
of times. with their row label, but ALSO ated
column heading. with a

But can we capture all these associations in code?

402   Chapter 11
 dealing with complexity

Marathon Magnets
Here’s some code that reads the raw data from from the CSV data
file. The column headings from the first line are loaded into a list
called column_headings. The rest of the data (all the rows of
times) are loaded into a dictionary called row_data, with each row
of data keyed with the row label string from the start of each line.
Of course, as luck would have it, someone was cleaning the fridge
door, and they’ve left a bunch of magnets on the floor. See if you
can arrange the magnets into their correct order.
What needs
to go here?

with open('PaceData.csv') as paces:

Process the
“column_headings”
list here.
for each_line in paces:

Process
“row_data”
here.
num_cols = len(column_headings)
print(num_cols, end=' -> ')
With the data
print(column_headings) loaded, this code
lets you check if it’s
num_2mi = len(row_data['2mi']) all OK.
print(num_2mi, end=' -> ')
print(row_data['2mi'])

num_Marathon = len(row_data['Marathon'])
print(num_Marathon, end=' -> ')
print(row_data['Marathon'])

row = each_line.strip().split(',')
column_hea
dings = row_label
row_data[r = row.pop(
ow_label] 0)
= row
row_data = {}

paces.readline().strip().split(',') column_headings.pop(0)

you are here 4   403

read table data

Marathon Magnets Solution

Here’s some code that reads the raw data from from the CSV data
file. The column headings from the first line are loaded into a list
called column_headings. The rest of the data (all the rows of
times) are loaded into a dictionary called row_data, with each row
of data keyed with the row label string from the start of each line.
Of course, as luck would have it, someone was cleaning the fridge
door, and they’ve left a bunch of magnets on the floor. You were to
see if you could arrange the magnets into their correct order.

You need to be sure to create an

row_data = {} dictionary for the row times. empty
with open('PaceData.csv') as paces:
Create the
column headings paces.readline().strip().split(',')
from the first column_headings =
line of data. ip it
column_headings.pop(0)
Read a line from the file, strth en
of unwanted whitespa ce, and
Delete the first split the line on comma .
heading, the for each_line in paces: Process the rest of the file.
“V02”string—you don’t
need it. row = each_line.strip().split(',') It’s the same deal here:
take the line, strip it, and
Extract the row_label = row.pop(0)
then split on comma.
row label. row_data[row_label] = row
Use the row label
num_cols = len(column_headings) together with the rest
print(num_cols, end=' -> ') of the line’s data to
print(column_headings) update the dictionary.

num_2mi = len(row_data['2mi'])
print(num_2mi, end=' -> ')
print(row_data['2mi'])

num_Marathon = len(row_data['Marathon'])
print(num_Marathon, end=' -> ')
print(row_data['Marathon'])

404   Chapter 11
 dealing with complexity

Test Drive
Load your code into IDLE and, with the CSV in the same folder as your code, run it to see what you
get on screen.

Your code
in IDLE.

The output confirms

that each row of
data has 50 data
items.

The column
headings

The “2mi”
row of
data

The
“Marathon”
row of
data

That’s a great start: you’ve managed to read the data from the CSV and put
the headings into a list and the data into a dictionary.

What’s next?

you are here 4   405

link data structures

Did you forget to associate

each time on each row with its
heading? At the moment, the
list and the dictionary are
disconnected...

Yes, the two data structures should be linked.

At the moment, the column_headings list and the row_
data dictionary are not linked in any way, and they need to
be. What we need is some way to connect each of the times in
each row with the heading that tops their column of data.
What options do you have here?
When it comes to linking (or associating) two data items with
each other, the Python dictionary is the data strucutre of
choice, isn’t it?

406   Chapter 11
 dealing with complexity

Store each time as a dictionary

Rather than simply storing each time in the row_data dictionary as a
number, let’s store the data as as a dictionary, with the key set to the time and
the value set to the column heading. That way, you can quickly and easily
determine for any time which column it is associated with, right?
Here’s a portion of what the data structure looks like in Python’s memory
once this association exists:
Instead of a single time value, eac
The “row_data” row item is now an inner dictionarh
dictionary no longer that associates the time with its y
contains a list. column heading.

row_data['2mi']
={ 8:00 8:10 8:21

84.8 82.9 81.1

}
All you need to do is work out how to populate the inner dictionary with the
row data and the associated columns headings…and you’ll have all the data
you need.
The trick in creating the data structure is to realize that each row, including
the column headings, are of a fixed size: 50 items. Knowing this, it’s not much
work to create the dictionary you need:

row_data = {}
with open('PaceData.csv') as paces:
No changes column_headings = paces.readline().strip().split(',')
are needed column_headings.pop(0)
here. for each_line in paces:
Let’s not hard-
code 50, calc the
row = each_line.strip().split(',') value instead. Associate the
row_label = row.pop(0) column heading
Create another, with the time
empty dictionary.
inner_dict = {}
value from the
for i in range(len(column_headings)): row.
inner_dict[row[i]] = column_headings[i]
row_data[row_label] = inner_dict
With each iteration,
“i” is the current
column number. With the dictionary populated, ass
label in “row_data”. ign it to its

you are here 4   407

idle session

Go ahead and add the extra dictionary populating code to your program. Let’s remove all of those print()
statements from the end of your program, because you’ll use the IDLE shell to test your code. Run the code by
pressing F5 or by selecting the Run Module option from the Run menu. Use the dir() BIF to confirm that your
program code executed and that a collection of variables have been created in Python’s namespace:

>>> dir()
['__builtins__', '__doc__', '__name__', '__package__', 'column_headings', 'each_line', 'i',
'inner_dict', 'paces', 'row', 'row_data', 'row_label']
All of your code’s variables exist.

Take another look at (part of ) the spreadsheet data file above, and let’s try and find the column heading
associated with the 43:24 time on the 15k row. Let’s then use the column heading to find the predicted time for a
20k run:

>>> column_heading = row_data['15k']['43:24']

ed as “81.1”.
The associated column heading is correctly identifi
>>> column_heading
'81.1'

>>> prediction = [k for k in row_data['20k'].keys() if row_data['20k'][k] == column_heading]

>>> prediction
['59:03']
A time of “59:03” is correctly predicted, too.

408   Chapter 11
 dealing with complexity

Dissect the prediction code

Let’s take a moment to review what just happened at the bottom of the IDLE
Session from the last page. This line of code is a double-dictionary lookup on the
dictionary-of-dictionaries stored in row_data:
And this is another
dictionary key,
This is a dictionary key which indexes into
within “row_data”. the dictionary at
“row_data[‘15k’]”.
column_heading = row_data['15k']['43:24']

then look up the

Look up the ‘15k’ row data, 3:2 time, which
value associated with the ‘4 hea4’
din g”.
is then assigned to “column_

Working out the predicted time in the 20k row of data involves finding the
key in the row’s dictionary whose value is set to the just-discovered value stored
in column_heading.
You are interested only in data
satisfies this conditional. that

prediction = [k for k in row_data['20k'].keys() if row_data['20k'][k] == column_heading]

.
This is the data you’re searching

A conditional list comprehension is put to good use here. Recall that the list
comprehension syntax is a shorthand notation for a for loop. The loop searches
through the data in the list of keys associated with the dictionary stored at
row_data['20k']. If the value associated with the key (in k) is the same
as column_heading, the value of k is added to the comprehensions results,
which are then assigned to a new list call predicton.

There’s really an awful lot going on in that comprehension.

you are here 4   409

list comprehension

Sweet mother of all things

Python! What’s going on
here? I think my brain is
going to explode...

Don’t let the list comprehension put you off.

Recall that you can always rewrite a list comprehension using
an equivalent for loop…
Ummm…now there’s an idea.

410   Chapter 11
 dealing with complexity

Rewrite each of the list comprehensions on this page to use a for

loop.

times = [t for t in row_data['Marathon'].keys()]

headings = [h for h in sorted(row_data['10mi'].values(), reverse=True)]

time = [t for t in row_data['20k'].keys() if row_data['20k'][t] == '79.3']

you are here 4   411

for loop

You were to rewrite each of the list comprehensions to use a for

loop.

times = [t for t in row_data['Marathon'].keys()]

Start with an times = [] Turn the

empty list. dictionary’s keys
for each_t in row_data[‘Marathon’].keys(): into a list.
times.append(each_t)
time
With each iteration, append the key (which is a
value) onto the “times” list.
headings = [h for h in sorted(row_data['10mi'].values(), reverse=True)]

headings = [] Turn the dictionary’s values into a list…

Start with an
empty list. for each_h in sorted(row_data[‘10mi’].values(), reverse=True):
headings.append(each_h) …being sure to sort the values in
reverse order (biggest first).

With each iteration, append the value (which is a

column heading) onto the “times” list.

time = [t for t in row_data['20k'].keys() if row_data['20k'][t] == '79.3']

time = [] Turn the dictionary’s keys into a

Start with an list.
empty list. for each_t in row_data[‘20k’].keys():
if row_data[‘20k’][each_t] == ‘79.3’:
There’s a time.append(each_t)
definite pattern
emerging here. § With each iteration, check to see if the
column heading (the value part of the
dictionary) equals “79.3” and if it does,
append the time to the list.

412   Chapter 11
 dealing with complexity

Get input from your user

Now that you have your data within a Python data structure, it’s time to ask
your user what it is they are looking for.
Specifically, you need to know three things: the distance run, the time recorded,
and the distance a prediction is required for. Gimme, gimme,
gimme...input() !!!
When you get to move your app onto Android, you can use a nice graphical
dialog to ask your user for input, but for now, let’s quickly create a text-
based user interface, which will allow you to develop and test the rest of the
functionality required from your application. When you’re done, you’ll create
the Android app.

Use input() for input >>>

Python has the input() BIF that can help here, which is used to display
a prompt on screen, and then accept keyboard input, returning what was
entered as a string to your code.

Using the input() BIF is best demonstrated with some examples:

Provide the prompt to display
>>> res = input('What is your favorite programming language: ') to your user.
What is your favorite programming language: Python
>>> res
The entered data is assigned to “res”
'Python'
and it’s a STRING.
The input() BIF returns a string, which has been stripped of any trailing newline character, which would
typically be included at the end of any input string. It is important to note that any input is returned as a string,
regardless of what type of data you think you might be entering:

The entered data is assigned to “age” and >>> age = input('What is your age: ')
it’s a string, even though you might want to What is your age: 21
treat it like it’s a number. >>> age
'21'
[Editor’s note: Yeah…
>>> int(age) dream on, Paul. §]
Convert the input to the tytape. you 21
need BEFORE using the da

you are here 4   413

input error

Getting input raises an issue…

It’s not hard to use input() to get the, um, er…input you need. Here’s your
code from earlier with three calls to input() added to interact with your
user.

There’s nothing to this, as ()”

user-interaction with “input
doesn’t get much easier than
this..

When your program runs, your user enters some data, and look what happens:

A “KeyError” exception has

been raised…but why?

414   Chapter 11
 dealing with complexity

If it’s not in the dictionary, it can’t be found.

The data in the row_data dictionary originally comes from the spreadsheet
and is read into your program from the CSV file.
If the data value entered into the recorded_time variable is in the dictionary,
things are going to be fine, because there’s a match. However, if the data entered
into the recorded_time variable doesn’t match anything in the dictionary,
you’ll get a KeyError.
But how is this “problem” handled during training?

If we have a match, great.

If not, we look for the
closest match and work
from there...

The entered time for a 20k run

(59:59) falls between these
values on the pace sheet. two

you are here 4   415

close enough

Search for the closest match

All you need to do is search the row of data for the closest match, right?
And guess what? The Head First Code Review Team think they have some
functions that might help here.
There’s nothing better
than sharing our code
with our fellow Python
programmers. Check out
our “find_it” module.

This code is in a file called

“find_it.py” and you can
download a copy from this
book’s support website.

The “find_closest”
function does a simple
linear search, returning
the value in “target_data”
Here’s an example of a nested that most closely matches
function, which is allowed the “look_for” argument.
in Python. Given two values,
this function returns the
difference between them.

This may not be the most

efficient search code ever
written, but it works.

416   Chapter 11
 dealing with complexity

Let’s test the find_it.py module to try and determine if it meets the requirements of your application. Load
the module into IDLE and then press F5 or choose Run Module from the Run menu:

>>> find_closest(3.3, [1.5, 2.5, 4.5, 5.2, 6])

2.5
>>> find_closest(3, [1, 5, 6])
1
Given a value to look
>>> find_closest(3, [1, 3, 4, 6]) for and some target
3 data, the “find_closest”
>>> find_closest(3.6, [1.5, 2.5, 4.5, 5.2, 6]) function seems to be
4.5 doing the trick.
>>> find_closest(3, [1, 4, 6])
4
>>> find_closest(2.6, [1.5, 2.5, 4.5, 5.2, 6])
2.5

Let’s try it with some of data that more closely resembles your CSV data:

>>> find_closest('59:59', ['56:29', '57:45', '59:03', '1:00:23', '1:01:45'])

Traceback (most recent call last):
File "<pyshell#23>", line 1, in <module>
find_closest('59:59', ['56:29', '57:45', '59:03', '1:00:23', '1:01:45'])
File "/Users/barryp/HeadFirstPython/chapter11/find_it.py", line 15, in find_closest
if diff == 0:
File "/Users/barryp/HeadFirstPython/chapter11/find_it.py", line 11, in whats_the_difference
TypeError: unsupported operand type(s) for -: 'str' and 'str'
Yikes! Something’s seriously
broken here.

What do you think has gone wrong here? Why

does the find_closest() function crash when
asked to work with data from your CSV file?

you are here 4   417

time trials

The trouble is with time

The data in your CSV file is a representation of timing values. Rather than
actual numbers, the values in the CSV are strings. This is great for you,
because you understand what the representation means. Python, on the other
hand, sees the data only as strings.
When you send your data to the find_closest() function, Python
attempts to treat your strings as numbers and chaos ensues. What might work
would be to convert the time-strings into numbers. But how?

When I have to work with

times, I always convert my Yeah, of course!
time strings to seconds Didn’t we write the
first... “tm2secs2tm” module to
handle this sort of thing?

418   Chapter 11
 dealing with complexity

The time-to-seconds-to-time module

The Head First Code Review Team’s generosity knows no bounds. Sure enough, their
rather strangely name tm2secs2tm.py module looks like it might help. Grab a copy of this code fro
this book’s support website. m
Here’s the guy’s
“tm2secs2tm.py” module.

This function ensures that all times are form atted in

le when doing
“HH:MM:SS” format. This helps keep things simp
conversions to seconds.

Given a “time string”, convert it to a value in seco

nds.

Convert a value in seconds to a “time string”.

Now that you have the tm2secs2tm.py and find_it.py modules, let’s create a function
that uses the facilities provided by these modules to solve your searching problem. Your new
function, called find_nearest_time(), takes two arguments: the time to look for and a
list of times to search. The function returns the closest time found as a string:
The code you from find_it import find_closest
need has been
started for
from tm2secs2tm import time2secs, secs2time

you.
def find_nearest_time(look_for, target_data):

Unlike in the previous

chapter, it is possible
to do what you need
to do here in only
four lines of code.

you are here 4   419

time to string

Now that you have the tm2secs2tm.py and find_it.py modules, you were to create a
function that uses the facilities provided by these modules to solve your searching problem. Your
new function, called find_nearest_time(), takes two arguments: the time to look for and
a list of times to search. The function returns the closest time found as a string:

Import the
from find_it import find_closest
The function takes two
team’s code. from tm2secs2tm import time2secs, secs2time
arguments, a time string
and a list of time strings.
def find_nearest_time(look_for, target_data):
Convert the time string
you are looking for into its what = time2secs(look_for)
equivalent value in seconds. Convert the lines of time
where = [time2secs(t) for t in target_data] strings into seconds.
Call “find_closest()”, res = find_closest(what, where) Return the closest match to the calling
supplying the converted data. code, after converting it back to a time
return(secs2time(res))
string.

Here’s some of your

pace data. Let’s
Let’s try out your code at the IDLE shell to see if your time “problems” have been resolved: work with data from
the “20k” row.

>>> find_nearest_time('59:59’, ['56:29', '57:45', '59:03', '1:00:23', '1:01:45'])

'01:00:23'
>>> find_nearest_time('1:01:01', ['56:29', '57:45', '59:03', '1:00:23', '1:01:45']) Great!
'01:00:23' This
>>> find_nearest_time('1:02:01', ['56:29', '57:45', '59:03', '1:00:23', '1:01:45']) appears
'01:01:45' to be
>>> find_nearest_time('57:06', ['56:29’, '57:45', '59:03', '1:00:23', '1:01:45'])
working
'00:56:29'
fine.

420   Chapter 11
 dealing with complexity

Test Drive
With all this code available to you, it’s an easy exercise to put it all together in your program and
produce a complete solution to the Marathon Club’s prediction problem. Let’s take it for a test run.

This code is used “as is”.

Find the nearest time Search for

within the data. a predicted
Extract the tim e at
column heading. the desired
distance and
display it on
screen.

same
Try out your program with the
data input from earlier.

Another “KeyError”…

After all that, you’re getting the same error as before. Bummer.

you are here 4   421

more time trials

The trouble is still with time…

Or, to be more precise, with how the tm2secs2tm.py module formats
time strings. Take another look at the results from the previous IDLE Session.
Do you notice anything strange about the results returned by the call to the
find_nearest_time() function?

>>> find_nearest_time('59:59’, ['56:29', '57:45', '59:03', '1:00:23', '1:01:45'])

'01:00:23'
All the >>> find_nearest_time('1:01:01', ['56:29', '57:45', '59:03', '1:00:23', '1:01:45'])
returned '01:00:23'
times
use the
>>> find_nearest_time('1:02:01', ['56:29', '57:45', '59:03', '1:00:23', '1:01:45'])

“HH:MM:SS” '01:01:45'

format. >>> find_nearest_time('57:06', ['56:29’, '57:45', '59:03', '1:00:23', '1:01:45'])

'00:56:29'

When your code takes one of these returned values and tries to index into
your dictionary, there’s no match found, because your dictionary’s keys do not
confirm to the HH:MM:SS format. The solution to this problem is to ensure
that every time you use a time-string in your code, make sure it’s in HH:MM:SS
format:

Import the “format_time()”

function from the
“tm2secs2tm.py” module.
Use the function to ensure e
used internally by your codethare times
formatted in “HH:MM:SS” forma
t.

422   Chapter 11
 dealing with complexity

Test Drive
Let’s try your code one more time. Hopefully, now that all of the time strings within the system
conform to HH:MM:SS format, your code will behave itself.

This is the previous test, which

crashed with a “KeyError”.

behaves
This time around, your program
itself and works fine.

Another test confirms that thing

working well. s are

And one final test makes sure.

This is working well. You’ve solved your application’s central problem: your
program reads in the spreadsheet data from the CSV file, turns it into a
dictionary of dictionaries, and lets you interact with your user to acquire
the recorded time at a particular distance before predicting a time for
another distance.
Not counting the code provided by the Head First Code Review Team,
you’ve written fewer than 40 lines of code to solve this problem. That’s
quite an achievement. All that’s left to do is to port your program to the club’s
Android’s phones.

And porting to Android won’t take too long, will it?

you are here 4   423

android interface

Port to Android
Your code is working great. Now it’s time to port your text-based Python
program to Android. Most of your code doesn’t need to change, only the
parts that interact with your user.
Obviously, you’ll want to make things as easy to use as possible for users of
your latest Android app, providing an interface not unlike this one.

2. Enter the
1. Start by recorded time…
picking a distance
run…

These are both

“dialogSetSingleChoiceItems”
dialog boxes.
This is a
“dialogGetInput”
dialog box.

3. Select a
distance to
predict to…
4. After the
lookup, display
the predicted
time.
This is a
“dialogSetItems”
dialog box.
424   Chapter 11
 dealing with complexity

Your Android app is a bunch of dialogs

Your Android app interacts with your users through a series of dialogs. Other
than the single dialog that requests data from your user, the other three share
certain similarities. You can take advantage of these shared features by
creating a utility function which abstracts the dialog creation details:
The dialog The text for
The list of creation the buttons,
The dialog’s values to display. method name. with defaults.
title string.

def do_dialog(title, data, func, ok='Select', notok='Quit'):

app.dialogCreateAlert(title)
func(data)
app.dialogSetPositiveButtonText(ok)
if notok:
app.dialogSetNegativeButtonText(notok)
app.dialogShow()
return(app.dialogGetResponse().result)

Display the dialog and then

return the selected item.

Assume the existence of a list called distances, which contains

the row distance labels (2mi, 5k, 5mi, and so on). In the space below,
provide the two calls to the do_dialog() function needed to create
the two dialogSetSingleChoiceItems shown on the left of
the previous page.

you are here 4   425

adding dialog

You were to assume the existence of a list called distances, which

contains the row distance labels. In the space below, you were to
provide the two calls to the do_dialog() function needed to create
the two dialogSetSingleChoiceItems.

do_dialog("Pick a distance", distances, Provide the list of

items to display.
Provide the
dialog title. app.dialogSetSingleChoiceItems)
do_dialog("Pick a distance to predict", distances, Provide the
Ditto: do it all type of dialog
again for the app.dialogSetSingleChoiceItems) to use.
other dialog. do_dialog('The predicited time running ' + predicted_distance +
' is: ', prediction, app.dialogSetItems, "OK", None)

Here’s This last one’s a little trickier,

because Use a different dialog Override the
another you have to build up the dialog tit creating method this default values for
example. from some variables (that you’ll le time. the dialog’s buttons.
have created first). need to

Get your Android app code ready

To use your dialog creating code, import the necessary libraries, define
some constants, create an Android object, and reuse some code from
earlier in this book:

import time Do your imports.

import android
...
distances = [ '2mi', '5k', '5mi', '10k', '15k', '10mi', '20k', Create a list of
'13.1mi', '25k', '30k', 'Marathon' ] row labels.
...
hello_msg = "Welcome to the Marathon Club's App"
quit_msg = "Quitting the Marathon Club's App." Define two
...
friendly messages.
Create an Android
app = android.Android()
app object.
def status_update(msg, how_long=2):

This function is taken “as-is”

app.makeToast(msg)
time.sleep(how_long) from earlier in this book.

426   Chapter 11
 dealing with complexity

Android Pool Puzzle

Your job is to take the code from the pool and place
it into the blank lines in your Android app code.
You can assume that the row_data dictionary
exists and has been populated. The variables
shown at the bottom of the last page have also
been created, and the status_update()
and do_dialog() functions are available
to you. Your goal is to arrange the code so that it
implements the UI interactions you need.

status_update(hello_msg)
resp = do_dialog("Pick a distance", distances, )

distance_run = The dialogGetInput() method

distance_run = distances[distance_run]
displays the input dialog box.
= app.dialogGetInput("Enter a " + distance_run + " time:",
"Use HH:MM:SS format:").result
closest_time = find_nearest_time(format_time( ), row_data[distance_run])
closest_column_heading = row_data[distance_run][closest_time]
resp = do_dialog("Pick a distance to predict", distances, )

predicted_distance =
predicted_distance = distances[predicted_distance]
prediction = [k for k in row_data[predicted_distance].keys()
if row_data[predicted_distance][k] == closest_column_heading]
do_dialog('The predicted time running ' + predicted_distance + ' is: ',
prediction, app.dialogSetItems, "OK", None)
status_update(quit_msg)

recorded_time

app.dialogGetSelec
tedItems().result[
SetSingleChoiceItems 0]
app.dialog te ms
eChoiceI
app.dialogSetSingl
if resp['which'] in ('positive'):
in ('positive'): if resp['which']
app.dialogGetSelec
recorded_time tedItems().result[
0]

you are here 4   427

 out of the pool

Android Pool Puzzle Solution

Your job was to take the code from the pool and
place it into the blank lines in your Android app
code.You were to assume that the row_data
dictionary exists and has been populated. The
variables you need also have been created, and
the status_update() and do_dialog()
functions were available to you. Your goal was Ask your user to pick a
to arrange the code so that it implements the UI distance from the list of
interactions you need.
labels.
status_update(hello_msg)
resp = do_dialog("Pick a distance", distances, app.dialogSetSingleChoiceItems)
if resp['which'] in ('positive'):
Assign the selected distance label
distance_run = app.dialogGetSelectedItems().result[0] to “distance_run”.
distance_run = distances[distance_run] Ask your user enter the
recorded_time = app.dialogGetInput("Enter a " + distance_run + " time:",
recorded time.
"Use HH:MM:SS format:").result
closest_time = find_nearest_time(format_time(recorded_time), row_data[distance_run])
closest_column_heading = row_data[distance_run][closest_time]
Work out resp = do_dialog("Pick a distance to predict", distances, app.dialogSetSingleChoiceItems)
what column
heading to
if resp['which'] in ('positive'):
Ask your user to pick a
use. predicted_distance = app.dialogGetSelectedItems().result[0] distance from the list of
predicted_distance = distances[predicted_distance] labels to predict to.
Look up the prediction = [k for k in row_data[predicted_distance].keys()
prediction. if row_data[predicted_distance][k] == closest_column_heading]
do_dialog('The predicted time running ' + predicted_distance + ' is: ',
e
prediction, app.dialogSetItems, "OK", None)
Display the predicted time atr. th
status_update(quit_msg) selected distance to your use

428   Chapter 11
 dealing with complexity

Put your app together…

You now have all the code you need to create your app:

Do your imports.

Include your “find_nearest()”

function.
Declare your constants.

NOTE: the location of the ta

SDCARD is specific to Andrda
oid.
file on the

Grab and preprocess your

CSV data.

Create your Android app object

include your helper functions. and

process
Display your UI to your user and
the resulting interaction.

you are here 4   429

test drive

Test Drive
It’s time to test your Android app on the Android Emulator before loading a working application
onto a “real” phone. Start your Android emulator and begin by transferring your code and the
files it needs onto the emulator’s SDCARD. Use the adb command in the tools folder to copy
marathonapp.py, find_it.py, tm2sec2tm.py and PaceData.csv to the emulator, and
then take your app for a spin.

File Edit Window Help CopyToEmulator

$ tools/adb push marathonapp.py /mnt/sdcard/sl4a/scripts
43 KB/s (2525 bytes in 0.056s)
$ tools/adb push find_it.py /mnt/sdcard/sl4a/scripts
7 KB/s (555 bytes in 0.069s)
$ tools/adb push tm2secs2tm.py /mnt/sdcard/sl4a/scripts
Copy your code 12 KB/s (628 bytes in 0.050s)
and its support $ tools/adb push PaceData.csv /mnt/sdcard/sl4a/scripts
files to the 59 KB/s (4250 bytes in 0.069s)

emulator with
these commands.

And there it is…

waiting for you
to test it.

Go on. You know you want to: tap that app!

430   Chapter 11
 dealing with complexity

Your app’s a wrap!

All that’s left to do is transfer your working Android app to the Marathon
Club’s phones…and that’s easy when you use AndFTP. When you show off
your latest work, the club’s members can’t believe their eyes.

This is fantastic! Now I can work with

my coach and the other club members
to hit my target times at my chosen
distances. There’s no stopping me now...

And there’s no stopping you!

You’ve put your Python skills and techniques to great use here.
Whether you’re building an app for the smallest handheld device or the
biggest web server, your Python skills help you get the job done.

Congratulations!

you are here 4   431

 python toolbox

Your Python Toolbox

You’ve got Chapter 11 under your
belt and you’ve demonstrated a
CHAPTER 11

mastery of your Python toolbox.

Congratulations and well done!

ƒƒ The input() BIF lets you prompt and

Python Lingo receive input from your users.

ƒƒ If you find yourself using Python 2 and in
• A “conditional” list
comprehension need of the input()function, use the
”
is one that includes a trailing “if l which raw_input() function instead.
statement, allowing you to controas the ƒƒ Build complex data structures by
items are added to the new list combining Python’s built-in lists, sets,
comprehension runs. and dictionaries.

• List comprehension
s can be rewritten ƒƒ The time module, which is part of
as an equivalent “for” loop. the standard library, has a number of
functions that make converting between
time formats possible.

432   Chapter 11
 dealing with complexity

It’s time to go…

It’s been a blast having

you with us here on Lake
Python. Come back soon
and often. We love it
when you drop by.

This is just the beginning

We’re sad to see you leave, but there’s nothing like taking what you’ve learned and putting it to use. You’re
just beginning your Python journey and you’re in the driver’s seat. We’re dying to hear how things go, so drop us a
line at the Head First Labs website, www.headfirstlabs.com, and let us know how Python is paying off for YOU!

you are here 4   433

 appendix: leftovers
The Top Ten Things
(we didn’t cover)
I don’t know about you, but
I think it could do with
more spam...

You’ve come a long way.

But learning about Python is an activity that never stops. The more Python you code,
the more you’ll need to learn new ways to do certain things. You’ll need to master new
tools and new techniques, too. There’s just not enough room in this book to show you
everything you might possibly need to know about Python. So, here’s our list of the top ten
things we didn’t cover that you might want to learn more about next.

this is an appendix   435

pro tools

#1: Using a “professional” IDE

Throughout this book, you’ve used Python’s IDLE, which is great to use
when first learning about Python and, although it’s a little quirky, can handle
most programming tasks. It even comes with a built-in debugger (check
out the Debug menu), which is surprisingly well equipped. Chances are,
however, sooner or later, you’ll probably need a more full-featured integrated
development environment.
One such tool worth looking into is the WingWare Python IDE. This
professional-level development tool is specifically geared toward the Python
programmer, is written by and maintained by Python programmers, and is
itself written in Python. WingWare Python IDE comes in various licencing
flavor: it’s free if you’re a student or working on an open source project, but
you’ll need to pay for it if you are working within a for-profit development
environment.

Written in Python by Python

programmers for other
Python programmers...what
else could you ask for?

The WingWare
Python IDE

More general tools also exist. If you are running Linux, the KDevelop IDE
integrates well with Python.
And, of course,there are all those programmer editors which are often all
you’ll ever need. Many Mac OS X programmers swear by the TextMate
programmer’s editor. There’s more than a few Python programmers using
emacs and vi (or its more common variant, vim). Your author is a huge fan
of vim, but also spends large portions of his day using IDLE and the Python
shell.

436   appendix
 leftovers

#2: Coping with scoping

Consider the following short program:

A global variable called “name”.

A function which
attempts to read
from and write to
the global variable
called “name”.
Call the function.

See what “name” is set to

after the function runs.

If you try to run this program, Python complains with this error message:
UnboundLocalError: local variable ‘name’ referenced before assignment…whatever that
means!
When it comes to scope, Python is quite happy to let you access and read
the value of a global variable within a function, but you cannot change
it. When Python sees the assignment, it looks for a local variable called
name, doesn’t find it, and throws a hissy fit and an UnboundLocalError
exception. To access and change a global variable, you must explicitly declare
that’s your intention, as follows:

Some programmers find this

quite ugly. Others think it’s
what comes to pass when you
watch Monty Python reruns
while designing your programm
language. No matter what ing
everyone thinks: this is what
we’re stuck with! §

you are here 4   437

testing, testing

#3: Testing
Writing code is one thing, but testing it is quite another. The combination of
the Python shell and IDLE is great for testing and experimenting with small
snippets of code, but for anything substantial, a testing framework is a must.
Python comes with two testing frameworks out of the box.
The first is familiar to programmers coming from another modern language,
because it’s based on the popular xUnit testing framework. Python’s
unittest module (which is part of the standard library) lets you create test
code, test data, and a test suite for your modules. These exist in separate files
from you code and allow you to exercise your code in various ways. If you
already use a similar framework with your current language, rest assured that
Python’s implementation is essentially the same.
The other testing framework, called doctest, is also part of the standard
library. This framework allows you to take the output from a Python shell or
IDLE session and use it as a test. All you need to do is copy the content from
the shell and add it to your modules documentation strings. If you add code like
this to the end of your modules, they’ll be ready for “doctesting”:

If your code is imported What do you mean:

if __name__ == "__main__": as a module, this code you can’t hear me...I
does NOT run. If you run guess I should’ve
import doctest your module from the tested this first, eh?
doctest.testmod() command line, your tests
run.

If you then run your module at your operating systems comand line, your
tests run. If all you want to do is import your module’s code and not run your
tests, the previous if statement supports doing just that.
For more on unittest and doctest, search the online Python
documentation on the Web or via IDLE’s Help menu.

438   appendix
 leftovers

#4: Advanced language features

With a book like this, we knew we’d never get to cover the entire Python
language unless we tripled the page count.
And let’s face it, no one would thank us for that!
There’s a lot more to Python, and as your confidence grows, you can take the
time to check out these advanced language features:
Anonymous functions: the lambda expression lets you create small, one-
line, non-named functions that can be incredibly useful once you understand
what’s going on.
Generators: like iterators, generators let you process sequences of data.
Unlike iterators, generators, through the use of the yield expression, let
you minimize the amount of RAM your program consumes while providing
iterator-like functionality on large datasets.
Custom exceptions: create your own exception object based on those
provided as standard by Python.
Function decorators: adjust the behavior of a preexisting function by
hooking into its start-up and teardown mechanisms.
Metaclasses: custom classes that themselves can create custom classes.
These are really only for the truely brave, although you did use a metaclass
when you created your Sightings form using the Django form validation I know I look
framework in Chapter 10. complex, but I really
am quite useful.
Most (but not all) of these language features are primarily of interest to the
Python programmer building tools or language extensions for use by other
Python programmers.
You might never need to use some of these language features in your code,
but they are all worth knowing about. Take the time to understand when and
where to use them.
See #10 of this appendix for a list of my favorite Python books (other than
this one), which are all great starting points for learning more about these
language features.

you are here 4   439

regexes

#5: Regular expressions

When it comes to working with textual data, Python is a bit of a natural.
The built-in string type comes with so many methods that most of the
standard string operations such as finding and splitting are covered. However,
what if you need to extract a specific part of a string or what if you need
to search and replace within a string based on a specific specification? It is
possible to use the built-in string methods to implement solutions to these
types of problems, but—more times than most people would probably like to
admit to—using a regular expression works better.
Consider this example, which requires you to extract the area code from the
phone_number string and which uses the built-in string methods:

Find the opening “(“. Jeff Friedl’s regular

Calculate where the area code is expression “bible”,
in the string. which is well worth a
look if you want to
Extract the area code. learn more. Look up
the “re” module in
Python’s docs, too.

This code works fine, but it breaks when presented with the following value for
phone_number:
Why does this phone number
phone_number = "Cell (mobile): (555)-918-8271" cause the program to fail? Try
it and see what happens…
When you use a regular expression, you can specify exactly what it is you
are looking for and improve the robustness of your code:
This looks a little strange,
but this regular expression
is looking for an opening “(“
followed by three digits
and then a closing “)”.
This specification is much
more likely to find the
area code and won’t break
as quickly as the other
version of this program.

440   appendix
 leftovers

#6: More on web frameworks

When it comes to building web applications, CGI works, but it’s a little
old-fashioned. As you saw in Chapter 10, Google’s App Engine technology
supports CGI, but also WSGI and a number of web framework technologies.
If you aren’t deploying to the cloud and prefer to roll your own, you have
plenty of choices. What follows is a representative sample. My advice: try a
few on for size and see which one works best for you.
Search for the following terms in your favorite search engine: Django, Zope,
TurboGears, Web2py, and Pylons.

The “old timers”…but

don’t let maturity fool
you: these are cracking
web frameworks.

The “new kids on the

block”: leaner, meaner
and stuffed full of
features.

you are here 4   441

data mappers and nosql

#7: Object relational mappers and NoSQL

Working with SQL-based databases in Python is well supported, with the
inclusion of SQLite in the standard library a huge boon. Of course, the
assumption is you are familiar with SQL and happy to use SQL to work with
your data.
But what if you aren’t? What if you detest SQL?
An object relational mapper (ORM) is a software technology that lets
you use an underlying SQL-based database without having to know anything
about SQL. Rather than the procedural interface based on the Python
database API, ORMs provide an object-oriented interface to your data,
exposing it via method calls and attribute lookups as opposed to columns and
rows.
Many programmers find ORMs a much more natural mechanism for
working with stored datasets and the Python community creates and supports
a number of them.
One of the most interesting is SQL Alchemy, which is popular and included
in a number of the web framework technologies discussed in #6. Despite
being hugely popular anyway, SQL Alchemy is also interesting because it
supports both Python 2 and Python 3, which makes it a standout technology
(for now).
If you find yourself becoming increasingly frustrated by SQL, check out an
ORM. Of course, you have already experienced a similar technology: Google
App Engine’s datastore API is very similar in style to those APIs provided by
the major Python ORMs.

There’s NoSQL, too.

In addition to database technologies that let you avoid working with the
underlying SQL-based database, a new breed of technologies have emerged
that let you drop your SQL database in its entirety. Known collectively as
NoSQL, these data tools provide an alternative non-SQL API to your
data and do not use an SQL-based database management system at all. As
these technologies are relatively new, there’s been more activity around
Python 2 than Python 3, but they are still worth checking out. CouchDB
and MongoDB are the two most closely associated with robust Python
implementations. If you like working with your data in a Python dictionary
and wished your database technology let you store your data in much the
same way, then you need to take a look at NoSQL: it’s a perfect fit.

442   appendix
 leftovers

#8: Programming GUIs

In this book, you’ve created text-based interfaces, web-based interfaces and
interfaces that ran on Android devices. But what if you want to create a
desktop application that runs on your or your user’s desktop computer? Are
you out of luck, or can Python help here, too?
Well…as luck would have it, Python comes preinstalled with a GUI-building
toolkit called tkinter (shorthand for Tk Interface). It’s possible to create a usable
and useful graphical user interface (GUI) with tkinter and deploy it on Mac
OS X, Windows, and Linux. With the latest version of Tk, your developed app
takes on the characteristics of the underlying operating system, so when you
run on Windows, your app looks like a Windows desktop app, when it run on
Linux, it looks like a Linux desktop app, and so on.
You write your Python and tkinter code once, then run it anywhere and it just
works. There are lots of great resources for learning to program with tkinter,
with one of the best being the last few chapters of Head First Programming, but
since plugging that book would be totally shameless, I won’t mention it again.
Other GUI-building technologies do exist, with the PyGTK, PyKDE,
wxPython, and PyQT toolkits coming up in conversation more than most. Be
warned, however, that most of these toolkits target Python 2, although support
for Python 3 is on its way. Search the Web for any of the project names to learn
more.

Oh, look: it’s one of the

GUIs created in “Head Firstd
Programming”…and yes, I sai
I wouldn’t mention THAT I
book again, but isn’t this GU
beautiful? §

you are here 4   443

your bugs, my bugs, and threads

#9: Stuff to avoid

When it comes to stuff to avoid when using Python, there’s a very short list. A
recent tweet on Twitter went something like this:

“There are three types of

bugs: your bugs, my bugs...
and threads.”

Threads do indeed exist in Python but should be avoided where possible. Don’t you like my
This has nothing to do with the quality of Python’s threading library and threads...?
everything to do with Python’s implementation, especially the implementation
known as CPython (which is more than likely the one you’re running
now). Python is implemented using a technology known as the Global
Interpreter Lock (GIL), which enforces a restriction that Python can only
ever run on a single interpreter process, even in the presence of multiple
processors.
What all this means to you is that your beautifully designed and implemented
program that uses threads will never run faster on multiple processors even
if they exist, because it can’t use them. Your threaded application will run
serially and, in many cases, run considerably slower than if you had developed
the same functionality without resorting to threads.
Main message: don’t use threads with Python until the GIL restriction is
removed…if it ever is.

444   appendix
 leftovers

#10: Other books

There are lots of great books that cover Python in general, as well as
If you are a
specifically within a particular problem domain. Here is a collection of my
sysadmin, then
favorite Python books, which we have no hestitation in recommending to you.
this is the Python
book for you.

Includes a great case study

involving the porting of a comple
Python 2 module to Python 3. x

The best desktop

reference on the
market

At 1,200 pageua s, this is the for

ge reference b ig examples with
definitive lang t everything in it! In cl ud es so m e
ML, parsing
Python: it’s go big technology:laXnguage features.
and advanced

you are here 4   445

 Index
Symbols and Numbers
addition operator (+) 138
Alt-N keystroke, IDLE 5
404 Error, from web server 242 Alt-P keystroke, IDLE 5
405 Error, from web server 378 AndFTP app 288–289
>>> (chevron, triple) IDLE prompt 4 Android apps
: (colon) accepting input from 278–282, 295, 304–307
converting from Python code 424–430
in for loop 16
creating 274–277, 281–282
in function definition 29
data for. See JSON data interchange format
in if statement 20
integrating with SQLite 342–348
, (comma) separating list items 7 running on phone 288–289
{} (curly braces) scripting layer for. See SL4A
creating dictionaries 180 troubleshooting 277
creating sets 166 Android emulator
= (equal sign) installing and configuring 260–262
assignment operator 7 running scripts on 264–265, 272–273, 283
in function argument definition 63 Android Market 288
(...) (parentheses) Android Virtual Device. See AVD
enclosing function arguments 29 anonymous functions 439
enclosing immutable sets 91
append() method, lists 10, 14
+ (plus sign) addition or concatenation operator 138
apps. See Android apps; webapps
# (pound sign) preceding one-line comments 38
app.yaml file 356, 395
@property decorator 250, 253, 285
arguments for functions
? (question mark) parameter placeholder 321, 350
adding 52, 66–68
“...” or ‘...’ (quotes) enclosing each list item 7 optional 63–64, 134
“””...””” or ‘’’...’’’ (quotes, triple) enclosing comments 37 arrays
; (semicolon) separating statements on one line 38 associative. See dictionaries
[...] (square brackets) similarity to lists 9–10, 17
accessing dictionary items 180, 212 as keyword 119, 138
accessing specific list items 9, 18 assignment operator (=) 7
enclosing all list items 7, 18 associative arrays. See dictionaries

A attributes, class 190, 194, 212

authorization, user 389–393
“a” access mode 110 AVD (Android Virtual Device) 261, 291
access modes for files 110, 133

this is the index   447

the index

B
sending data to 300–303
tracking module for 248–249
troubleshooting 242, 247–250
“batteries included” 32 writing 236–238, 244–246
BIFs. See built-in functions writing for Android. See SL4A
BigTable technology 354, 359 cgi library 300
blue text in IDLE 4 cgitb module 248–249, 253
books chaining
Dive Into Python 3 (CreateSpace) 445 functions 146, 172
Head First HTML with CSS & XHTML (O’Reilly) methods 142, 172
374 chevron, triple (>>>) IDLE prompt 4
Head First Programming (O’Reilly) 443 chmod command 239, 253
Head First SQL (O’Reilly) 313
Learning Python (O’Reilly) 445 classes 189–191
Mastering Regular Expressions (O’Reilly) 440 attributes of 190, 194, 212
Programming in Python 3 (Addison-Wesley Profes- benefits of 189
sional) 445 converting data to dictionary 285–286
Python Essential Reference (Addison-Wesley Profes- defining 190–193, 194, 195–196
sional) 445 inherited 204–209, 212
Python for Unix and Linux System Administration instances of 190, 191, 194, 195–196
(O’Reilly) 445 metaclasses 439
braces. See curly braces methods of 190, 195–196, 198–200
brackets, regular. See parentheses in modules 209, 212
class keyword 191, 212
brackets, square. See square brackets
close() method, database connection 315, 350
BSD, running CGI scripts on 239
close() method, files 75, 103
build folder 42
code editors 35, 436. See also IDLE
built-in functions (BIFs). See also specific functions
displayed as purple text in IDLE 4 colon (:)
help on 21 in for loop 16
importing of, not needed 55 in function definition 29
namespace for 55 in if statement 20
number of, in Python 21 comma (,) separating list items 7
__builtins__ namespace 55, 71 comments 37–38

C
commit() method, database connection 315, 350
Common Gateway Interface scripts. See CGI scripts
cascading style sheet (CSS) 374–375 comprehension, list 154–159, 172, 409–411, 432
case sensitivity of identifiers 17 concatenation operator (+) 138
cgi-bin folder 234, 235 conditional list comprehension 409–411, 432
CGI (Common Gateway Interface) scripts 217, 235, 243, conditions. See if/else statement
253. See also WSGI connection, database
location of 234, 235 closing 314, 315
running 239 creating 314, 315
running from Android 264–265, 272–273, 283

448   Index
 the index

connect() method, sqlite3 315, 350 rollback changes to data 314, 315, 350
context management protocol 120 schema for 317
SQLite for. See SQLite
Controller, in MVC pattern 221 tables in 317, 319–320, 350
for GAE webapps 359, 370–373
data folder 234
for webapps 234–238, 244–246
“copied” sorting 144, 145–146, 172 data interchange format. See JSON data interchange
format
CREATE TABLE statement, SQL 317, 319–320
data objects. See also specific data objects
CSS (cascading style sheet) 374–375 getting next item from 54
CSV format, converting to Python data types 401–405 ID for 54
curly braces ({}) length of, determining 32
creating dictionaries 180 names of. See identifiers
creating sets 166 datastore, for GAE 359–360, 380–383, 384–387, 395
cursor() method, database connection 315, 350 data types
custom code 131 converting CSV data into 401–405
converting strings to integers 54
custom exceptions 439
in datastore 360

D
immutable 91, 103, 116, 138
for JSON 285
for list items 8, 12
data
date and time data
for Android apps. See JSON data interchange format
format compatibility issues 418–423
bundling with code. See classes
property type for 362, 384–385
duplicates in, removing 161–163, 166–167
external. See database management system; files db.Blob() type 360
for GAE webapps. See datastore, for GAE db.DateProperty() type 360
nonuniform, cleaning 148–153 db.IntegerProperty() type 360
race conditions with 309–310
db.StringProperty() type 360, 385
Robustness Principle for 384–387
searching for closest match 416–417 db.TimeProperty() type 360
sending to web server 275, 291 db.UserProperty() type 360, 390
sorting 144–147, 172 decision statement. See if/else statement
storing. See persistence
transforming, list comprehensions for 154–159 decorators, function 439
database API 314–315, 350 def keyword 29, 191, 212
database management system 312 dialogCreateAlert() method, Android API 274, 276, 280
closing connection to 314, 315 dialogGetInput() method, Android API 295, 304–306
commit changes to data 314, 315, 350 dialogGetResponse() method, Android API 274, 276,
connecting to 314, 315 278, 280
cursor for, manipulating data with 314, 315
dialogGetSelectedItems() method, Android API 278, 280
designing 316–318
inserting data into 321, 324, 348 dialogSetItems() method, Android API 279, 280
integrating with Android apps 342–348 dialogSetNegativeButtonText() method, Android API
integrating with webapps 327–341 274, 276
managing and viewing data in 326 dialogSetPositiveButtonText() method, Android API 274,
process for interacting with 314–315 276, 280
querying 322, 332–333

you are here 4   449

the index

dialogSetSingleChoiceItems() method, Android API 274, enable() function, cgitb 248, 253
276 end_form() function, yate 231, 233
dialogShow() method, Android API 276, 280 entities, in datastore 360, 395
dict() factory function 180, 212 enumerate() built-in function 54
dictionaries 178–182, 212 environ dictionary 300, 350
accessing items in 180, 212
equal sign (=)
compared to lists 179
converting class data to 285–286 assignment operator 7
creating 180, 182, 186 in function argument definition 63
dictionaries within 407–409 errors. See exception handling; troubleshooting
keys for 178, 180, 212 exception handling 88–95, 103. See also troubleshooting
populating 180, 212 benefits of 95, 100
reading CSV data into 403–404 closing files after 114–115, 120–123
values of 178, 180, 212 custom exceptions 439
dir() command 225 defining with try/except statement 89, 91–94
directory structure. See folder structure ignoring found errors 93–94
IndexError exception 17
dist folder 42
IOError exception 103, 112–114, 117–119
distribution for missing files 96–98
creating 40–42 NameError exception 44, 118
updating 60–61, 65 PickleError exception 133–134
uploading to PyPI 48 specific errors, checking for 101–102
Dive Into Python 3 (CreateSpace) 445 specific errors, details about 117–119
djangoforms.ModelForm class 368 TypeError exception 56–57, 116, 247–249, 283–285
ValueError exception 78–79, 81–82, 103
Django Project
exception objects 119, 138
Form Validation Framework 368–369, 395
templates 363–366, 395 except keyword. See try/except statement
doctest framework 438 execute() method, cursor 315, 322, 324, 350
documentation for Python 3 3, 80, 103 extend() method, lists 10
dot notation 10, 194, 196
double quotes. See quotes F
dump() function, pickle 133–134, 138 F5 key, IDLE 39, 44, 49, 71
dumps() function, json 269, 272, 281, 291 factory functions 166
dynamic content 216, 217 favicon.ico file, for webapp 234

E
fetchall() method, cursor 322
fetchmany() method, cursor 322
Eclipse editor 35 fetchone() method, cursor 322
editors 35, 436. See also IDLE FieldStorage() method, cgi 244, 253, 296, 300, 350
elif keyword 108. See also if/else statement files. See also persistence
else keyword. See if/else statement access modes for 110, 133
appending data to 110
emacs editor 35, 436 checking for existence of 118

450   Index
 the index

closing 75, 110 chaining 146, 172

closing after exception 114–115, 120–123 creating 28–30, 170–171
CSV format, converting to Python data types 401– optional arguments for 63–64, 134
405 recursive 31
exceptions involving, determining type of 117–119 sharing. See modules
flushing 110
missing, exception handling for 96–98
opening 75, 109–110
G
opening in binary access mode 133 GAE (Google App Engine) 354
reading data from 75–78, 142–143 configuration and setup for 356–357
rewinding 76 controller code for 370–373
splitting lines in 77–78 data modeling with 360–362
writing 110–113 datastore for 359, 380–383, 384–387, 395
writing, custom formats for 126–130 deploying webapps to Google cloud 391
writing, default format for 124–125 folder structure for 356, 370
writing, pickle library for. See pickle library form generation for 368–369
finally keyword 115, 138 form input restrictions for 376–377, 384–387
find() method, strings 84–86, 103 form stylesheets for 374–375
MVC pattern used by 359
Firefox, SQLite Manager for 326
SDK for, downloading 355
folder structure troubleshooting 378
for distribution 42 user authorization for 389–393
for GAE 356, 370 view for, desigining 363–369
for webapps 234 GAE Launcher 357, 391, 395
for loop 15–17, 32 garbage collection 116
compared to list comprehension 432
nesting 19–22 generators 439
forms, HTML 295 get() method, GAE 370, 379, 395
creating from template 296–299 GET web request 370
Form Validation Framework for 368–369 GIL (Global Interpreter Lock) 444
input restrictions for 376–377, 384–387
glob module 237, 253
sending data to CGI scripts 300–303
stylesheets for 374–375 Google App Engine. See GAE
Form Validation Framework 368–369, 395 Google BigTable technology 354, 359
405 Error, from web server 378 GQL (Google Query Language) API 359
404 Error, from web server 242 green text in IDLE 4
Friedl, Jeff (author, Mastering Regular Expressions) 440 GUI (graphical user interface), building 443

H
from statement 46, 49
functional programming concepts 157
function decorators 439 hashes. See dictionaries
functions header() function, yate 231, 233
adding arguments to 52, 66–68 Head First HTML with CSS & XHTML (O’Reilly) 374
anonymous 439
built-in. See built-in functions (BIFs) Head First Programming (O’Reilly) 443

you are here 4   451

the index

Head First SQL (O’Reilly) 313 IndexError exception 17

help() built-in function 80, 103 index.html file, for webapp 234
HTML inherited classes 204–209, 212
generating for webapp interface 230–231 __init__() method 191, 212
learning 226
in operator 16, 118, 138
templates for, with Django 363–366
HTML forms. See forms, HTML “in-place” sorting 144, 145, 172
HTTP server 235 input
from Android apps 278–282, 295, 304–307
http.server module 235, 253 HTML forms for. See forms, HTML

I
from keyboard after screen prompt 413–414, 432
input() built-in function 413–414, 432
id() built-in function 54 insert() method, lists 10, 14
IDE 436. See also IDLE INSERT statement, SQL 321, 324, 348
identifiers 7, 17, 32 instances of classes 190, 191, 194, 195–196
IDLE 3–5, 32 int() built-in function 54
colored syntax used in 4 integers, converting strings to 54
indenting enforced in 4 interface. See View, in MVC pattern
preferences, setting 5
IOError exception 103, 112–114, 117–119
prompt in (>>>) 4
recalling and editing code statements 5 I/O (input/output), handling. See files
running or loading code in 39, 44, 49, 71 isinstance() built-in function 20–22, 32
TAB completion 5 iterations
if/else statement 20, 32 for loop 15–17, 19–22, 32
elif keyword 108 generating with range() function 54–56
in list comprehension 432 while loop 16–17
negating conditions in 86, 103
images folder 234
immutable data types 138
J
lists 91, 103, 116 JSON data interchange format 266–267, 291
numbers 116 API for, using 269–272
strings 116 browser differences with 272
data types supported by 285
import statement 43, 46, 49, 71
incompatibility with pickle data objects 284–285
include_footer() function, yate 230, 232, 233
include_header() function, yate 230, 232
indentation rules
K
enforced in IDLE 4 KDevelop IDE 436
for for loops 16 keys, in dictionary 178, 180, 212
for function definitions 29 keywords, displayed as orange text in IDLE 4
for if statement 20

452   Index
 the index

L
locals() built-in function 118, 138
loops. See iterations

M
lambda expression 439
Learning Python (O’Reilly) 445
len() built-in function 10, 32 Mac OS X
lib folder 42 code editors for 35
Linux GAE log messages on 378
IDEs for 436
code editors for 35
installing Python 3 on 3
GAE log messages on 378
running CGI scripts on 239, 272
IDEs for 436
running GAE Launcher on 357
installing Python 3 on 3
transferring files to Android device 288
running CGI scripts on 239, 272
running GAE Launcher on 357 __main__ namespace 45
transferring files to Android device 288 MANIFEST file 42
list() built-in function 54 mappings. See dictionaries
list comprehension 154–159, 172, 409–411, 432 Mastering Regular Expressions (O’Reilly) 440
lists 32. See also data objects metaclasses 439
adding items to 10–14 methods 190. See also specific methods
bounds checking for 17 chaining 142, 172
classes inherited from 204–208 for classes 195–196, 198–200
compared to dictionaries 179 creating 212
compared to sets 167 results of, as attributes 250, 253
creating 6–7, 54 self argument of 212
data types in 8, 12
duplicates in, removing 161–163 ModelForm class, djangoforms 368
extracting specific item from 175–176 Model, in MVC pattern 221
getting next item from 54 for GAE webapps 359, 360–362
identifiers for 7 for webapps 222–225
immutable 91, 103, 116 Model-View-Controller pattern. See MVC pattern
iterating 15–17, 157
modules 34–36, 71
length of, determining 10, 32
adding functionality to 50–52
methods for 10
classes in 209, 212
nested, checking for 20–22
creating 35
nested, creating 18–19
distribution for, creating 40–42
nested, handling 23–25, 28–31
distribution for, updating 60–61, 65
numbered, creating 54
distribution for, uploading to PyPI 48
reading CSV data into 403–404
importing 43–44, 46
removing items from 10
loading in IDLE 39, 49, 71
similarity to arrays 9–10, 17
locations for 38, 49
slice of 160, 172
namespaces for 45–46, 71
load() function, pickle 133, 138 in Python Standard Library 36
loads() function, json 269, 276, 280, 291 third-party 36

you are here 4   453

the index

Monty Python 17 pickle library 132–137, 138

multiple inheritance 209 data modeling using 222–224
incompatibility with JSON data types 284–285
MVC (Model-View-Controller) pattern 221, 232, 253,
359 transferring data to a database 321–325
Controller 234–238, 244–246, 370–373 plus sign (+) addition or concatenation operator 138
Model 222–225, 360–362 pop() method, lists 10, 175–176
View 226–233, 363–369 post() method, GAE 379–383, 395

N POST web request 379

pound sign (#) preceding one-line comments 38
NameError exception 44, 118 print() built-in function 10, 32, 124–125
names. See identifiers disabling automatic new-line for 56, 71
namespaces 45–46, 71 displaying TAB character with 56
writing to a file 110, 128, 138
next() built-in function 54
Programming in Python 3 (Addison-Wesley Professional)
NoSQL 359, 442 445
NotePad editor 35 properties, in datastore 360, 395
not in operator 161–162 @property decorator 250, 253, 285
not keyword 86, 103 purple text in IDLE 4
numbered lists 54 put() method, GAE 395

O
.pyc file extension 42, 49
.py file extension 35
object relational mapper. See ORM (object relational PyPI (Python Package Index) 36
mapper) registering on website 47
objects. See data objects uploading distributions to 48
uploading modules to 209
open() built-in function 75, 103, 109–110
Python 2
orange text in IDLE 4 compared to Python 3 17
ORM (object relational mapper) 442 raw_input() built-in function 432
os module 76, 300, 350 running on Android smartphones 258–259, 291
using with Google App Engine 355
P Python 3
compared to Python 2 17
para() function, yate 231, 233 documentation for 3, 80, 103
parentheses ((...)) editors for 35, 436
enclosing function arguments 29 installing 3
enclosing immutable lists 91 interpreter for. See IDLE
learning 445
pass statement 93, 103
python3 command
persistence 105 building a distribution 41
pickle library for 132–137 checking for Python version 3
reading data from files 222–224 installing a distribution 41
writing data to files 110–113, 222–224 uploading a new distribution 68
PickleError exception 133–134

454   Index
 the index

Python Essential Reference (Addison-Wesley Professional) sdist command 41

445 seek() method, files 76, 103
Python for Unix and Linux System Administration SELECT/OPTION, HTML tag 376
(O’Reilly) 445
SELECT statement, SQL 322, 332–333
Python, Monty 17
self argument 192–193, 212
Python Package Index. See PyPI
self.request object 379, 395
Python Standard Library 36
self.response object 372, 379, 395

Q semicolon (;) separating statements on one line 38

set() built-in function 166, 172
querying a database 322, 332–333
sets 166, 167, 172
question mark (?) parameter placeholder 321, 350
setup() built-in function 40
quotes (“...” or ‘...’) enclosing each list item 7
setup.py file 40, 71
quotes, triple (“””...””” or ‘’’...’’’) enclosing comments 37
single quotes. See quotes

R SL4A (Scripting Layer for Android) 258, 291

adding Python to 263
“r” access mode 110 Android apps, creating 274–277
automatic rotation mode, setting 264
race conditions 309–310 documentation for 274
radio_button() function, yate 231, 233 installing 262
range() built-in function 54–56, 71 Python versions supported 258–259
raw_input() built-in function 432 slice of a list 160, 172
readline() method, files 76, 103, 142 smartphones, apps on. See Android apps
recursion 31 sorted() built-in function 144–147, 153, 158, 172
regular brackets. See parentheses sort() method, lists 144–145, 153, 172
regular expressions 440 split() method, strings 77–78, 80–81, 103, 142
re module 440 SQL Alchemy 442
remove() method, lists 10 SQLite 313, 350
closing connection to 314, 315
render() function, template 364, 366
committing data to 314, 315
Robustness Principle 384–387 connecting to 314, 315
rollback() method, database connection 315, 350 cursor for, manipulating data with 314, 315
runtime errors 88. See also exception handling; trouble- designing database 316–318
shooting inserting data into 321, 324, 348
integrating with Android apps 342–348

S integrating with webapps 327–341

managing data in 326
schema, database 317 process for interacting with 314–315
querying 322, 332–333
scoping of variables 437 rollback changes to data 314
Scripting Layer for Android. See SL4A schema for database 317
scripts. See CGI scripts; SL4A tables in, creating 319–320

you are here 4   455

the index

T
sqlite3 command 326
sqlite3 library 313, 315, 350
SQLite Manager, for Firefox 326 TAB character, printing 56
SQL (Structured Query Language) 313, 350. See TAB completion, IDLE 5
also NoSQL; SQLite; ORM
tables, database 317, 319–320, 350
square brackets ([...])
target identifiers, from split() method 77, 91
accessing dictionary items 180, 212
accessing specific list items 9, 18 .tar.gz file extension 42
enclosing all list items 7, 18 Template class 230, 253
standard error (sys.stderr) 248, 291 template module 364
standard input (sys.stdin) 291 templates folder 234, 370
Standard Library, Python 36 templates for GAE 363–366, 395
standard output (sys.stdout) 126–128, 291 testing code 438
start_form() function, yate 231, 233 TextMate editor 35, 436
start_response() function, yate 230, 232 third-party modules 36
static content 216, 217 threads 444
static folder 370 time data
str() built-in function 119, 138 format compatibility issues 418–423
strings property type for 362, 384–385
concatenating 138 time module 419, 432
converting other objects to 119 Tk Interface (tkinter) 443
converting to integers 54 traceback 88, 103. See also exception handling; trouble-
displayed as green text in IDLE 4 shooting
finding substrings in 84–86
tracing code 58–59
immutable 116
sorting 148 triple chevron (>>>) IDLE prompt 4
splitting 77–78, 80–81 triple quotes (“””...””” or ‘’’...’’’) enclosing comments 37
substitution templates for 230, 253 troubleshooting. See also exception handling
strip() method, strings 108, 138, 142 404 Error, from web server 242
Structured Query Language. See SQL 405 Error, from web server 378
stylesheets for HTML forms 374–375 Android apps 277
GAE webapps 378
suite 16, 29, 32
testing code 438
sys module 291 tracing code 58–59
sys.stdout file 126–128, 138 try/except statement 89, 93–94, 101–102, 103
finally keyword for 115, 138
with statement and 120–123
tuples 91, 103, 116
TypeError exception 56–57, 116, 247–249, 283–285

456   Index
 the index

U
webapps 215–217, 253
controlling code for 221, 234–238, 244–246
data modeling for 221, 222–225
u_list() function, yate 231, 233 designing with MVC 221
unittest module 438 design requirements for 218–220
urlencode() function, urllib 291 directory structure for 234
Google App Engine for. See GAE
urllib2 module 291
input data, sending to CGI scripts 300–303
urllib module 291 input forms for. See forms, HTML
urlopen() function, urllib2 291 SQLite used with 327–341
user authorization 389–393 view for 221, 226–233
Web-based applications. See webapps
user input. See forms, HTML; input
web frameworks 441. See also CGI; WSGI
UserProperty() type, db 390
web request 216, 253, 395

V web response 216–217, 253, 395

web server 216–217, 235
ValueError exception 78–79, 81–82, 103
Web Server Gateway Interface (WSGI) 356, 370. See
values, part of dictionary 178, 180, 212 also CGI scripts
variables, scope of 437 while loop 16–17, 55
vi editor 35, 436 WingIDE editor 35
View, in MVC pattern 221 WingWare Python IDE 436
for GAE webapps 359, 363–369 with statement 120–123, 138
for webapps 226–233
WSGI (Web Server Gateway Interface) 356, 370. See
vim editor 436 also CGI scripts

W Y
“w” access mode 110 yate (Yet Another Template Engine) library 226–233
“w+” access mode 110 yield expression 439
“wb” access mode 133

you are here 4   457