Python 3

Metaprogramming
David Beazley
@dabeaz
http://www.dabeaz.com

Presented at PyCon'2013, Santa Clara, CA

March 14, 2013

Copyright (C) 2013, http://www.dabeaz.com 1

Requirements

Python 3.3 or more recent

Don't even attempt on any earlier version
Support files:
http://www.dabeaz.com/py3meta

Copyright (C) 2013, http://www.dabeaz.com 2

 Welcome!

An advanced tutorial on two topics

Python 3
Metaprogramming
Honestly, can you have too much of either?
No!

Copyright (C) 2013, http://www.dabeaz.com 3

Metaprogramming
In a nutshell: code that manipulates code
Common examples:
Decorators
Metaclasses
Descriptors
Essentially, it's doing things with code
Copyright (C) 2013, http://www.dabeaz.com 4
 Why Would You Care?

Extensively used in frameworks and libraries

Better understanding of how Python works
It's fun
It solves a practical problem

Copyright (C) 2013, http://www.dabeaz.com 5

DRY
Copyright (C) 2013, http://www.dabeaz.com 6
 DRY Don't Repeat Yourself

Copyright (C) 2013, http://www.dabeaz.com 7

DRY Don't Repeat Yourself

Don't Repeat Yourself

Copyright (C) 2013, http://www.dabeaz.com 8

 Don't Repeat Yourself

Highly repetitive code sucks

Tedious to write
Hard to read
Difficult to modify

Copyright (C) 2013, http://www.dabeaz.com 9

This Tutorial

A modern journey of metaprogramming

Highlight unique aspects of Python 3
Explode your brain

Copyright (C) 2013, http://www.dabeaz.com 10

 Target Audience

Framework/library builders
Anyone who wants to know how things work
Programmers wishing to increase "job security"

Copyright (C) 2013, http://www.dabeaz.com 11

Reading
Tutorial loosely based
on content in "Python
Cookbook, 3rd Ed."
Published May, 2013
You'll find even more
information in the
book

Copyright (C) 2013, http://www.dabeaz.com 12

 Preliminaries

Copyright (C) 2013, http://www.dabeaz.com 13

Basic Building Blocks

statement1 def func(args):
statement2 statement1
statement3 statement2
... statement3
Code ...

class A:
def method1(self, args):
statement1
statement2
def method2(self, args):
statement1
statement2
...
Copyright (C) 2013, http://www.dabeaz.com 14
 Statements
statement1
statement2
statement3
...

Perform the actual work of your program

Always execute in two scopes
globals - Module dictionary
locals - Enclosing function (if any)
exec(statements [, globals [, locals]])
Copyright (C) 2013, http://www.dabeaz.com 15

Functions
def func(x, y, z):
statement1
statement2
statement3
...

The fundamental unit of code in most programs

Module-level functions
Methods of classes

Copyright (C) 2013, http://www.dabeaz.com 16

 Calling Conventions
def func(x, y, z):
statement1
statement2
statement3
...

Positional arguments
func(1, 2, 3)

Keyword arguments
func(x=1, z=3, y=2)

Copyright (C) 2013, http://www.dabeaz.com 17

Default Arguments
def func(x, debug=False, names=None):
if names is None:
names = []
...

func(1)
func(1, names=['x', 'y'])

Default values set at definition time

Only use immutable values (e.g., None)
Copyright (C) 2013, http://www.dabeaz.com 18
 *args and **kwargs
def func(*args, **kwargs):
# args is tuple of position args
# kwargs is dict of keyword args
...

func(1, 2, x=3, y=4, z=5)

args = (1, 2) kwargs = {

'x': 3,
'y': 4,
'z': 5
}
Copyright (C) 2013, http://www.dabeaz.com 19

*args and **kwargs

args = (1, 2) kwargs = {
'x': 3,
'y': 4,
'z': 5
}

func(*args, **kwargs)

same as
func(1, 2, x=3, y=4, z=5)

Copyright (C) 2013, http://www.dabeaz.com 20

 Keyword-Only Args
def recv(maxsize, *, block=True):
...

def sum(*args, initial=0):

...

Named arguments appearing after '*' can

only be passed by keyword
recv(8192, block=False) # Ok
recv(8192, False) # Error

Copyright (C) 2013, http://www.dabeaz.com 21

Closures
You can make and return functions
def make_adder(x, y):
def add():
return x + y
return add

Local variables are captured

>>> a = make_adder(2, 3)
>>> b = make_adder(10, 20)
>>> a()
5
>>> b()
30
>>>
Copyright (C) 2013, http://www.dabeaz.com 22
 Classes
class Spam:
a = 1
def __init__(self, b):
self.b = b
def imethod(self):
pass

>>> Spam.a # Class variable

1
>>> s = Spam(2)
>>> s.b # Instance variable
2
>>> s.imethod() # Instance method
>>>

Copyright (C) 2013, http://www.dabeaz.com 23

Different Method Types

Usage
class Spam:
def imethod(self): s = Spam()
pass s.imethod()

@classmethod
def cmethod(cls): Spam.cmethod()
pass

@staticmethod
def smethod(): Spam.smethod()
pass

Copyright (C) 2013, http://www.dabeaz.com 24

 Special Methods
class Array:
def __getitem__(self, index):
...
def __setitem__(self, index, value):
...
def __delitem__(self, index):
...
def __contains__(self, item):
...

Almost everything can be customized

Copyright (C) 2013, http://www.dabeaz.com 25

Inheritance
class Base:
def spam(self):
...

class Foo(Base):
def spam(self):
...
# Call method in base class
r = super().spam()

Copyright (C) 2013, http://www.dabeaz.com 26

 Dictionaries
Objects are layered on dictionaries
class Spam:
def __init__(self, x, y):
self.x = x
self.y = y
def foo(self):
pass

Example:
>>> s = Spam(2,3)
>>> s.__dict__
{'y': 3, 'x': 2}
>>> Spam.__dict__['foo']
<function Spam.foo at 0x10069fc20>
>>>
Copyright (C) 2013, http://www.dabeaz.com 27

Metaprogramming Basics

"I love the smell of debugging in the morning."

Copyright (C) 2013, http://www.dabeaz.com 28

 Problem: Debugging

Will illustrate basics with a simple problem

Debugging
Not the only application, but simple enough
to fit on slides

Copyright (C) 2013, http://www.dabeaz.com 29

Debugging with Print

A function
def add(x, y):
return x + y

A function with debugging

def add(x, y):
print('add')
return x + y

The one and only true way to debug...

Copyright (C) 2013, http://www.dabeaz.com 30
 Many Functions w/ Debug
def add(x, y):
print('add')
return x + y

def sub(x, y):

print('sub')
return x - y

def mul(x, y):

print('mul')
return x * y

def div(x, y):

print('div')
return x / y

Copyright (C) 2013, http://www.dabeaz.com 31

Many Functions w/ Debug

def add(x, y):
print('add')
return x + y

def sub(x, y):

print('sub')
return x - y
Bleah!
def mul(x, y):
print('mul')
return x * y

def div(x, y):

print('div')
return x / y

Copyright (C) 2013, http://www.dabeaz.com 32

 Decorators

A decorator is a function that creates a

wrapper around another function
The wrapper is a new function that works
exactly like the original function (same
arguments, same return value) except that
some kind of extra processing is carried out

Copyright (C) 2013, http://www.dabeaz.com 33

A Debugging Decorator
from functools import wraps

def debug(func):
msg = func.__qualname__
@wraps(func)
def wrapper(*args, **kwargs):
print(msg)
return func(*args, **kwargs)
return wrapper

Application (wrapping)
func = debug(func)

Copyright (C) 2013, http://www.dabeaz.com 34

 A Debugging Decorator
from functools import wraps

def debug(func):
msg = func.__qualname__
@wraps(func)
def wrapper(*args, **kwargs):
print(msg)
return func(*args, **kwargs)
return wrapper

A decorator creates a "wrapper" function

Copyright (C) 2013, http://www.dabeaz.com 35

A Debugging Decorator
from functools import wraps

def debug(func):
msg = func.__qualname__
@wraps(func)
def wrapper(*args, **kwargs):
print(msg)
return func(*args, **kwargs)
return wrapper

A decorator creates a "wrapper" function

Around a function that you provide

Copyright (C) 2013, http://www.dabeaz.com 36

 Function Metadata
from functools import wraps

def debug(func):
msg = func.__qualname__
@wraps(func)
def wrapper(*args, **kwargs):
print(msg)
return func(*args, **kwargs)
return wrapper

@wraps copies metadata

Name and doc string
Function attributes
Copyright (C) 2013, http://www.dabeaz.com 37

The Metadata Problem

If you don't use @wraps, weird things happen
def add(x,y):
"Adds x and y"
return x+y
add = debug(add)

>>> add.__qualname__
'wrapper'
>>> add.__doc__
>>> help(add)
Help on function wrapper in module
__main__:

wrapper(*args, **kwargs)
>>>
Copyright (C) 2013, http://www.dabeaz.com 38
 Decorator Syntax
The definition of a function and wrapping
almost always occur together
def add(x,y):
return x+y
add = debug(add)

@decorator syntax performs the same steps

@debug
def add(x,y):
return x+y

Copyright (C) 2013, http://www.dabeaz.com 39

Example Use
@debug
def add(x, y):
return x + y

@debug
def sub(x, y):
return x - y

@debug
def mul(x, y):
return x * y

@debug
def div(x, y):
return x / y

Copyright (C) 2013, http://www.dabeaz.com 40

 Example Use
@debug
def add(x, y):
return x + y

@debug Each function is

def sub(x, y):
return x - y decorated, but
there are no other
@debug
def mul(x, y):
implementation
return x * y details
@debug
def div(x, y):
return x / y

Copyright (C) 2013, http://www.dabeaz.com 41

Big Picture

Debugging code is isolated to single location

This makes it easy to change (or to disable)
User of a decorator doesn't worry about it
That's really the whole idea

Copyright (C) 2013, http://www.dabeaz.com 42

 Variation: Logging
from functools import wraps
import logging

def debug(func):
log = logging.getLogger(func.__module__)
msg = func.__qualname__
@wraps(func)
def wrapper(*args, **kwargs):
log.debug(msg)
return func(*args, **kwargs)
return wrapper

Copyright (C) 2013, http://www.dabeaz.com 43

Variation: Optional Disable

from functools import wraps
import os

def debug(func):
if 'DEBUG' not in os.environ:
return func
msg = func.__qualname__
@wraps(func)
def wrapper(*args, **kwargs):
print(msg)
return func(*args, **kwargs)
return wrapper

Key idea: Can change decorator independently of

code that uses it
Copyright (C) 2013, http://www.dabeaz.com 44
 Debugging with Print
A function with debugging
def add(x, y):
print('add')
return x + y

Everyone knows you really need a prefix

def add(x, y):
print('***add')
return x + y

You know, for grepping...

Copyright (C) 2013, http://www.dabeaz.com 45

Decorators with Args

Calling convention
@decorator(args)
def func():
pass

Evaluates as
func = decorator(args)(func)

It's a little weird--two levels of calls

Copyright (C) 2013, http://www.dabeaz.com 46

 Decorators with Args
from functools import wraps

def debug(prefix=''):
def decorate(func):
msg = prefix + func.__qualname__
@wraps(func)
def wrapper(*args, **kwargs):
print(msg)
return func(*args, **kwargs)
return wrapper
return decorate

Usage
@debug(prefix='***')
def add(x,y):
return x+y
Copyright (C) 2013, http://www.dabeaz.com 47

Decorators with Args

from functools import wraps

def debug(prefix=''):
def decorate(func):
msg = prefix + func.__qualname__
@wraps(func)
def wrapper(*args, **kwargs):
print(msg)
return func(*args, **kwargs)
return wrapper
return decorate

Outer function defines Normal

variables for use in regular decorator function
decorator
Copyright (C) 2013, http://www.dabeaz.com 48
 A Reformulation
from functools import wraps, partial

def debug(func=None, *, prefix=''):

if func is None:
return partial(debug, prefix=prefix)

msg = prefix + func.__qualname__

@wraps(func)
def wrapper(*args, **kwargs):
print(msg)
return func(*args, **kwargs)
return wrapper

A test of your function calling skills...

Copyright (C) 2013, http://www.dabeaz.com 49

Usage
Use as a simple decorator
@debug
def add(x, y):
return x + y

Or as a decorator with optional configuration

@debug(prefix='***')
def add(x, y):
return x + y

Copyright (C) 2013, http://www.dabeaz.com 50

 Debug All Of This
Debug all of the methods of a class
class Spam:
@debug
def grok(self):
pass
@debug
def bar(self):
pass
@debug
def foo(self):
pass

Can you decorate all methods at once?

Copyright (C) 2013, http://www.dabeaz.com 51

Class Decorator
def debugmethods(cls):
for name, val in vars(cls).items():
if callable(val):
setattr(cls, name, debug(val))
return cls

Idea:
Walk through class dictionary
Identify callables (e.g., methods)
Wrap with a decorator
Copyright (C) 2013, http://www.dabeaz.com 52
 Example Use
@debugmethods
class Spam:
def grok(self):
pass
def bar(self):
pass
def foo(self):
pass

One decorator application

Covers all definitions within the class
It even mostly works...
Copyright (C) 2013, http://www.dabeaz.com 53

Limitations
@debugmethods
class BrokenSpam:
@classmethod
def grok(cls): # Not wrapped
pass
@staticmethod
def bar(): # Not wrapped
pass

Only instance methods get wrapped

Why? An exercise for the reader...
Copyright (C) 2013, http://www.dabeaz.com 54
 Variation: Debug Access
def debugattr(cls):
orig_getattribute = cls.__getattribute__

def __getattribute__(self, name):

print('Get:', name)
return orig_getattribute(self, name)
cls.__getattribute__ = __getattribute__

return cls

Rewriting part of the class itself

Copyright (C) 2013, http://www.dabeaz.com 55

Example
@debugattr
class Point:
def __init__(self, x, y):
self.x = x
self.y = y

>>> p = Point(2, 3)
>>> p.x
Get: x
2
>>> p.y
Get: y
3
>>>

Copyright (C) 2013, http://www.dabeaz.com 56

 Debug All The Classes
@debugmethods

Many classes with debugging

class Base:
...

@debugmethods Didn't we just solve this?

Bleah!!
class Spam(Base):
...

@debugmethods
class Grok(Spam):
...

@debugmethods
class Mondo(Grok):
...

Copyright (C) 2013, http://www.dabeaz.com 57

Solution: A Metaclass
class debugmeta(type):
def __new__(cls, clsname, bases, clsdict):
clsobj = super().__new__(cls, clsname,
bases, clsdict)
clsobj = debugmethods(clsobj)
return clsobj

Usage
class Base(metaclass=debugmeta):
...

class Spam(Base):
...

Copyright (C) 2013, http://www.dabeaz.com 58

 Solution: A Metaclass
class debugmeta(type):
def __new__(cls, clsname, bases, clsdict):
clsobj = super().__new__(cls, clsname,
bases, clsdict)
clsobj = debugmethods(clsobj)
return clsobj

Idea
Class gets created normally

Copyright (C) 2013, http://www.dabeaz.com 59

Solution: A Metaclass
class debugmeta(type):
def __new__(cls, clsname, bases, clsdict):
clsobj = super().__new__(cls, clsname,
bases, clsdict)
clsobj = debugmethods(clsobj)
return clsobj

Idea
Class gets created normally
Immediately wrapped by class decorator

Copyright (C) 2013, http://www.dabeaz.com 60

Copyright (C) 2013, http://www.dabeaz.com 61

Types
All values in Python have a type
Example:
>>> x = 42
>>> type(x)
<type 'int'>
>>> s = "Hello"
>>> type(s)
<type 'str'>
>>> items = [1,2,3]
>>> type(items)
<type 'list'>
>>>

Copyright (C) 2013, http://www.dabeaz.com 62

 Types and Classes
Classes define new types
class Spam:
pass

>>> s = Spam()
>>> type(s)
<class '__main__.Spam'>
>>>

The class is the type of instances created

The class is a callable that creates instances
Copyright (C) 2013, http://www.dabeaz.com 63

Types of Classes
Classes are instances of types
>>> type(int)
<class 'int'>
>>> type(list)
<class 'list'>
>>> type(Spam)
<class '__main__.Spam'>
>>> isinstance(Spam, type)
True
>>>

This requires some thought, but it should

make some sense (classes are types)

Copyright (C) 2013, http://www.dabeaz.com 64

 Creating Types
Types are their own class (builtin)
class type:
...

>>> type
<class 'type'>
>>>

This class creates new "type" objects

Used when defining classes
Copyright (C) 2013, http://www.dabeaz.com 65

Classes Deconstructed
Consider a class:
class Spam(Base):
def __init__(self, name):
self.name = name
def bar(self):
print "I'm Spam.bar"

What are its components?

Name ("Spam")
Base classes (Base,)
Functions (__init__,bar)
Copyright (C) 2013, http://www.dabeaz.com 66
 Class Definition Process
What happens during class definition?
class Spam(Base):
def __init__(self, name):
self.name = name
def bar(self):
print "I'm Spam.bar"

Step1: Body of class is isolated

body = '''
def __init__(self, name):
self.name = name
def bar(self):
print "I'm Spam.bar"
'''

Copyright (C) 2013, http://www.dabeaz.com 67

Class Definition
Step 2: The class dictionary is created
clsdict = type.__prepare__('Spam', (Base,))

This dictionary serves as local namespace for

statements in the class body
By default, it's a simple dictionary (more later)

Copyright (C) 2013, http://www.dabeaz.com 68

 Class Definition
Step 3: Body is executed in returned dict
exec(body, globals(), clsdict)

Afterwards, clsdict is populated

>>> clsdict
{'__init__': <function __init__ at 0x4da10>,
'bar': <function bar at 0x4dd70>}
>>>

Copyright (C) 2013, http://www.dabeaz.com 69

Class Definition
Step 4: Class is constructed from its name,
base classes, and the dictionary
>>> Spam = type('Spam', (Base,), clsdict)
>>> Spam
<class '__main__.Spam'>
>>> s = Spam('Guido')
>>> s.bar()
I'm Spam.bar
>>>

Copyright (C) 2013, http://www.dabeaz.com 70

 Changing the Metaclass
metaclass keyword argument
Sets the class used for creating the type
class Spam(metaclass=type):
def __init__(self,name):
self.name = name
def bar(self):
print "I'm Spam.bar"

By default, it's set to 'type', but you can

change it to something else

Copyright (C) 2013, http://www.dabeaz.com 71

Defining a New Metaclass

You typically inherit from type and redefine
__new__ or __init__
class mytype(type):
def __new__(cls, name, bases, clsdict):
clsobj = super().__new__(cls,
name,
bases,
clsdict)
return clsobj

To use
class Spam(metaclass=mytype):
...

Copyright (C) 2013, http://www.dabeaz.com 72

 Using a Metaclass

Metaclasses get information about class

definitions at the time of definition
Can inspect this data
Can modify this data
Essentially, similar to a class decorator
Question: Why would you use one?

Copyright (C) 2013, http://www.dabeaz.com 73

Inheritance
Metaclasses propagate down hierarchies
class Base(metaclass=mytype):
...

class Spam(Base): # metaclass=mytype

...

class Grok(Spam): # metaclass=mytype

...

Think of it as a genetic mutation

Copyright (C) 2013, http://www.dabeaz.com 74
 Solution: Reprise
class debugmeta(type):
def __new__(cls, clsname, bases, clsdict):
clsobj = super().__new__(cls, clsname,
bases, clsdict)
clsobj = debugmethods(clsobj)
return clsobj

Idea
Class gets created normally
Immediately wrapped by class decorator

Copyright (C) 2013, http://www.dabeaz.com 75

Debug The Universe

class Base(metaclass=debugmeta):
...

class Spam(Base):
... Debugging gets applied
across entire hierarchy
class Grok(Spam):
... Implicitly applied in
class Mondo(Grok):
subclasses
...

Copyright (C) 2013, http://www.dabeaz.com 76

 Big Picture

It's mostly about wrapping/rewriting

Decorators : Functions
Class Decorators: Classes
Metaclasses : Class hierarchies
You have the power to change things

Copyright (C) 2013, http://www.dabeaz.com 77

Interlude

Copyright (C) 2013, http://www.dabeaz.com 78

 Journey So Far

Have seen "classic" metaprogramming

Already widely used in Python 2
Only a few Python 3 specific changes

Copyright (C) 2013, http://www.dabeaz.com 79

Journey to Come

Let's build something more advanced

Using techniques discussed
And more...

Copyright (C) 2013, http://www.dabeaz.com 80

 Problem : Structures
class Stock:
def __init__(self, name, shares, price):
self.name = name
self.shares = shares
self.price = price

class Point:
def __init__(self, x, y):
self.x = x
self.y = y

class Host:
def __init__(self, address, port):
self.address = address
self.port = port

Copyright (C) 2013, http://www.dabeaz.com 81

Problem : Structures
class Stock:
def __init__(self, name, shares, price):
self.name = name
self.shares = shares
self.price = price
Why must I keep
class Point: writing these
def __init__(self, x, y):
self.x = x
boilerplate init
self.y = y methods?
class Host:
def __init__(self, address, port):
self.address = address
self.port = port

Copyright (C) 2013, http://www.dabeaz.com 82

 A Solution : Inheritance
class Structure:
_fields = []
A generalized
def __init__(self, *args): __init__()
if len(args) != self._fields:
raise TypeError('Wrong # args')
for name, val in zip(self._fields, args):
setattr(self, name, val)

class Stock(Structure):
_fields = ['name', 'shares', 'price']

class Point(Structure):
_fields = ['x', 'y']

class Host(Structure):
_fields = ['address', 'port']
Copyright (C) 2013, http://www.dabeaz.com 83

Usage
>>> s = Stock('ACME', 50, 123.45)
>>> s.name
'ACME'
>>> s.shares
50
>>> s.price
123.45

>>> p = Point(4, 5)
>>> p.x
4
>>> p.y
5
>>>

Copyright (C) 2013, http://www.dabeaz.com 84

 Some Issues
No support for keyword args
>>> s = Stock('ACME', price=123.45, shares=50)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: __init__() got an unexpected keyword
argument 'shares'
>>>

Missing calling signatures

>>> import inspect
>>> print(inspect.signature(Stock))
(*args)
>>>

Copyright (C) 2013, http://www.dabeaz.com 85

Put a Signature on It

Copyright (C) 2013, http://www.dabeaz.com 86

 New Approach: Signatures
Build a function signature object
from inspect import Parameter, Signature

fields = ['name', 'shares', 'price']

parms = [ Parameter(name,
Parameter.POSITIONAL_OR_KEYWORD)
for name in fields]
sig = Signature(parms)

Signatures are more than just metadata

Copyright (C) 2013, http://www.dabeaz.com 87

Signature Binding
Argument binding
def func(*args, **kwargs):
bound_args = sig.bind(*args, **kwargs)
for name, val in bound_args.arguments.items():
print(name, '=', val)

sig.bind() binds positional/keyword args to signature

.arguments is an OrderedDict of passed values

Copyright (C) 2013, http://www.dabeaz.com 88

 Signature Binding
Example use:
>>> func('ACME', 50, 91.1)
name = ACME
shares = 50
price = 91.1

>>> func('ACME', price=91.1, shares=50)

name = ACME
shares = 50
price = 91.1

Notice: both positional/keyword args work

Copyright (C) 2013, http://www.dabeaz.com 89

Signature Binding
Error handling
>>> func('ACME', 50)
Traceback (most recent call last):
...
TypeError: 'price' parameter lacking default value

>>> func('ACME', 50, 91.1, 92.3)

Traceback (most recent call last):
...
TypeError: too many positional arguments
>>>

Binding: it just "works"

Copyright (C) 2013, http://www.dabeaz.com 90
 Solution w/Signatures
from inspect import Parameter, Signature

def make_signature(names):
return Signature(
Parameter(name,
Parameter.POSITIONAL_OR_KEYWORD)
for name in names)

class Structure:
__signature__ = make_signature([])
def __init__(self, *args, **kwargs):
bound = self.__signature__.bind(
*args, **kwargs)
for name, val in bound.arguments.items():
setattr(self, name, val)

Copyright (C) 2013, http://www.dabeaz.com 91

Solution w/Signatures
class Stock(Structure):
__signature__ = make_signature(
['name','shares','price'])

class Point(Structure):
__signature__ = make_signature(['x', 'y'])

class Host(Structure):
__signature__ = make_signature(
['address', 'port'])

Copyright (C) 2013, http://www.dabeaz.com 92

 Solution w/Signatures
>>> s = Stock('ACME', shares=50, price=91.1)
>>> s.name
'ACME'
>>> s.shares
50
>>> s.price
91.1
>>> import inspect
>>> print(inspect.signature(Stock))
(name, shares, price)
>>>

Copyright (C) 2013, http://www.dabeaz.com 93

New Problem
This is rather annoying
class Stock(Structure):
__signature__ = make_signature(
['name','shares','price'])

class Point(Structure):
__signature__ = make_signature(['x', 'y'])

class Host(Structure):
__signature__ = make_signature(
['address', 'port'])

Can't it be simplified in some way?

Copyright (C) 2013, http://www.dabeaz.com 94
 Solutions

Ah, a problem involving class definitions

Class decorators
Metaclasses
Which seems more appropriate?
Let's explore both options

Copyright (C) 2013, http://www.dabeaz.com 95

Class Decorators
def add_signature(*names):
def decorate(cls):
cls.__signature__ = make_signature(names)
return cls
return decorate

Usage:
@add_signature('name','shares','price')
class Stock(Structure):
pass

@add_signature('x','y')
class Point(Structure):
pass

Copyright (C) 2013, http://www.dabeaz.com 96

 Metaclass Solution
class StructMeta(type):
def __new__(cls, name, bases, clsdict):
clsobj = super().__new__(cls, name,
bases, clsdict)
sig = make_signature(clsobj._fields)
setattr(clsobj, '__signature__', sig)
return clsobj

class Structure(metaclass=StructMeta):
_fields = []
def __init__(self, *args, **kwargs):
bound = self.__signature__.bind(
*args, **kwargs)
for name, val in bound.arguments.items():
setattr(self, name, val)

Copyright (C) 2013, http://www.dabeaz.com 97

Metaclass Solution
class StructMeta(type):
def __new__(cls, name, bases, clsdict):
clsobj = super().__new__(cls, name,
bases, clsdict)
sig = make_signature(clsobj._fields)
setattr(clsobj, '__signature__', sig)
return clsobj

Read _fields attribute

class Structure(metaclass=StructMeta):
_fields = []
and
defmake a proper *args, **kwargs):
__init__(self,
signature
boundout of it
= self.__signature__.bind(
*args, **kwargs)
for name, val in bound.arguments.items():
setattr(self, name, val)

Copyright (C) 2013, http://www.dabeaz.com 98

 Usage
class Stock(Structure):
_fields = ['name', 'shares', 'price']

class Point(Structure):
_fields = ['x', 'y']

class Host(Structure):
_fields = ['address', 'port']

It's back to original 'simple' solution

Signatures are created behind scenes
Copyright (C) 2013, http://www.dabeaz.com 99

Considerations
How much will the Structure class be expanded?
Example: supporting methods
class Structure(metaclass=StructMeta):
_fields = []
...
def __repr__(self):
args = ', '.join(repr(getattr(self, name))
for name in self._fields)
return type(self).__name__ + \
'(' + args + ')'

Is type checking important?

isinstance(s, Structure)

Copyright (C) 2013, http://www.dabeaz.com 100

 Advice
Use a class decorator if the goal is to tweak
classes that might be unrelated
Use a metaclass if you're trying to perform
actions in combination with inheritance
Don't be so quick to dismiss techniques
(e.g., 'metaclasses suck so .... blah blah')
All of the tools are meant to work together

Copyright (C) 2013, http://www.dabeaz.com 101

Owning the Dot

Q: "Who's in charge here?"

A: "In charge? I don't know, man."
Copyright (C) 2013, http://www.dabeaz.com 102
 Problem : Correctness

Types like a duck, rhymes with ...

>>> s = Stock('ACME', 50, 91.1)
>>> s.name = 42
>>> s.shares = 'a heck of a lot'
>>> s.price = (23.45 + 2j)
>>>

Bah, real programmers use Haskell!

Copyright (C) 2013, http://www.dabeaz.com 103

Properties
You can upgrade attributes to have checks
class Stock(Structure):
_fields = ['name', 'shares', 'price']

@property
def shares(self):
(getter) return self._shares

@shares.setter
def shares(self, value):
if not isinstance(value, int):
raise TypeError('expected int')
(setter) if value < 0:
raise ValueError('Must be >= 0')
self._shares = value

Copyright (C) 2013, http://www.dabeaz.com 104

 Properties
Example use:
>>> s = Stock('ACME', 50, 91.1)
>>> s.shares = 'a lot'
Traceback (most recent call last):
...
TypeError: expected int
>>> s.shares = -10
Traceback (most recent call last):
...
ValueError: Must be >= 0
>>> s.shares = 37
>>> s.shares
37
>>>

Copyright (C) 2013, http://www.dabeaz.com 105

An Issue
It works, but it quickly gets annoying
@property
def shares(self):
return self._shares

@shares.setter
def shares(self, value):
if not isinstance(value, int):
raise TypeError('expected int')
if value < 0:
raise ValueError('Must be >= 0')
self._shares = value

Imagine writing same code for many attributes

Copyright (C) 2013, http://www.dabeaz.com 106
 A Complexity
Want to simplify, but how?
Two kinds of checking are intertwined
Type checking: int, float, str, etc.
Validation: >, >=, <, <=, !=, etc.
Question: How to structure it?

Copyright (C) 2013, http://www.dabeaz.com 107

Descriptor Protocol
Properties are implemented via descriptors
class Descriptor:
def __get__(self, instance, cls):
...
def __set__(self, instance, value):
...
def __delete__(self, instance)
...

Customized processing of attribute access

Copyright (C) 2013, http://www.dabeaz.com 108

 Descriptor Protocol
Example:
class Spam:
x = Descriptor()
s = Spam()

s.x # x.__get__(s, Spam)

s.x = value # x.__set__(s, value)
del s.x # x.__delete__(s)

Customized handling of a specific attribute

Copyright (C) 2013, http://www.dabeaz.com 109

A Basic Descriptor
class Descriptor:
def __init__(self, name=None):
self.name = name

def __get__(self, instance, cls):

if instance is None:
return self
else:
return instance.__dict__[self.name]

def __set__(self, instance, value):

instance.__dict__[self.name] = value

def __delete__(self, instance):

del instance.__dict__[self.name]

Copyright (C) 2013, http://www.dabeaz.com 110

 A Basic Descriptor
class Descriptor:
def __init__(self, name=None):
self.name = name

def __get__(self, instance, cls):

name of attribute
if instance is None:
return self being stored. A key
else:
in the instance dict.
return instance.__dict__[self.name]

def __set__(self, instance, value):

instance.__dict__[self.name] = value

def __delete__(self, instance):

del instance.__dict__[self.name]

Copyright (C) 2013, http://www.dabeaz.com 111

A Basic Descriptor
class Descriptor:
def __init__(self, name=None):
self.name = name

def __get__(self, instance, Direct

cls): manipulation
if instance is None: of the instance dict.
return self
else:
return instance.__dict__[self.name]

def __set__(self, instance, value):

instance.__dict__[self.name] = value

def __delete__(self, instance):

del instance.__dict__[self.name]

Copyright (C) 2013, http://www.dabeaz.com 112

 A Simpler Descriptor
class Descriptor:
def __init__(self, name=None):
self.name = name

def __set__(self, instance, value):

instance.__dict__[self.name] = value

def __delete__(self, instance):

raise AttributeError("Can't delete")

You don't need __get__() if it merely returns

the normal dictionary value

Copyright (C) 2013, http://www.dabeaz.com 113

Descriptor Usage
class Stock(Structure):
_fields = ['name', 'shares', 'price']
name = Descriptor('name')
shares = Descriptor('shares')
price = Descriptor('price')

If it works, will capture set/delete operations

>>> s = Stock('ACME', 50, 91.1)
>>> s.shares
50
>>> s.shares = 50 # shares.__set__(s, 50)
>>> del s.shares
Traceback (most recent call last):
...
AttributeError: Can't delete
>>>
Copyright (C) 2013, http://www.dabeaz.com 114
 Type Checking
class Typed(Descriptor):
ty = object
def __set__(self, instance, value):
if not isinstance(value, self.ty):
raise TypeError('Expected %s' % self.ty)
super().__set__(instance, value)

Specialization
class Integer(Typed):
ty = int
class Float(Typed):
ty = float
class String(Typed):
ty = str

Copyright (C) 2013, http://www.dabeaz.com 115

Usage
class Stock(Structure):
_fields = ['name', 'shares', 'price']
name = String('name')
shares = Integer('shares')
price = Float('price')

Example:
>>> s = Stock('ACME', 50, 91.1)
>>> s.shares = 'a lot'
Traceback (most recent call last):
...
TypeError: Expected <class 'int'>
>>> s.name = 42
Traceback (most recent call last):
...
TypeError: Expected <class 'str'>
>>>
Copyright (C) 2013, http://www.dabeaz.com 116
 Value Checking
class Positive(Descriptor):
def __set__(self, instance, value):
if value < 0:
raise ValueError('Expected >= 0')
super().__set__(instance, value)

Use as a mixin class

class PosInteger(Integer, Positive):
pass

class PosFloat(Float, Positive):

pass

Copyright (C) 2013, http://www.dabeaz.com 117

Usage
class Stock(Structure):
_fields = ['name', 'shares', 'price']
name = String('name')
shares = PosInteger('shares')
price = PosFloat('price')

Example:
>>> s = Stock('ACME', 50, 91.1)
>>> s.shares = -10
Traceback (most recent call last):
...
ValueError: Expected >= 0
>>> s.shares = 'a lot'
Traceback (most recent call last):
...
TypeError: Expected <class 'int'>
>>>
Copyright (C) 2013, http://www.dabeaz.com 118
 Building Blocks!
class PosInteger(Integer, Positive):
pass

super()

Copyright (C) 2013, http://www.dabeaz.com 119

Understanding the MRO

class PosInteger(Integer, Positive):
pass

>>> PosInteger.__mro__
(<class 'PosInteger'>, This chain defines the
<class 'Integer'>, order in which the
<class 'Typed'>,
<class 'Positive'>, value is checked by
<class 'Descriptor'>, different __set__()
<class 'object'>) methods
>>>

Base order matters (e.g., int before < 0)

Copyright (C) 2013, http://www.dabeaz.com 120
 Length Checking
class Sized(Descriptor):
def __init__(self, *args, maxlen, **kwargs):
self.maxlen = maxlen
super().__init__(*args, **kwargs)

def __set__(self, instance, value):

if len(value) > self.maxlen:
raise ValueError('Too big')
super().__set__(instance, value)

Use:
class SizedString(String, Sized):
pass

Copyright (C) 2013, http://www.dabeaz.com 121

Usage
class Stock(Structure):
_fields = ['name', 'shares', 'price']
name = SizedString('name', maxlen=8)
shares = PosInteger('shares')
price = PosFloat('price')

Example:
>>> s = Stock('ACME', 50, 91.1)
>>> s.name = 'ABRACADABRA'
Traceback (most recent call last):
...
ValueError: Too big
>>>

Copyright (C) 2013, http://www.dabeaz.com 122

 Pattern Checking
import re
class Regex(Descriptor):
def __init__(self, *args, pat, **kwargs):
self.pat = re.compile(pat)
super().__init__(*args, **kwargs)

def __set__(self, instance, value):

if not self.pat.match(value):
raise ValueError('Invalid string')
super().__set__(instance, value)

Use:
class SizedRegexString(String, Sized, Regex):
pass

Copyright (C) 2013, http://www.dabeaz.com 123

Usage
class Stock(Structure):
_fields = ['name', 'shares', 'price']
name = SizedRegexString('name', maxlen=8,
pat='[A-Z]+$')
shares = PosInteger('shares')
price = PosFloat('price')

Example:
>>> s = Stock('ACME', 50, 91.1)
>>> s.name = 'Head Explodes!'
Traceback (most recent call last):
...
ValueError: Invalid string
>>>

Copyright (C) 2013, http://www.dabeaz.com 124

 Whoa, Whoa, Whoa
Mixin classes with __init__() functions?
class SizedRegexString(String, Sized, Regex):
pass

Each with own unique signature

a = String('name')
b = Sized(maxlen=8)
c = Regex(pat='[A-Z]+$')

This works, how?

Copyright (C) 2013, http://www.dabeaz.com 125

Keyword-only Args
SizedRegexString('name', maxlen=8, pat='[A-Z]+$')

class Descriptor:
def __init__(self, name=None):
...

class Sized(Descriptor):
def __init__(self, *args, maxlen, **kwargs):
...
super().__init__(*args, **kwargs)

class Regex(Descriptor):
def __init__(self, *args, pat, **kwargs):
...
super().__init__(*args, **kwargs)

Copyright (C) 2013, http://www.dabeaz.com 126

 Keyword-only Args
SizedRegexString('name', maxlen=8, pat='[A-Z]+$')

class Descriptor:
def __init__(self, name=None):
...

class Sized(Descriptor):
def __init__(self, *args, maxlen, **kwargs):
...
super().__init__(*args, **kwargs)

class Regex(Descriptor):
Keyword-only argument is
def __init__(self, *args, pat, **kwargs):
isolated
... and removed from
all other passed args
super().__init__(*args, **kwargs)

Copyright (C) 2013, http://www.dabeaz.com 127

"Awesome, man!"
Copyright (C) 2013, http://www.dabeaz.com 128
 Annoyance
class Stock(Structure):
_fields = ['name', 'shares', 'price']
name = SizedRegexString('name', maxlen=8,
pat='[A-Z]+$')
shares = PosInteger('shares')
price = PosFloat('price')

Still quite a bit of repetition

Signatures and type checking not unified
Maybe we can push it further
Copyright (C) 2013, http://www.dabeaz.com 129

Copyright (C) 2013, http://www.dabeaz.com 130

 A New Metaclass
from collections import OrderedDict
class StructMeta(type):
@classmethod
def __prepare__(cls, name, bases):
return OrderedDict()

def __new__(cls, name, bases, clsdict):

fields = [ key for key, val in clsdict.items()
if isinstance(val, Descriptor) ]
for name in fields:
clsdict[name].name = name

clsobj = super().__new__(cls, name, bases,

dict(clsdict))
sig = make_signature(fields)
setattr(clsobj, '__signature__', sig)
return clsobj
Copyright (C) 2013, http://www.dabeaz.com 131

New Usage

class Stock(Structure):
name = SizedRegexString(maxlen=8,pat='[A-Z]+$')
shares = PosInteger()
price = PosFloat()

Oh, that's rather nice...

Copyright (C) 2013, http://www.dabeaz.com 132

 New Metaclass
from collections import OrderedDict
class StructMeta(type):
@classmethod
def __prepare__(cls, name, bases):
return OrderedDict()

__prepare__()
def __new__(cls, name, bases, clsdict): creates
fields = [ key for key, val in clsdict.items()
and returns
if isinstance(val, dictionary]
Descriptor)
for name in fields: to use for execution of
clsdict[name].name = name
the class body.
clsobj = super().__new__(cls, name, bases,
An OrderedDict will
dict(clsdict))
sig = make_signature(fields)
preserve the definition
setattr(clsobj, '__signature__', sig)
return clsobj order.
Copyright (C) 2013, http://www.dabeaz.com 133

Ordering of Definitions
class Stock(Structure):
name = SizedRegexString(maxlen=8,pat='[A-Z]+$')
shares = PosInteger()
price = PosFloat()

clsdict = OrderedDict(
('name', <class 'SizedRegexString'>),
('shares', <class 'PosInteger'>),
('price', <class 'PosFloat'>)
)

Copyright (C) 2013, http://www.dabeaz.com 134

 Duplicate Definitions
If inclined, you could do even better
Make a new kind of dict
class NoDupOrderedDict(OrderedDict):
def __setitem__(self, key, value):
if key in self:
raise NameError('%s already defined'
% key)
super().__setitem__(key, value)

Use in place of OrderedDict

Copyright (C) 2013, http://www.dabeaz.com 135

Duplicate Definitions
class Stock(Structure):
name = String()
shares = PosInteger()
price = PosFloat()
shares = PosInteger()

Traceback (most recent call last):

File "<stdin>", line 1, in <module>
File "<stdin>", line 5, in Stock
File "./typestruct.py", line 107, in __setitem__
raise NameError('%s already defined' % key)
NameError: shares already defined

Won't pursue further, but you get the idea

Copyright (C) 2013, http://www.dabeaz.com 136
 New Metaclass
from collections import OrderedDict
class StructMeta(type):
@classmethod
def __prepare__(cls, name, bases):
return OrderedDict()

def __new__(cls, name, bases, clsdict):

fields = [ key for key, val in clsdict.items()
if isinstance(val, Descriptor) ]
for name in fields:
clsdict[name].name = name

clsobj = super().__new__(cls, name, bases,

Collect Descriptors and dict(clsdict))
set their
sig =names
make_signature(fields)
setattr(clsobj, '__signature__', sig)
return clsobj
Copyright (C) 2013, http://www.dabeaz.com 137

Name Setting
Old code
class Stock(Structure):
_fields = ['name', 'shares', 'price']
name = SizedRegexString('name', ...)
shares = PosInteger('shares')
price = PosFloat('price')

New Code
class Stock(Structure):
name = SizedRegexString(...)
shares = PosInteger()
price = PosFloat()

Names are set from dict keys

Copyright (C) 2013, http://www.dabeaz.com 138
 New Metaclass
from collections import OrderedDict
class StructMeta(type):
@classmethod
def __prepare__(cls, name, bases):
return OrderedDict()

def __new__(cls, name, bases, clsdict):

fields = [ key for key, val in clsdict.items()
Make the class and signature
if isinstance(val, Descriptor) ]
for name inexactly
fields:as before.
clsdict[name].name = name

clsobj = super().__new__(cls, name, bases,

dict(clsdict))
sig = make_signature(fields)
setattr(clsobj, '__signature__', sig)
return clsobj
Copyright (C) 2013, http://www.dabeaz.com 139

New Metaclass
from collections import OrderedDict
class StructMeta(type):
@classmethod
def __prepare__(cls, name, bases):
return OrderedDict()

def __new__(cls, name, bases, clsdict):

fields = [ key for key, val in clsdict.items()
if isinstance(val, Descriptor) ]
A technicality:
for name in fields:Must create a
proper dict for class contents
clsdict[name].name = name

clsobj = super().__new__(cls, name, bases,

dict(clsdict))
sig = make_signature(fields)
setattr(clsobj, '__signature__', sig)
return clsobj
Copyright (C) 2013, http://www.dabeaz.com 140
 Performance

Copyright (C) 2013, http://www.dabeaz.com 141

The Costs
Option 1 : Simple
class Stock:
def __init__(self, name, shares, price):
self.name = name
self.shares = shares
self.price = price

Option 2 : Meta
class Stock(Structure):
name = SizedRegexString(...)
shares = PosInteger()
price = PosFloat()

Copyright (C) 2013, http://www.dabeaz.com 142

 A Few Tests
Instance creation Simple Meta
s = Stock('ACME', 50, 91.1) 1.07s 91.8s
(86x)
Attribute lookup
s.price 0.08s 0.08s

Attribute assignment
s.price = 10.0 0.11s 3.40s
(31x)
Attribute assignment
s.name = 'ACME' 0.14s 8.14s
(58x)

Copyright (C) 2013, http://www.dabeaz.com 143

A Few Tests
Instance creation Simple Meta
s = Stock('ACME', 50, 91.1) 1.07s 91.8s
(86x)
Attribute lookup
s.price 0.08s 0.08s

Attribute assignment A bright

s.price = 10.0 spot 0.11s 3.40s
(31x)
Attribute assignment
s.name = 'ACME' 0.14s 8.14s
(58x)

Copyright (C) 2013, http://www.dabeaz.com 144

 Thoughts

Several large bottlenecks

Signature enforcement
Multiple inheritance/super in descriptors
Can anything be done without a total rewrite?

Copyright (C) 2013, http://www.dabeaz.com 145

Code Generation
def _make_init(fields):
code = 'def __init__(self, %s):\n' % \
', '.join(fields)
for name in fields:
code += ' self.%s = %s\n' % (name, name)
return code

Example:
>>> code = _make_init(['name','shares','price'])
>>> print(code)
def __init__(self, name, shares, price):
self.name = name
self.shares = shares
self.price = price

>>>
Copyright (C) 2013, http://www.dabeaz.com 146
 Code Generation
class StructMeta(type):
...
def __new__(cls, name, bases, clsdict):
fields = [ key for key, val in clsdict.items()
if isinstance(val, Descriptor) ]

for name in fields:

clsdict[name].name = name

if fields:
exec(_make_init(fields),globals(),clsdict)

clsobj = super().__new__(cls, name, bases,

dict(clsdict))
setattr(clsobj, '_fields', fields)
return clsobj
Copyright (C) 2013, http://www.dabeaz.com 147

Code Generation
class StructMeta(type):
...
def __new__(cls, name, bases, clsdict):
fields = [ key for key, val in clsdict.items()
if isinstance(val, Descriptor) ]

for name in fields:

clsdict[name].name = name

if fields:
exec(_make_init(fields),globals(),clsdict)

clsobj = super().__new__(cls, name, bases,

dict(clsdict))
setattr(clsobj, '_fields', fields)
return clsobj No signature, but set _fields
Copyright (C) 2013, http://www.dabeaz.com for code that wants it 148
 New Code
class Structure(metaclass=StructMeta):
pass

class Stock(Structure):
name = SizedRegexString(...)
shares = PosInteger()
price = PosFloat()

Instance creation:
Simple 1.1s
Old Meta (w/signatures) 91.8s
New Meta (w/exec) 17. 6s

Copyright (C) 2013, http://www.dabeaz.com 149

New Thought
class Descriptor:
...
def __set__(self, instance, value):
instance.__dict__[self.name] = value

class Typed(Descriptor):
def __set__(self, instance, value):
if not isinstance(value, self.ty):
raise TypeError('Expected %s' % self.ty)
super().__set__(instance, value)
Could you merge
class Positive(Descriptor):
this code together?
def __set__(self, instance, value):
if value < 0:
raise ValueError('Expected >= 0')
super().__set__(instance, value)
Copyright (C) 2013, http://www.dabeaz.com 150
 Reformulation
class Descriptor(metaclass=DescriptorMeta):
def __init__(self, name=None):
self.name = name

@staticmethod
def set_code():
return [
'instance.__dict__[self.name] = value'
]

def __delete__(self, instance):

raise AttributeError("Can't delete")

Change __set__ to a method that returns source

Introduce a new metaclass (later)
Copyright (C) 2013, http://www.dabeaz.com 151

Reformulation
class Typed(Descriptor):
ty = object
@staticmethod
def set_code():
return [
'if not isinstance(value, self.ty):',
' raise TypeError("Expected %s"%self.ty)'
]

class Positive(Descriptor):
@staticmethod
def set_code(self):
return [
'if value < 0:',
' raise ValueError("Expected >= 0")'
]
Copyright (C) 2013, http://www.dabeaz.com 152
 Reformulation
class Sized(Descriptor):
def __init__(self, *args, maxlen, **kwargs):
! self.maxlen = maxlen
super().__init__(*args, **kwargs)

@staticmethod
def set_code():
! return [
'if len(value) > self.maxlen:',
' raise ValueError("Too big")'
]

Copyright (C) 2013, http://www.dabeaz.com 153

Reformulation
import re
class RegexPattern(Descriptor):
def __init__(self, *args, pat, **kwargs):
self.pat = re.compile(pat)
super().__init__(*args, **kwargs)

@staticmethod
def set_code():
return [
'if not self.pat.match(value):',
' raise ValueError("Invalid string")'
]

Copyright (C) 2013, http://www.dabeaz.com 154

 Generating a Setter
def _make_setter(dcls):
code = 'def __set__(self, instance, value):\n'
for d in dcls.__mro__:
if 'set_code' in d.__dict__:
for line in d.set_code():
code += ' ' + line + '\n'
return code

Takes a descriptor class as input

Walks its MRO and collects output of set_code()
Concatenate to make a __set__() method
Copyright (C) 2013, http://www.dabeaz.com 155

Example Setters
>>> print(_make_setter(Descriptor))
def __set__(self, instance, value):
instance.__dict__[self.name] = value

>>> print(_make_setter(PosInteger))
def __set__(self, instance, value):
if not isinstance(value, self.ty):
raise TypeError("Expected %s" % self.ty)
if value < 0:
raise ValueError("Expected >= 0")
instance.__dict__[self.name] = value

>>>

Copyright (C) 2013, http://www.dabeaz.com 156

 Descriptor Metaclass
class DescriptorMeta(type):
def __init__(self, clsname, bases, clsdict):
if '__set__' not in clsdict:
code = _make_setter(self)
exec(code, globals(), clsdict)
setattr(self, '__set__',
clsdict['__set__'])
else:
raise TypeError('Define set_code()')

class Descriptor(metaclass=DescriptorMeta):
...

For each Descriptor class, create setter code

exec() and drop result onto created class
Copyright (C) 2013, http://www.dabeaz.com 157

Just to be Clear
class Stock(Structure):
name = SizedRegexString(...)
shares = PosInteger()
price = PosFloat()

User has no idea about this code generation

They're just using the same code as before
It's an implementation detail of descriptors

Copyright (C) 2013, http://www.dabeaz.com 158

 New Performance
Instance creation Simple Meta Exec
s = Stock('ACME',50,91.1) 1.07s 91.8s 7.19s

Attribute lookup
(86x) (6.7x)

s.price 0.08s 0.08s 0.08s

Attribute assignment
s.price = 10.0 0.11s 3.40s 1.11s
(31x) (10x)
Attribute assignment
s.name = 'ACME' 0.14s 8.14s 2.95s
(58x) (21x)
Copyright (C) 2013, http://www.dabeaz.com 159

The Horror! The Horror!

@alex_gaynor

Copyright (C) 2013, http://www.dabeaz.com 160

 Remaining Problem
Convincing a manager about all of this
class Stock(Structure):
name = SizedRegexString(maxlen=8, pat='[A-Z]+$')
shares = PosInteger()
price = PosFloat()

class Point(Structure):
x = Integer()
y = Integer()

class Address(Structure):
hostname = String()
port = Integer()

Copyright (C) 2013, http://www.dabeaz.com 161

Solution: XML
<structures>
<structure name="Stock">
<field type="SizedRegexString" maxlen="8"
pat="'[A-Z]+$'">name</field>
<field type="PosInteger">shares</field>
<field type="PosFloat">price</field>
</structure>
<structure name="Point">
<field type="Integer">x</field>
<field type="Integer">y</field>
</structure>
<structure name="Address">
<field type="String">hostname</field>
<field type="Integer">port</field>
</structure>
</structures>
Copyright (C) 2013, http://www.dabeaz.com 162
 Solution: XML
<structures>
<structure name="Stock">
<field type="SizedRegexString" maxlen="8"
pat="'[A-Z]+$'">name</field>
<field type="PosInteger">shares</field>
<field type="PosFloat">price</field>
+5 extra credit
</structure>
Regex
<structure + XML
name="Point">
<field type="Integer">x</field>
<field type="Integer">y</field>
</structure>
<structure name="Address">
<field type="String">hostname</field>
<field type="Integer">port</field>
</structure>
</structures>
Copyright (C) 2013, http://www.dabeaz.com 163

XML to Classes
XML Parsing
from xml.etree.ElementTree import parse

def _xml_to_code(filename):
doc = parse(filename)
code = 'import typestruct as _ts\n'
for st in doc.findall('structure'):
code += _xml_struct_code(st)
return code

Continued...
Copyright (C) 2013, http://www.dabeaz.com 164
 XML to Classes
def _xml_struct_code(st):
stname = st.get('name')
code = 'class %s(_ts.Structure):\n' % stname
for field in st.findall('field'):
name = field.text.strip()
dtype = '_ts.' + field.get('type')
kwargs = ', '.join('%s=%s' % (key, val)
for key, val in field.items()
if key != 'type')
code += ' %s = %s(%s)\n' % \
(name, dtype, kwargs)
return code

Copyright (C) 2013, http://www.dabeaz.com 165

Example
>>> code = _xml_to_code('data.xml')
>>> print(code)
import typestruct as _ts
class Stock(_ts.Structure):
name = _ts.SizedRegexString(maxlen=8, pat='[A-Z]+$')
shares = _ts.PosInteger()
price = _ts.PosFloat()
class Point(_ts.Structure):
x = _ts.Integer()
y = _ts.Integer()
class Address(_ts.Structure):
hostname = _ts.String()
port = _ts.Integer()

>>>

Copyright (C) 2013, http://www.dabeaz.com 166

 $$!!@!&!**!!!

Now WHAT!?!?
Allow structure .xml files to be imported
Using the import statement
Yes!

Copyright (C) 2013, http://www.dabeaz.com 167

Import Hooks
sys.meta_path
>>> import sys
>>> sys.meta_path
[<class '_frozen_importlib.BuiltinImporter'>,
<class '_frozen_importlib.FrozenImporter'>,
<class '_frozen_importlib.PathFinder'>]
>>>

A collection of importer/finder instances

Copyright (C) 2013, http://www.dabeaz.com 168

 An Experiment
class MyImporter:
def find_module(self, fullname, path=None):
print('*** Looking for', fullname)
return None

>>> sys.meta_path.append(MyImporter())
>>> import foo
*** Looking for foo
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'foo'
>>>

Yes, you've plugged into the import statement

Copyright (C) 2013, http://www.dabeaz.com 169

Structure Importer
class StructImporter:
def __init__(self, path):
self._path = path

def find_module(self, fullname, path=None):

name = fullname.rpartition('.')[-1]
if path is None:
path = self._path
for dn in path:
filename = os.path.join(dn, name+'.xml')
if os.path.exists(filename):
return StructXmlLoader(filename)
return None

Copyright (C) 2013, http://www.dabeaz.com 170

 Structure Importer
class StructImporter:
def __init__(self, path):
self._path = path

def find_module(self, fullname, path=None):

name = fullname.rpartition('.')[-1]
if path is None:
Fully
pathqualified
= self._path Package path
module
for dn in name
path: (if any)
filename = os.path.join(dn, name+'.xml')
if os.path.exists(filename):
return StructXmlLoader(filename)
return None

Copyright (C) 2013, http://www.dabeaz.com 171

Structure Importer
class StructImporter:
def __init__(self, path):
self._path = path

Walk path=None):
def find_module(self, fullname, path, check for
existence of .xml file
name = fullname.rpartition('.')[-1]
if path is None:
path = self._path
and return a loader
for dn in path:
filename = os.path.join(dn, name+'.xml')
if os.path.exists(filename):
return StructXmlLoader(filename)
return None

Copyright (C) 2013, http://www.dabeaz.com 172

 XML Module Loader
import imp
class StructXMLLoader:
def __init__(self, filename):
self._filename = filename

def load_module(self, fullname):

mod = sys.modules.setdefault(fullname,
imp.new_module(fullname))
mod.__file__ = self._filename
mod.__loader__ = self
code = _xml_to_code(self._filename)
exec(code, mod.__dict__, mod.__dict__)
return mod

Copyright (C) 2013, http://www.dabeaz.com 173

XML Module Loader

import imp
class StructXMLLoader:
Create
def __init__(self, filename):
a new module
and put
self._filename = filename in sys.modules
def load_module(self, fullname):
mod = sys.modules.setdefault(fullname,
imp.new_module(fullname))
mod.__file__ = self._filename
mod.__loader__ = self
code = _xml_to_code(self._filename)
exec(code, mod.__dict__, mod.__dict__)
return mod

Copyright (C) 2013, http://www.dabeaz.com 174

 XML Module Loader
import imp
class StructXMLLoader:
def __init__(self, filename):
self._filename = filename

def load_module(self, fullname):

Convertmod
XML = to code and
sys.modules.setdefault(fullname,
exec() resulting source
imp.new_module(fullname))
mod.__file__ = self._filename
mod.__loader__ = self
code = _xml_to_code(self._filename)
exec(code, mod.__dict__, mod.__dict__)
return mod

Copyright (C) 2013, http://www.dabeaz.com 175

Installation and Use

Add to sys.meta_path
def install_importer(path=sys.path):
sys.meta_path.append(StructImporter(path))

install_importer()

From this point, structure .xml files will import

>>> import datadefs
>>> s = datadefs.Stock('ACME', 50, 91.1)
>>> s.name
'ACME'
>>> datadefs
<module 'datadefs' from './datadefs.xml'>
>>>

Copyright (C) 2013, http://www.dabeaz.com 176

 Look at the Source
>>> datadefs
<module 'datadefs' from './datadefs.xml'>
>>>
>>> import inspect
>>> print(inspect.getsource(datadefs))
<structures>
<structure name="Stock">
<field type="SizedRegexString" maxlen="8" pat="'[A-Z]+
$'">name</field>
<field type="PosInteger">shares</field>
<field type="PosFloat">price</field>
</structure>
...

Copyright (C) 2013, http://www.dabeaz.com 177

Final Thoughts

(probably best to start packing up)

Copyright (C) 2013, http://www.dabeaz.com 178

 Extreme Power
Think about all of the neat things we did
class Stock(Structure):
name = SizedRegexString(maxlen=8, pat='[A-Z]+$')
shares = PosInteger()
price = PosFloat()

Descriptors as building blocks

Hiding of annoying details (signatures, etc.)
Dynamic code generation
Even customizing import
Copyright (C) 2013, http://www.dabeaz.com 179

Hack or by Design?
Python 3 is designed to do this sort of stuff
More advanced metaclasses (e.g., __prepare__)
Signatures
Import hooks
Keyword-only args
Observe: I didn't do any mind-twisting "hacks" to
work around a language limitation.

Copyright (C) 2013, http://www.dabeaz.com 180

 Python 3 FTW!
Python 3 makes a lot of little things easier
Example : Python 2 keyword-only args
def __init__(self, *args, **kwargs):
self.maxlen = kwargs.pop('maxlen')
...
In Python 3
def __init__(self, *args, maxlen, **kwargs):
self.maxlen = maxlen
...

There are a lot of little things like this

Copyright (C) 2013, http://www.dabeaz.com 181

Just the Start

We've only scratched the surface
Function annotations
def add(x:int, y:int) -> int:
return x + y

Non-local variables
def outer():
x = 0
def inner():
nonlocal x
x = newvalue
...

Copyright (C) 2013, http://www.dabeaz.com 182

 Just the Start
Context managers
with m:
...

Frame-hacks
import sys
f = sys._getframe(1)

Parsing/AST-manipulation
import ast

Copyright (C) 2013, http://www.dabeaz.com 183

You Can, But Should You?

Metaprogramming is not for "normal" coding

Frameworks/libraries are a different story
If using a framework, you may be using this
features without knowing it
You can do a lot of cool stuff
OTOH: Keeping it simple is not a bad strategy
Copyright (C) 2013, http://www.dabeaz.com 184
 That is All!

Thanks for listening

Hope you learned a few new things
Buy the "Python Cookbook, 3rd Ed." (O'Reilly)
Twitter: @dabeaz

Copyright (C) 2013, http://www.dabeaz.com 185