Opensource.

com

30 hidden gems in Python 3

Three cool features from each of the

first ten versions of Python 3
OPENSOURCE.COM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ABOUT OPENSOURCE.COM

What is Opensource.com?

OPENSOURCE.COM publishes stories about creating,

adopting, and sharing open source
solutions. Visit Opensource.com to learn more about how the open source
way is improving technologies, education, business, government, health, law,
entertainment, humanitarian efforts, and more.

Submit a story idea: opensource.com/story

Email us: open@opensource.com

2 30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM

 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ABOUT THE AUTHOR

MOSHE ZADKA

MOSHE ZADKA has been involved in the Linux community since

1998, helping in Linux "installation parties". He
has been programming Python since 1999, and has
contributed to the core Python interpreter. Moshe
has been a DevOps/SRE since before those
terms existed, caring deeply about software
reliability, build reproducibility and other
such things. He has worked in companies
as small as three people and as big as
tens of thousands -- usually some place
around where software meets system
administration.
Follow me at @moshezadka

30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM 3

CONTENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

CHAPTERS

3 features that debuted in Python 3.0 you should use now 6

3 features released in Python 3.1 you should use in 2021 8

3 Python 3.2 features that are still relevant today 9

What Python 3.3 did to improve exception handling 11

in your code

Looking back at what Python 3.4 did for enum 13

Convenient matrices and other improvements Python 3.5 15

brought us

Are you using this magic method for filesystems from 17

Python 3.6?

Slice infinite generators with this Python 3.7 feature 19

Make your API better with this positional trick from 20

Python 3.8

How Python 3.9 fixed decorators and improved dictionaries 21

4 30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . INTRODUCTION

Introduction
THE RELEASE OF PYTHON 3, a backwards incompatible
version of Python, was a
news-making event. As Python rose in popularity, every version since has also been an event.

From async to the so-called “walrus operator” (the := looks like walrus eyes and teeth),
Pythonistas have been atwitter before, after, and during every single major release.

But what about the features that didn’t make the news?

In each one of those releases, there are hidden gems. Small improvements to the standard
library. A little improved ergonomics in the interpreter. Maybe even a new operator, one that
is not important to be on the front page.

Python 3 has been out since 2008, and it has had ten minor releases between 3.0 and 3.9.
Each of those releases packed more features than most people know. Some of those are
still little known.

The major challenge is not to find three cool things first released in a new version of Py-
thon. It’s not even to find three cool things that few people use. The challenge is how to pick
just three from all the delightful options.

Enjoy these curated picks. Here are 30 features, three from each of the first ten versions of
Python 3, that you might want to start using.

30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM 5

3 FEATURES THAT DEBUTED IN PYTHON 3.0 YOU SHOULD USE NOW . . . . . . . . . . . . . . . . . . . . . . . . . .

3 features that debuted in

Python 3.0 you should use now
Explore some of the underutilized but still useful Python features.

THIS IS THE FIRST in a series of articles

about features that
first appeared in a version of Python 3.x. Python 3.0 was
While it is possible to call this function with keyword argu-
ments, it is not obvious that this is the best way. Instead, you
can mark these arguments as keyword-only:
first released in 2008, and even though it has been out for a
while, many of the features it introduced are underused and def show_arguments(
base, *, extended=None, improved=None,
pretty cool. Here are three you should know about. augmented=None):
print("base is", base)
Keyword-only arguments if extended is not None:
Python 3.0 first introduced the idea of keyword-only argu- print("extended is", extended)
ments. Before this, it was impossible to specify an API where if improved is not None:
some arguments could be passed in only via keywords. This print("improved is", improved)
is useful in functions with many arguments, some of which if augmented is not None:
might be optional. print("augmented is", augmented)
Consider a contrived example:
Now, you can’t pass in the extra arguments with positional
def show_arguments(
base, extended=None, improved=None, arguments:
augmented=None):
print("base is", base) show_arguments("hello", "extra")
if extended is not None: -----------------------------------------------------------
print("extended is", extended)
if improved is not None: TypeError Traceback (most recent call last)
print("improved is", improved)
if augmented is not None: <ipython-input-7-6000400c4441> in <module>
print("augmented is", augmented) ----> 1 show_arguments("hello", "extra")

When reading code that calls this function, it is sometimes

hard to understand what is happening: TypeError: 
show_arguments() takes 1 positional argument but 2
were given
show_arguments("hello", "extra")
base is hello Valid calls to the function are much easier to predict:
extended is extra
show_arguments("hello", None, "extra") show_arguments("hello", improved="extra")
base is hello base is hello
improved is extra improved is extra

6 30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM

. . . . . . . . . . . . . . . . . . . . . . . . . . 3 FEATURES THAT DEBUTED IN PYTHON 3.0 YOU SHOULD USE NOW

nonlocal Extended destructuring

Sometimes, functional programming folks judge a language Imagine you have a CSV file where each row consists of
by how easy is it to write an accumulator. An accumulator is several elements:
a function that, when called, returns the sum of all arguments
sent to it so far. • T he first element is a year
The standard answer in Python before 3.0 was: • The second element is a month
• The other elements are the total articles published that
class _Accumulator: month, one entry for each day
def __init__(self):
self._so_far = 0 Note that the last element is total articles, not articles pub-
def __call__(self, arg): lished per day. For example, a row can begin with:
self._so_far += arg
return self._so_far 2021,1,5,8,10

def make_accumulator(): This means that in January 2021, five articles were published
return _Accumulator() on the first day. On the second day, three more articles were
published, bringing the total to 8. On the third day, two more
While admittedly somewhat verbose, this does work: articles were published.
Months can have 28, 30, or 31 days. How hard is it to ex-
acc = make_accumulator() tract the month, day, and total articles?
print("1", acc(1)) In versions of Python before 3.0, you might write some-
print("5", acc(5)) thing like:
print("3", acc(3))
year, month, total = row[0], row[1], row[-1]
The output for this would be:
This is correct, but it obscures the format. With extended
1 1 destructuring, the same can be expressed this way:
5 6
3 9 year, month, *rest, total = row

In Python 3.x, nonlocal can achieve the same behavior with This means that if the format ever changes to prefix a de-
significantly less code. scription, you can change the code to:

def make_accumulator(): _, year, month, *rest, total = row

so_far = 0
def accumulate(arg): Without needing to add 1 to each of the indices.
nonlocal so_far
so_far += arg What’s next?
return so_far Python 3.0 and its later versions have been out for more
return accumulate than 12 years, but some of its features are underutilized.
In the next article in this series, I’ll look at three more of
While accumulators are contrived examples, the ability to them.
use the nonlocal keyword to have inner functions with state
is a powerful tool.

30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM 7

3 FEATURES RELEASED IN PYTHON 3.1 YOU SHOULD USE IN 2021 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3 features released in Python 3.1

you should use in 2021
Explore some of the underutilized but still useful Python features.

THIS IS THE SECOND in a series of

articles about
features that first appeared in a version of Python 3.x.
I'll try and tell you what I mean.

In Summer, when the days are long,

Python 3.1 was first released in 2009, and even though Perhaps you'll understand the song.
it has been out for a long time, many of the features it
introduced are underused and pretty cool. Here are three In Autumn, when the leaves are brown,
of them. Take pen and ink, and write it down.
"""
Thousands formatting import collections
When formatting large numbers, it is common to place com-
mas every three digits to make the number more readable collections.Counter(hd_song.lower().replace(' ', ''))
(e.g., 1,048,576 is easier to read than 1048576). Since .most_common(5)
Python 3.1, this can be done directly when using string for- [('e', 29), ('n', 27), ('i', 18), ('t', 18), ('r', 15)]
matting functions:
Executing packages
"2 to the 20th power is {:,d}".format(2**20) Python allows the -m flag to execute modules from the
'2 to the 20th power is 1,048,576' command line. Even some standard-library modules do
something useful when they’re executed; for example, py-
The ,d format specifier indicates that the number must be thon -m cgi is a CGI script that debugs the web server’s
formatted with commas. CGI configuration.
However, until Python 3.1, it was impossible to execute
Counter class packages like this. Starting with Python 3.1, python -m
The collections.Counter class, part of the standard library package will execute the __main__ module in the package.
module collections, is a secret super-weapon in Python. This is a good place to put debug scripts or commands that
It is often first encountered in simple solutions to interview are executed mostly with tools and do not need to be short.
questions in Python, but its value is not limited to that.
For example, find the five most common letters in the first Python 3.0 was released over 11 years ago, but some of
eight lines of Humpty Dumpty’s song [1]: the features that first showed up in this release are cool—and
underused. Add them to your toolkit if you haven’t already.
hd_song = """
In winter, when the fields are white, Links
I sing this song for your delight. [1] http://www2.open.ac.uk/openlearn/poetryprescription/
In Spring, when woods are getting green, humpty-dumptys-recitation.html

8 30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 PYTHON 3.2 FEATURES THAT ARE STILL RELEVANT TODAY

3 Python 3.2 features

that are still relevant today
Explore some of the underutilized but still useful Python features.

THIS THE THIRD article in a series about

features that first ap-
peared in a version of Python 3.x. Some of those Python
negate = subparsers.add_parser("negate")
negate.set_defaults(subcommand="negate")
negate.add_argument("number", type=float)
versions have been out for a while. For example, Python 3.2 multiply = subparsers.add_parser("multiply")
was first released in 2011, yet some of the cool and useful multiply.set_defaults(subcommand="multiply")
features introduced in it are still underused. Here are three multiply.add_argument("number1", type=float)
of them. multiply.add_argument("number2", type=float)

argparse subcommands One of my favorite argparse features is that, because it sep-

The argparse module first appeared in Python 3.2. There arates parsing from running, testing the parsing logic is par-
are many third-party modules for command-line parsing. But ticularly pleasant.
the built-in argparse module is more powerful than many
give it credit for.. parser.parse_args(["negate", "5"])
Documenting all the ins and outs of argparse would take Namespace(number=5.0, subcommand='negate')
its own article series. For a small taste, here is an example parser.parse_args(["multiply", "2", "3"])
of how you can do subcommands with argparse. Namespace(number1=2.0, number2=3.0, subcommand='multiply')
Imagine a command with two subcommands: negate,
which takes one argument, and multiply which takes two: contextlib.contextmanager
Contexts are a powerful tool in Python. While many use
$ computebot negate 5 them, writing a new context often seems like a dark art. With
-5 the contextmanager decorator, all you need is a one-shot
$ computebot multiply 2 3 generator.
6 Writing a context that prints out the time it took to do some-
import argparse thing is as simple as:

parser = argparse.ArgumentParser() import contextlib, timeit

subparsers = parser.add_subparsers()
@contextlib.contextmanager
The add_subparsers() methods creates an object that you def timer():
can add subcommands to. The only trick to remember is that before = timeit.default_timer()
you need to add what subcommand was called through a try:
set_defaults(): yield

30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM 9

3 PYTHON 3.2 FEATURES THAT ARE STILL RELEVANT TODAY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

finally: It turns out that when you calculate how many ways you can
after = timeit.default_timer() do something like making change from 50 cents, you use
print("took", after - before) the same coins repeatedly. You can use lru_cache to avoid
recalculating this over and over.
And you can use it with just:
import functools
import time
def change_for_a_dollar():
with timer(): @functools.lru_cache
time.sleep(10.5) def change_for(amount, coins):
took 10.511025413870811 if amount == 0:
return 1
functools.lru_cache if amount < 0 or len(coins) == 0:
Sometimes the caching results from a function in memory return 0
make sense. For example, imagine the classical problem: some_coin = next(iter(coins))
“How many ways can you make change for a dollar with return (
quarters, dimes, nickels, and cents?” change_for(amount, coins - set([some_coin]))
The code for this can be deceptively simple: +
change_for(amount - some_coin, coins)
def change_for_a_dollar(): )
def change_for(amount, coins): return change_for(100, frozenset([25, 10, 5, 1]))
if amount == 0: with timer():
return 1 change_for_a_dollar()
if amount < 0 or len(coins) == 0: took 0.004180959425866604
return 0
some_coin = next(iter(coins)) A three-fold improvement for the cost of one line. Not bad.
return (
change_for(amount, coins - set([some_coin])) Welcome to 2011
+ Although Python 3.2 was released 10 years ago, many of
change_for(amount - some_coin, coins) its features are still cool—and underused. Add them to your
) toolkit if you haven’t already.
return change_for(100, frozenset([25, 10, 5, 1]))

On my computer, this takes around 13ms:

with timer():
change_for_a_dollar()
took 0.013737603090703487

10 30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM

. . . . . . . . . . . . . . . . . . . . WHAT PYTHON 3.3 DID TO IMPROVE EXCEPTION HANDLING IN YOUR CODE

What Python 3.3 did to improve

exception handling in your code
Explore exception handling and other underutilized but still useful Python features.

THIS IS THE FOURTH in a series of ar-

ticles about fea-
tures that first appeared in a version of Python 3.x. Python
in Python. While Parasol has open sourced some of its
code, some of it is too proprietary or specialized for open
source.
3.3 was first released in 2012, and even though it has been The company uses an internal DevPI [3] server to man-
out for a long time, many of the features it introduced are age the internal packages. It does not make sense for every
underused and pretty cool. Here are three of them. Python programmer at Parasol to find an unused name on
PyPI, so all the internal packages are called parasol.<busi-
yield from ness division>.<project>. Observing best practices, the
The yield keyword made Python much more powerful. Pre- developers want the package names to reflect that naming
dictably, everyone started using it to create a whole ecosys- system.
tem of iterators. The itertools [1] module and the more-iter- This is important! If the package parasol.accounting.
tools [2] PyPI package are just two examples.. numeric_tricks installs a top-level module called numeric_
Sometimes, a new generator will want to use an existing tricks, this means nobody who depends on this package
generator. As a simple (if somewhat contrived) example, will be able to use a PyPI package that is called numeric_
imagine you want to enumerate all pairs of natural numbers. tricks, no matter how nifty it is.
One way to do it is to generate all pairs in the order of sum However, this leaves the developers with a dilemma:
of pair, first item of pair. Implementing this with yield Which package owns the parasol/__init__.py file? The
from is natural. best solution, starting in Python 3.3, is to make parasol, and
The yield from <x> keyword is short for: probably parasol.accounting, to be namespace packages
[4], which don’t have the __init__.py file.
for item in x:
yield item Suppressing exception context
import itertools Sometimes, an exception in the middle of a recovery from
an exception is a problem, and having the context to trace
def pairs(): it is useful. However, sometimes it is not: the exception
for n in itertools.count(): has been handled, and the new situation is a different er-
yield from ((i, n-i) for i in range(n+1)) ror condition.
list(itertools.islice(pairs(), 6)) For example, imagine that after failing to look up a key in
[(0, 0), (0, 1), (1, 0), (0, 2), (1, 1), (2, 0)] a dictionary, you want to fail with a ValueError() if it cannot
be analyzed:
Implicit namespace packages
Imagine a fictional company called Parasol that makes import time
a bunch of stuff. Much of its internal software is written

30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM 11

WHAT PYTHON 3.3 DID TO IMPROVE EXCEPTION HANDLING IN YOUR CODE . . . . . . . . . . . . . . . . . . . .

def expensive_analysis(data): 10 cached[data] = analyzed

time.sleep(10) 11 return analyzed[-1]
if data[0:1] == ">":
return data[1:]
return None ValueError: ('invalid data', 'stuff')

This function takes a long time, so when you use it, you want If you use raise ... from None, you can get much more
to cache the results: readable tracebacks:

cache = {} def last_letter_analyzed(data):

try:
def last_letter_analyzed(data): analyzed = cache[data]
try: except KeyError:
analyzed = cache[data] analyzed = expensive_analysis(data)
except KeyError: if analyzed is None:
analyzed = expensive_analysis(data) raise ValueError("invalid data", data) from None
if analyzed is None: cached[data] = analyzed
raise ValueError("invalid data", data) return analyzed[-1]
cached[data] = analyzed last_letter_analyzed("stuff")
return analyzed[-1] -------------------------------------------------------------
Unfortunately, when there is a cache miss, the traceback
looks ugly: ValueError Traceback (most recent call last)

last_letter_analyzed("stuff") <ipython-input-21-40dab921f9a9> in <module>

------------------------------------------------------------- ----> 1 last_letter_analyzed("stuff")

KeyError Traceback (most recent call last)

<ipython-input-20-5691e33edfbc> in last_letter_analyzed(data)
<ipython-input-16-a525ae35267b> in last_letter_analyzed(data) 5 analyzed = expensive_analysis(data)
4 try: 6 if analyzed is None:
----> 5 analyzed = cache[data] ----> 7 raise ValueError("invalid data", data)
6 except KeyError: from None
8 cached[data] = analyzed
9 return analyzed[-1]
KeyError: 'stuff'

During handling of the above exception, another exception ValueError: ('invalid data', 'stuff')
occurs:
Welcome to 2012
ValueError Traceback (most recent call last) Although Python 3.3 was released almost a decade ago,
many of its features are still cool—and underused. Add them
<ipython-input-17-40dab921f9a9> in <module> to your toolkit if you haven’t already.
----> 1 last_letter_analyzed("stuff")

Links
<ipython-input-16-a525ae35267b> in last_letter_analyzed(data) [1] https://docs.python.org/3/library/itertools.html
7 analyzed = expensive_analysis(data) [2] https://more-itertools.readthedocs.io/en/stable/
8 if analyzed is None: [3] https://opensource.com/article/18/7/setting-devpi
----> 9 raise ValueError("invalid data", data) [4] https://www.python.org/dev/peps/pep-0420/

12 30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LOOKING BACK AT WHAT PYTHON 3.4 DID FOR ENUM

Looking back at what Python 3.4

did for enum
Plus explore some of the underutilized but still useful Python features.

THIS IS THE FIFTH in a series of articles

about features that
first appeared in a version of Python 3.x. Python 3.4 was first
@enum.unique
class Identity(enum.Enum):
RANDOM = enum.auto()
released in 2014, and even though it has been out for a long TRUE = enum.auto()
time, many of the features it introduced are underused and FALSE = enum.auto()
pretty cool. Here are three of them.

enum @enum.unique
One of my favorite logic puzzles is the self-descriptive Hard- class Language(enum.Enum):
est Logic Puzzle Ever [1]. Among other things, it talks about ja = enum.auto()
three gods who are called A, B, and C. Their identities are True, da = enum.auto()
False, and Random, in some order. You can ask them ques-
tions, but they only answer in the god language, where “da” and One advantage of enums is that in debugging logs or excep-
“ja” mean “yes” and “no,” but you do not know which is which. tions, the enum is rendered helpfully:
If you decide to use Python to solve the puzzle, how would
you represent the gods’ names and identities and the words name = Name.A
in the god language? The traditional answer has been to use identity = Identity.RANDOM
strings. However, strings can be misspelled with disastrous answer = Language.da
consequences. print(
"I suspect", name, "is", identity, "because they
If, in a critical part of your solution, you compare to the answered", answer)
string jaa instead of ja, you will have an incorrect solution. I suspect Name.A is Identity.RANDOM because they answered
While the puzzle does not specify what the stakes are, that’s Language.da
probably best avoided.
The enum module gives you the ability to define these functools.singledispatch
things in a debuggable yet safe manner: While developing the “infrastructure” layer of a game, you
want to deal with various game objects generically but still
import enum allow the objects to customize actions. To make the example
easier to explain, assume it’s a text-based game. When you
@enum.unique use an object, most of the time, it will just print You are using
class Name(enum.Enum): <x>. But using a special sword might require a random roll,
A = enum.auto() and it will fail otherwise.
B = enum.auto() When you acquire an object, it is usually added to the
C = enum.auto() inventory. However, a particularly heavy rock will smash a

30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM 13

LOOKING BACK AT WHAT PYTHON 3.4 DID FOR ENUM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

random object; if that happens, the inventory will lose that else:
object. print("You fail")
One way to approach this is to have methods use and ac-
quire on objects. More and more of these methods will be deploy(sword)
added as the game’s complexity increases, making game You try to use sword
objects unwieldy to write. You succeed
Instead, functools.singledispatch allows you to add You have ['sword', 'torch']
methods retroactively—in a safe and namespace-respecting import random
manner.
You can define classes with no behavior: @acquire.register(Rock)
def acquire_rock(rock, inventory):
class Torch: to_remove = random.choice(list(inventory))
name="torch" inventory.remove(to_remove)
inventory.add(rock)
class Sword:
name="sword" deploy(Rock())
You use rock
class Rock: You have ['sword', 'rock']
name="rock"
import functools The rock might have crushed the torch, but your code is
much easier to read.
@functools.singledispatch
def use(x): pathlib
print("You use", x.name) The interface to file paths in Python has been “smart-string
manipulation” since the beginning of time. Now, with path-
@functools.singledispatch lib, Python has an object-oriented way to manipulate paths:
def acquire(x, inventory):
inventory.add(x) import pathlib
gitconfig = pathlib.Path.home() / ".gitconfig"
For the torch, those generic implementations are enough: text = gitconfig.read_text().splitlines()

inventory = set() Admittedly, using / as an operator to generate path names

is a little cutesy, but it ends up being nice in practice.
def deploy(thing): Methods like .read_text() allow you to get text out of
acquire(thing, inventory) small files without needing to open and close file handles
use(thing) manually.
print("You have", [item.name for item in inventory]) This lets you concentrate on the important stuff:

deploy(Torch()) for line in text:

You use torch if not line.strip().startswith("name"):
You have ['torch'] continue
print(line.split("=")[1])
However, the sword and the rock need some specialized Moshe Zadka
functionality:
Welcome to 2014
import random Python 3.4 was released about seven years ago, but some of
the features that first showed up in this release are cool—and
@use.register(Sword) underused. Add them to your toolkit if you haven’t already.
def use_sword(sword):
print("You try to use", sword.name) Links
if random.random() < 0.9: [1] https://en.wikipedia.org/wiki/The_Hardest_Logic_Puzzle_
print("You succeed") Ever

14 30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM

. . . . . . . . . . . . . . . CONVENIENT MATRICES AND OTHER IMPROVEMENTS PYTHON 3.5 BROUGHT US

Convenient matrices and other

improvements Python 3.5 brought us
Explore some of the underutilized but still useful Python features.

THIS IS THE SIXTH in a series of articles

about features that
first appeared in a version of Python 3.x. Python 3.5 was first
This is one advantage of the new operator: especially in
complex formulas, the code looks more like the underlying
math:
released in 2015, and even though it has been out for a long
time, many of the features it introduced are underused and almost_zero = ((eighth_turn @ eighth_turn) - numpy.array([[0, 1],
pretty cool. Here are three of them. [-1, 0]]))**2
round(numpy.sum(almost_zero) ** 0.5, 10)
The @ operator 0.0
The @ operator is unique in Python in that there are no ob-
jects in the standard library that implement it! It was added Multiple keyword dictionaries in arguments
for use in mathematical packages that have matrices. Python 3.5 made it possible to call functions with multiple
Matrices have two concepts of multiplication; point-wise keyword-argument dictionaries. This means multiple sourc-
multiplication is done with the * operator. But matrix com- es of defaults can “co-operate” with clearer code.
position (also considered multiplication) needed its own For example, here is a function with a ridiculous amount of
symbol. It is done using @. keyword arguments:
For example, composing an “eighth-turn” matrix (rotating
the axis by 45 degrees) with itself results in a quarter-turn def show_status(
matrix: *,
the_good=None,
import numpy the_bad=None,
the_ugly=None,
hrt2 = 2**0.5 / 2 fistful=None,
eighth_turn = numpy.array([ dollars=None,
[hrt2, hrt2], more=None
[-hrt2, hrt2] ):
]) if the_good:
eighth_turn @ eighth_turn print("Good", the_good)
array([[ 4.26642159e-17, 1.00000000e+00], if the_bad:
[-1.00000000e+00, -4.26642159e-17]]) print("Bad", the_bad)
if the_ugly:
Floating-point numbers being imprecise, this is harder to see. print("Ugly", the_ugly)
It is easier to check by subtracting the quarter-turn matrix from if fistful:
the result, summing the squares, and taking the square root. print("Fist", fistful)

30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM 15

CONVENIENT MATRICES AND OTHER IMPROVEMENTS PYTHON 3.5 BROUGHT US . . . . . . . . . . . . . . .

if dollars: Bad I have to have respect

print("Dollars", dollars) Ugly Shoot, don't talk
if more: Fist Get three coffins ready
print("More", more) Dollars Remember me?
More It's a small world
When you call this function in the application, some argu-
ments are hardcoded: os.scandir
The os.scandir function is a new way to iterate through di-
defaults = dict( rectories’ contents. It returns a generator that yields rich data
the_good="You dig", about each object. For example, here is a way to print a di-
the_bad="I have to have respect", rectory listing with a trailing / at the end of directories:
the_ugly="Shoot, don't talk",
) for entry in os.scandir(".git"):
print(entry.name + ("/" if entry.is_dir() else ""))
More arguments are read from a configuration file: refs/
HEAD
import json logs/
index
others = json.loads(""" branches/
{ config
"fistful": "Get three coffins ready", objects/
"dollars": "Remember me?", description
"more": "It's a small world" COMMIT_EDITMSG
} info/
""") hooks/

You can call the function from both sources together without Welcome to 2015
having to construct an intermediate dictionary: Python 3.5 was released over six years ago, but some of
the features that first showed up in this release are cool—
show_status(**defaults, **others) and underused. Add them to your toolkit if you haven’t al-
Good You dig ready.

16 30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM

. . . . . . . . . . . . . . . . . ARE YOU USING THIS MAGIC METHOD FOR FILESYSTEMS FROM PYTHON 3.6?

Are you using this magic method

for filesystems from Python 3.6?
Explore os.fspath and two other underutilized but still useful Python features.

THIS IS THE SEVENTH in a series of

articles about
features that first appeared in a version of Python 3.x. Py-
print("Sin of a quarter turn should be 1, go",
round(math.sin(math.tau/4), 2))
Tan of an eighth turn should be 1, got 1.0
thon 3.6 was first released in 2016, and even though it has Cos of an sixth turn should be 1/2, got 0.5
been out for a while, many of the features it introduced are Sin of a quarter turn should be 1, go 1.0
underused and pretty cool. Here are three of them.
os.fspath
Separated numeral constants Starting in Python 3.6, there is a magic method that rep-
Quick, which is bigger, 10000000 or 200000? Would you be resents “convert to a filesystem path.” When given an str or
able to answer correctly while scanning through code? De- bytes, it returns the input.
pending on local conventions, in prose writing, you would For all types of objects, it looks for an __fspath__ method
use 10,000,000 or 10.000.000 for the first number. The trou- and calls it. This allows passing around objects that are “file-
ble is, Python uses commas and periods for other reasons. names with metadata.”
Fortunately, since Python 3.6, you can use underscores to Normal functions like open() or stat will still be able to use
separate digits. This works both directly in code and when them, as long as __fspath__ returns the right thing.
using the int() convertor from strings: For example, here is a function that writes some data into
a file and then checks its size. It also logs the file name to
import math standard output for tracing purposes:
math.log(10_000_000) / math.log(10)
7.0 def write_and_test(filename):
math.log(int("10_000_000")) / math.log(10) print("writing into", filename)
7.0 with open(filename, "w") as fpout:
fpout.write("hello")
Tau is right print("size of", filename, "is", os.path.getsize(filename))
What’s a 45-degree angle expressed in radians? One correct
answer is π/4, but that’s a little hard to remember. It’s much You can call it the way you would expect, with a string for a
easier to remember that a 45-degree angle is an eighth of filename:
a turn. As the Tau Manifesto [1] explains, 2π, called Τ, is a
more natural constant. write_and_test("plain.txt")
In Python 3.6 and later, your math code can use the more writing into plain.txt
intuitive constant: size of plain.txt is 5

print("Tan of an eighth turn should be 1, got", However, it is possible to define a new class that adds infor-
round(math.tan(math.tau/8), 2)) mation to the string representation of filenames. This allows
print("Cos of an sixth turn should be 1/2, got", the logging to be more detailed, without changing the original
round(math.cos(math.tau/6), 2)) function:

30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM 17

ARE YOU USING THIS MAGIC METHOD FOR FILESYSTEMS FROM PYTHON 3.6? . . . . . . . . . . . . . . . . . .

class DocumentedFileName: write_and_test(DocumentedFileName("documented.txt",

def __init__(self, fname, why): "because it's fun"))
self.fname = fname writing into DocumentedFileName(fname='documented.txt',
self.why = why why="because it's fun")
def __fspath__(self): size of DocumentedFileName(fname='documented.txt',
return self.fname why="because it's fun") is 5
def __repr__(self):
return f"DocumentedFileName(fname={self.fname!r}, Welcome to 2016
why={self.why!r})" Python 3.6 was released about five years ago, but some of
the features that first showed up in this release are cool—and
Running the function with a DocumentedFileName instance underused. Add them to your toolkit if you haven’t already.
as input allows the open and os.getsize functions to keep
working while enhancing the logs: Links
[1] https://tauday.com/tau-manifesto

18 30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SLICE INFINITE GENERATORS WITH THIS PYTHON 3.7 FEATURE

Slice infinite generators

with this Python 3.7 feature
Learn more about this and two other underutilized but still useful Python features.

THIS IS THE EIGHTH in a series of ar-

ticles about fea-
tures that first appeared in a version of Python 3.x. Py-
list(itertools.islice(itertools.count(), short_1, short_3))
[1, 2]

thon 3.7 [1] was first released in 2018, and even though functools.singledispatch() annotation registration
it has been out for a few years, many of the features it If you thought singledispatch [2] couldn’t get any cooler,
introduced are underused and pretty cool. Here are three you were wrong. Now it is possible to register based on
of them. annotations:

Postponed evaluation of annotations import attr

In Python 3.7, as long as the right __future__ flags are acti- import math
vated, annotations are not evaluated during runtime: from functools import singledispatch

from __future__ import annotations @attr.s(auto_attribs=True, frozen=True)

class Circle:
def another_brick(wall: List[Brick], brick: Brick) -> Education: radius: float
pass
another_brick.__annotations__ @attr.s(auto_attribs=True, frozen=True)
{'wall': 'List[Brick]', 'brick': 'Brick', 'return': 'Education'} class Square:
side: float
This allows recursive types (classes that refer to themselves)
and other fun things. However, it means that if you want to do @singledispatch
your own type analysis, you need to use ast explictly: def get_area(shape):
raise NotImplementedError("cannot calculate area for unknown
import ast shape", shape)
raw_type = another_brick.__annotations__['wall']
[parsed_type] = ast.parse(raw_type).body @get_area.register
subscript = parsed_type.value def _get_area_square(shape: Square):
f"{subscript.value.id}[{subscript.slice.id}]" return shape.side ** 2
'List[Brick]'
@get_area.register
itertools.islice supports __index__ def _get_area_circle(shape: Circle):
Sequence slices in Python have long accepted all kinds of return math.pi * (shape.radius ** 2)
int-like objects (objects that have __index__()) as valid slice
parts. However, it wasn’t until Python 3.7 that itertools.is- get_area(Circle(1)), get_area(Square(1))
lice, the only way in core Python to slice infinite generators, (3.141592653589793, 1)
gained this support.
For example, now it is possible to slice infinite generators Welcome to 2017
by numpy.short-sized integers: Python 3.7 was released about four years ago, but some of
the features that first showed up in this release are cool—and
import numpy underused. Add them to your toolkit if you haven’t already.
short_1 = numpy.short(1)
short_3 = numpy.short(3) Links
short_1, type(short_1) [1] https://opensource.com/downloads/cheat-sheet-python-
(1, numpy.int16) 37-beginners
import itertools [2] https://opensource.com/article/19/5/python-singledispatch

30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM 19

MAKE YOUR API BETTER WITH THIS POSITIONAL TRICK FROM PYTHON 3.8 . . . . . . . . . . . . . . . . . . . .

Make your API better with this

positional trick from Python 3.8
Explore positional-only parameters and two other underutilized but still useful Python features.

THIS IS THE NINTH in a series of articles

about features that
first appeared in a version of Python 3.x. Python 3.8 was first
def some_func(prefix, /, **kwargs):
print(prefix, kwargs)
some_func("a_prefix", prefix="prefix keyword value")
released in 2019, and two years later, many of its cool new a_prefix {'prefix': 'prefix keyword value'}
features remain underused. Here are three of them.
Note that, confusingly, the value of the variable prefix is dis-
importlib.metadata tinct from the value of kwargs["prefix"]. As in many places,
Entry points [1] are used for various things in Python packag- take care to use this feature carefully.
es. The most familiar are console_scripts [2] entrypoints, but
many plugin systems in Python use them. Self-debugging expressions
Until Python 3.8, the best way to read entry points from The print() statement (and its equivalent in other languages)
Python was to use pkg_resources, a somewhat clunky mod- has been a favorite for quickly debugging output for over
ule that is part of setuptools. 50 years.
The new importlib.metadata is a built-in module that al- But we have made much progress in print statements like:
lows access to the same thing:
special_number = 5
from importlib import metadata as importlib_metadata print("special_number = %s" % special_number)
special_number = 5
distribution = importlib_metadata.distribution("numpy")
distribution.entry_points Yet self-documenting f-strings make it even easier to be clear:
[EntryPoint(name='f2py', value='numpy.f2py.f2py2e:main',
group='console_scripts'), print(f"{special_number=}")
EntryPoint(name='f2py3', value='numpy.f2py.f2py2e:main', special_number=5
group='console_scripts'),
EntryPoint(name='f2py3.9', value='numpy.f2py.f2py2e:main', Adding an = to the end of an f-string interpolated section
group='console_scripts')] keeps the literal part while adding the value.
This is even more useful when more complicated expres-
Entry points are not the only thing importlib.metadata permits sions are inside the section:
access to. For debugging, reporting, or (in extreme circum-
stances) triggering compatibility modes, you can also check values = {}
the version of dependencies—at runtime! print(f"{values.get('something', 'default')=}")
values.get('something', 'default')='default'
f"{distribution.metadata['name']}=={distribution.version}"
'numpy==1.20.1' Welcome to 2019
Python 3.8 was released about two years ago, and some of
Positional-only parameters its new features are cool—and underused. Add them to your
After the wild success of keywords-only arguments at com- toolkit if you haven’t already.
municating API authors’ intentions, another gap was filled:
positional-only arguments. Links
Especially for functions that allow arbitrary keywords (for [1] https://packaging.python.org/specifications/entry-points/
example, to generate data structures), this means there are [2] https://python-packaging.readthedocs.io/en/latest/
fewer constraints on allowed argument names: command-line-scripts.html

20 30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM

. . . . . . . . . . . . . . . . . . . . . . . . . HOW PYTHON 3.9 FIXED DECORATORS AND IMPROVED DICTIONARIES

How Python 3.9 fixed decorators

and improved dictionaries
Explore some of the useful features of the recent version of Python.

Arbitrary decorator expressions

THIS IS THE TENTH in a series of ar-
ticles about fea-
tures that first appeared in a version of Python 3.x. Some
Previously, the rules about which expressions are allowed in
a decorator were underdocumented and hard to understand.
of these versions have been out for a while. Python 3.9 was For example, while:
first released in 2020 with cool new features that are still
underused. Here are three of them. @item.thing
def foo():
Adding dictionaries pass
Say you have a dictionary with “defaults,” and you want
to update it with parameters. Before Python 3.9, the best is valid, and:
option was to copy the defaults dictionary and then use the
.update() method. @item.thing()
Python 3.9 introduced the union operator to dictionaries: def foo():
pass
defaults = dict(who="someone", where="somewhere")
params = dict(where="our town", when="today") is valid, the similar:
defaults | params
{'who': 'someone', 'where': 'our town', 'when': 'today'} @item().thing
def foo():
Note that the order matters. In this case, the where value pass
from params overrides the default, as it should.
produces a syntax error.
Removing prefixes Starting in Python 3.9, any expression is valid as a decorator:
If you have done ad hoc text parsing or cleanup with Python,
you will have written code like: from unittest import mock

def process_pricing_line(line): item = mock.MagicMock()

if line.startswith("pricing:"):
return line[len("pricing:"):] @item().thing
return line def foo():
process_pricing_line("pricing:20") pass
'20' print(item.return_value.thing.call_args[0][0])
<function foo at 0x7f3733897040>
This kind of code is prone to errors. For example, if the string
is copied incorrectly to the next line, the price will become 0 While keeping to simple expressions in the decorator line is
instead of 20, and it will happen silently. still a good idea, it is now a human decision, rather than the
Since Python 3.9, strings have a .removeprefix() method: Python parser’s option.

>>> "pricing:20".removeprefix("pricing:") Welcome to 2020

'20' Python 3.9 was released about one year ago, but some of
the features that first showed up in this release are cool—and
underused. Add them to your toolkit if you haven’t already.

30 HIDDEN GEMS IN PYTHON 3 . CC BY-SA 4.0 . OPENSOURCE.COM 21