Algorithms

(for Game Design)

Session 7: Sound Effects and Music
 Last time
• Network Architecture

• TCP vs UDP

• Network Programming

• Meshed Peer Topology

• Client-Server Topology

• Client side prediction

• Starsiege: Tribes
 What do we want?

• At the Application Level = Programmer's View

• All the data you send as a message will get

instantly delivered to the other instances of your
application, running on other player's computers

• A useful abstraction, but there are problems

 All Together!
messages
Application Application

segments
Transport Transport

packets packets packets

Network Network Network Network

frames frames frames

Data Link Data Link Data Link Data Link

bits bits bits

Physical Physical Physical Physical
 Meshed Peer Topology
• Early days! Each player sent and
received data from all other players

• Peer-to-Peer Lockstep: Game play

happens in a series of turns

• In each turn, all peers send player input to all other peers
➙ sequence of commands

• Each player's machine executes all the commands in the

same order

• Identical outcomes on all computers!

• Solution: Client keeps track of all inputs and
when they happened
Turn N N+1 N+2 N+3
• When an update
comes from the Game
State
Fully
Known
Predicted Predicted Predicted
server, the events
are replayed against
Local
the past game state Events

• Known as
Server
Deterministic
lockstep Full
Update
 Today

• Sound Theory

• Sound in Video Games

• Sound Technology

• Sound in Python
 Sound is Good!
• Games are better with sound

• Play them without sound to see the difference

• Sounds:

• Set the Mood

• Provide Guidance
 Mood
• Music has emotional content

• Tempo and feel can make a player

• Tense

• Relax

• Pretty obvious ➙ just ask lmmakers how

important a soundtrack is
fi
 Guidance

• Music and sound can provide clues to the player

• Such as a non-player character's speech, distant

explosions, increased tension in background music

• Sound generation is similar to, but simpler than,

video/graphic generation

• in most games is not algorithmically generated, but

loaded from pre-recorded les
fi
 Audio Concepts
• Sound = series of pressure waves

• Generated by some physical phenomenon

• Travel through the atmosphere

• Ears convert pressure to brain signals

• Brain interprets them as information

• Each pressure wave is made up of one or more
sine waves

• This graph is 12 periods of 60 Hz data

• There are lots of different pressure data points per second

• This is an "analog" signal, not digital

• Brains do an amazing job of translating this to information

• Normal sounds ➙ lots of different sine waves

• This graph has 20, 60 and 70Hz signals all combined

sin(2π(20Hz)t) + sin(2π(60Hz)t) + sin(2π(70Hz)t)

• Amazing! You can see the sum of three values

• ... yet still be able to gure out the original values

• Because you know they are sine waves

fi
Digital Signal Processing
• Modern computer algorithms try to do the same
things your brain is doing

• By looking at a signal (i.e. such a graph)

• Try to gure out what it represents

• i.e. what sounds

fi
 The Sensor
• Your ear is a sensor that
detects the frequency of
the pressure waves

• Detecting the "color" of

sound

• But, it is a single "pixel"

sensor

Image credit: https://www.news-medical.net/whitepaper/20200114/Measuring-Vibration-Patterns-of-the-Ear.aspx

Combining Many Signals
• Your single-pixel ear sensor receives many different
signals from different sources
Chirp

• Each signal combines

many sine waves

• Your ear adds them Buzz

all together

• Brain does an amazing

job of separating some
sounds (background vs. conversation vs. alarm)
 Directionality
• You have two ears

• The distance between means pressure waves

arrive at slightly different times

• Takes time for wave to wrap around your head

• Phase difference between the signals

sin(2πf + θ)
 Sound Localization
Source

• As phase difference gets closer to zero that

means the source is closer to in front of you (or
behind you)
 Cocktail Party Problem
• Your brain is amazing at
pulling information from
many signals arriving
simultaneously at your ears

• and getting added together

• Often, modern signal processing algorithms

struggle to perform as well
sound in games
 Back to Video Games
• Sound has multiple uses in video games

• Background music: sets emotional tone,

sends useful information to the user

• Speech: a narrator says stuff. Or, guards are

overheard speaking to each other

• Player is supposed to hear and understand

this information
• Ambient sounds: a form of background music

• Shorter clips, often position speci c

• Ex: clank of dishes in a cafe, cows mooing on

a farm, raindrops on the roof

• Also give information

• When the bad guy steps into the bar, the

clank of dishes stop, other patrons murmer
ot each other, piano player stops playing...
fi
• Sound effects: the screech of a car crash

• or explosion of a grenade

• Gives important feedback and information to

the player

• Especially if it is his car or grenade

Localizing these sounds in 3D is often important!

Short History
• Sound doesn't have to be extensive to be very,
very effective

• Listen to Space_Invaders_Music.mp3

• The rst "continuous music" in a video game

• First time music was intended to drive an

emotional response from the player
fi
sound technology
 Sound File

• Stores "samples of sound" for playback

• How is this created?

• Microphone ➙ ADC ➙ Sampling process ➙

convert to number ➙ store in le
fi
 Microphone
• A microphone converts sound into electricity

• sound = sine wave of air pressure

• electricity = voltage that varies over time

Same graph
as before

y-axis is now
voltage
 ADC
• Analog to Digital Converter

• Gives a number for the voltage amount

• Ex: If the voltage varies from 0 to +16 volts

and the number is 4-bits, then anything from 0
to 1 volts is 0000, 1 to 2 volts is 0001...

• Takes a bit of time (say, 10 μsec) to do a

conversion
 Sampling
• If not enough samples are taken, the signal can't
be reconstructed

• Nyquist Theorem ➙ sample 2x faster than the

highest frequency in the signal

10 samples per
period sampling
5x faster than
required
 Music Sampling
• Human ears have a range of ~ 40Hz - 15,000Hz

• Some people can hear higher frequencies

• Older people can't

• Requires good speakers to reproduce a wider

frequency range

• Nyquist: 2 x 15kHz is 30K samples / second

Music Sampling Standards

• Compact Disc (CD) technology

samples at 44.1kHz

• Implies highest frequency sine waves in the

music is 22 kHz

• First common standard for digitized music

Image credit: wikipedia.org/wiki/Compact_disc

 File Size
• At 44.1kHz, one minute of music (with 16-bit integer
values) would take up:

seconds samples bits

one minute × 60 × 44,100 × 16
minute second sample

42,336,000 bits = 5,292,000 bytes

• Double that for stereo sound (one channel for left- and
another for right-ear sounds)

• Compression is essential ➙ MP3 format gets 70-95%

reduction in size
Sound in Pygame
 Pygame

• Almost all sound in pygame comes from playing

stored les

• Mostly WAV or OGG les

• Pygame version 2.0 and up can play MP3

• pygame.mixer module (and pygame.music)

fi
fi
 The Music Module
• Since version 2.0 of Pygame, there is a music
module

• Works the same as the mixer module

• but it streams data, rather than loading it all at

initialization time

• I haven't found any differences for small games

Mixer
• A mixer allows many
different sounds to be
played, each at different volumes

• Modeled after a real hardware device

• Found at music studio or concert

• Sound engineer uses it to add together all the

sounds coming from various instruments or
microphones
 Another Picture
• Here is a picture of different real-world mixer

• Notice all the cables and connection points for

different sounds to come into the mixer
 Mixer Capabilities
• The mixer module is limited by
the sound hardware on your
computer

• This picture is a 16-channel mixer

• More channels could be available on other

(more expensive) mixers

• Same with your computer and pygame.mixer

 Sound
• pygame.mixer.Sound class creates simple sounds

pygame.mixer

Sound(filename) Sound.play()
WAV file pygame.mixer.Sound
Sound
Sound.get_length() Hardware
Sound.set_volume(.8)
Sound.stop()
Sound.fadeout(500)

• Create a Sound object with a lename for the sampled data

• play( ) will convert the sound data into the right

messages to the sound hardware and your sound will play
fi
 pygame.mixer

Sound(filename) Sound.play()
WAV file pygame.mixer.Sound
Sound
Sound.get_length() Hardware
Sound.set_volume(.8)
Sound.stop()
Sound.fadeout(500)

• get_length( ) will tell you the number of seconds it will

take to play the sound

• set_volume( ) with a number 0-1 to represent the

loudness

• stop( ) will immediately cease playing the sound

• fadeout( ) will cease playing, but by linearly decreasing

the volume to zero over some number of milliseconds
 Minimal Sound Program
import pygame
mixer_minimal.py

pygame.init()
snd = pygame.mixer.Sound('JDB - Mysterious.wav')
snd.play()

running = True
while running: mixer needs the event
for event in pygame.event.get(): loop running
if event.type == pygame.KEYDOWN:
running = False
demo of sound0.py
 while True: Each time around game loop
clock.tick(30)
quit, up, down = check_events()
audio.update(up, down) update( ) and draw( )
surface.fill((0,0,0))
audio.draw(surface)
pygame.display.flip()

class AudioController:
make sound object (from le data)
def __init__(self):
self.volume = 0.5
self.mix = pygame.mixer.Sound('assets/sounds/JDB - Traveling.wav')
self.mix.play()
self.mix.set_volume(self.volume) start playing it
def update(self, up, down):
if down:
self.volume = max(0.0, self.volume - 0.1)
update( ) just changes
if up: volume on sound object
self.volume = min(1.0, self.volume + 0.1)
self.mix.set_volume(self.volume)

def draw(self, surface):

# Draw circle ....
fi
 Multiple Sounds
pygame.mixer

Sound(filename)
WAV file pygame.mixer.Sound Sound.play()

+ Sound
Hardware
Sound(filename)
WAV file pygame.mixer.Sound Sound.play()

• If you create and play( ) several Sound objects,

mixer will combine them into a single sound

• Result is the sum of all sounds

 The Channel
• Channel gives you ner control
pygame.mixer
a_sound a_channel.play(a_sound)
Sound(filename)
WAV file pygame.mixer.Sound pygame.mixer.Channel

Sound.get_length() + Sound
Hardware
Sound.set_volume(.8)

a_channel.set_volume(.2,.7)
a_channel.set_endevent(pygame.locals.USEREVENT)

• Create with pygame.mixer.Channel(id)

• The channel id is an integer from zero up through

pygame.mixer.get_num_channels( )

• Use your_channel.play(a_sound) to play a sound

fi
 Uses of the Channel
• There are two reasons to use a Channel

1. You want control of left and right channels of

stereo sound

2. You want to know when the sound is done playing

3. You want to know when the sound is done playing

• Yes, yes, #2 and #3 look alike

 Control the Volume
pygame.mixer
a_sound a_channel.play(a_sound)
Sound(filename)
WAV file pygame.mixer.Sound pygame.mixer.Channel

Sound.get_length() + Sound
Hardware
Sound.set_volume(.8)

a_channel.set_volume(.2,.7)
a_channel.set_endevent(pygame.locals.USEREVENT)

• Channel has a set_volume method also!

• Two oat parameters, 0 - 1, for left and right-side

volumes

• Note: Total volume is the product of the Sound

volume and the Channel volume
fl
 When is it done? (version 1)
pygame.mixer
a_sound a_channel.play(a_sound)
Sound(filename)
WAV file pygame.mixer.Sound pygame.mixer.Channel

Sound.get_length() + Sound
Hardware
Sound.set_volume(.8)

a_channel.set_volume(.2,.7)
a_channel.set_endevent(pygame.locals.USEREVENT)

• You can test if the sound is done playing

• Call a_channel.get_busy( )

• Returns True when a sound is being played

• "Polling"
 When is it done? (version 2)
pygame.mixer
a_sound a_channel.play(a_sound)
Sound(filename)
WAV file pygame.mixer.Sound pygame.mixer.Channel

Sound.get_length() + Sound
Hardware
Sound.set_volume(.8)

a_channel.set_volume(.2,.7)
a_channel.set_endevent(pygame.locals.USEREVENT)

• Tell the channel to send you an event when it is done

• Call a_channel.set_endevent(my_event_type)

• When the sound is done, an event of this type will be put on the
event queue

• Use my_event_type = pygame.event.custom_type( )

• Then, in the event loop, check if event.type == my_event_type

sound localization
 Localization
• Making sounds appear to come from a particular
source

• Thus, manipulating the phase difference

between sounds being sent to each ear

• Requires precise control over when each sound

le is played in each ear

• Synchronization would be critical

fi
 Localization Math
• Sound travels through air at about 340 m/s

• Your ears are about 25 cm apart

• How much time difference is that?

1 sec 1m
25cm × × = 0.000735 sec
340 m 100 cm
• Tough to synchronize in Pygame to within 735 μsec
 "Fake" Localization
• Your brain interprets loud sounds as if they are
close and quiet sounds as if they are far

• Exploit this to do rough localization

• Change the volume of sounds sent to each ear

• Not exact, but good enough for "guard's

footsteps are coming from the right" or "bomb
exploded to your left"
"Fake" Localization: Pygame
• Remember, use a channel's set_volume( )
method with different volumes for left/right
sounds

• Keep the sum of the two volumes the same if

you just want the sound to go from side to side
(panning the sound)

• Decrease / increase the overall sound to go

away or move towards the player
 Simple Code

def set_pan(leftness):
rightness = 1-leftness
self.channel.set_volume(leftness, rightness)
demo of sound1.py
class AudioController: Create a sound object
def __init__(self):
self.volume = 0.5 Get a channel object
self.right_channel = 0.5
self.mix = pygame.mixer.Sound('assets/sounds/JDB - Traveling.wav')
self.channel = pygame.mixer.find_channel()
self.channel.play(self.mix)
self.set_volume_pan() play the object in the channel

def set_volume_pan(self): set the channel volumes

left = (1-self.right_channel) * self.volume
right = self.right_channel * self.volume
self.channel.set_volume(left, right)

def update(self, left, right, up, down):

if down:
update( ) in the game loop
self.volume = max(0.0, self.volume - 0.1)
... # similar for up, left, right
self.set_volume_pan()
determines new volume/right values
mathematically manipulated sounds
 Create Your Own
• Just like it is possible to use graphic primitives to
create pictures

• It is possible to use mathematical operations to create

sounds

• The pygame.sndarray module allows sound data to be

loaded into numpy arrays (and vis versa)

• numpy is a numerical computing library for Python

• Does very fast arrays

 sndarray Example

• I loaded the Space Invaders music, converted it

to a numpy array and then plotted one channel

• I used matplotlib, a Python library for making

pretty plots

• Both numpy and matplotlib are extremely

useful, so look into them sometime
 Pulses are visible

y-axis: 16-bit
integer value
of the sample

x-axis: Sample number (and thus time)

44,100 samples per second, so this is ~5.5 seconds
Zoom in on that rst pulse

y-axis: 16-bit
integer value
of the sample

x-axis: Sample number (and thus time)

44,100 samples per second, so this is ~5.5 seconds
fi
 Can I Make My Own?
• I then added three sine waves together and
made a big array of their values

10000 sin(2π200) + 10000 sin(2π300) + 10000 sin(2π800)

• Here is a plot of the

rst .02 seconds
fi
 sndarray_demo.py
pygame.init()

snd = pygame.mixer.Sound('Space_Invaders_Music.ogg')
arr = pygame.sndarray.array(snd)
print_array_info(arr)
plt.plot(arr[ ,0]) This is the rst plot
plt.show()

print('Zooming in on a single buzz')

plt.plot(arr[ 5000,0])
This is the second,
plt.show() zoomed in, plot
fi
:
:
print('Now making my own buzz')
amp1, freq1, phase1 = 10000, 300, 30 # frequency in Hz
amp2, freq2, phase2 = 10000, 200, 0 # phase in degrees
amp3, freq3, phase3 = 10000, 800, 80

# Timescale An array with the sum of the

samples_per_second = 44100
three functions (in stereo)
seconds = 4
t = np.linspace(0, seconds, num=seconds*samples_per_second)
f1 = amp1 * np.sin(t * 2 * np.pi * freq1 + np.pi / 180 * phase1)
f2 = amp1 * np.sin(t * 2 * np.pi * freq2 + np.pi / 180 * phase2)
f3 = amp1 * np.sin(t * 2 * np.pi * freq3 + np.pi / 180 * phase3)
f = np.transpose(np.array([f1 + f2 + f3,f1 + f2 + f3],
dtype=np.int16))
f = f.copy() # Gets around non-contiguous error message
print_array_info(f)
samples_to_plot = round(.02 * samples_per_second)
plt.plot(f[ samples_to_plot,0])
plt.show() Plot the rst .02 seconds
fi
:
newsnd = pygame.sndarray.make_sound(f)
newsnd.play()
Convert the array into a Sound

running = True
while running:
for event in pygame.event.get():
if event.type == pygame.KEYDOWN:
running = False
Needs an event loop to play a Sound
preparing sound les
fi
 Sound Editing Apps
• There are tons of possibilities here

• Audacity is open-source (and looks it!)

• I've been using WavePad Audio Editor

• Free (ad supported) and fairly performant

• It also has an interface from the 90s

 Apple's Logic
• Logic is Apple's pro app for sound engineers

• Costs $199 and very capable and extremely

well-thought-of in the industry

• Probably very
overpowered
for simple
hobby game
makers
Free Game Sound Effects

• opengameart.org/content/library-of-game-
sounds

• soundbible.com
demo of sound2.py
What did you learn today?
• Sounds are very useful in games

• The physics of sound

• Sound recording technology

• Sounds in pygame

• Localization

• Mathematical primitives