Datacenter architectures

1
 Data center networks
 10’s to 100’s of thousands of hosts, often closely coupled, in
close proximity:
• e-business (e.g. Amazon)
• content-servers (e.g., YouTube, Akamai, Apple, Microsoft)
• search engines, data mining (e.g., Google)
 challenges:
 multiple applications, each
serving massive numbers of
clients
 managing/balancing load,
avoiding processing,
networking, data bottlenecks
Inside a 40-ft Microsoft container,
Chicago data center
Link Layer
5-2
Data center networks
load balancer: application-layer routing
 receives external client requests
 directs workload within data center
 returns results to external client (hiding
Internet
data center internals from client)
Border router
Load Load
balancer Access router
balancer

Tier-1 switches
B

A C Tier-2 switches

TOR switches

Server racks

1 2 3 4 5 6 7 8
Link Layer 5-3
Data center networks
 rich interconnection among switches, racks:
 increased throughput between racks (multiple routing paths
possible)
 increased reliability via redundancy

Tier-1 switches

Tier-2 switches

TOR switches

Server racks

1 2 3 4 5 6 7 8
 Broad questions
 How are massive numbers of commodity machines
networked inside a data center?
 Virtualization: How to effectively manage physical
machine resources across client virtual
machines?
 Operational costs:
• Server equipment
• Power and cooling

5
Source: NRDC research paper 6
Breakdown wrt DC size

Source: NRDC research paper

7
Chapter 5
Link Layer

Computer
Networking: A Top
Down Approach
6th edition
Jim Kurose, Keith Ross
Addison-Wesley
March 2012

All material copyright 1996-2012

J.F Kurose and K.W. Ross, All Rights Reserved

Link Layer 5-8

Chapter 5: Link layer
our goals:
 understand principles behind link layer services:
 error detection, correction
 multiple access: sharing a broadcast channel
 link layer addressing
 local area networking: Ethernet, VLANs
 instantiation, implementation of various link layer
technologies

Link Layer 5-9

Link layer, LANs: outline
5.1 introduction, services 5.5 link virtualization:
5.2 error detection, MPLS
correction 5.6 data center networking
5.3 multiple access 5.7 a day in the life of a
protocols web request
5.4 LANs
 addressing, ARP
 Ethernet
 switches
 VLANS

Link Layer 5-10

Link layer: introduction
terminology:
 hosts and routers: nodes
 communication channels that global ISP

connect adjacent nodes along

communication path: links
 wired links
 wireless links
 LANs
 layer-2 packet: frame,
encapsulates datagram

data-link layer has responsibility of

transferring datagram from one node
to physically adjacent node over a link
Link Layer 5-11
Link layer: context
 datagram transferred by transportation analogy:
different link protocols over  trip from Amherst to Lausanne
different links:  limo: Amherst to BOS
 e.g., Ethernet on first link,  plane: BOS to Geneva
frame relay on  train: Geneva to Lausanne
intermediate links, 802.11  tourist = datagram
on last link  transport segment =
 each link protocol provides communication link
different services  transportation mode = link
 e.g., may or may not layer protocol
provide rdt over link  travel agent = routing
algorithm

Link Layer 5-12

Link layer services
 framing, multiple link access:
 encapsulate datagram into frame, adding header, trailer
 channel access if shared medium
 “MAC” addresses used in frame headers to identify source
and destination
• different from IP address!
• Q: why two addresses for the same interface?
 reliable delivery between adjacent nodes
 we learned how to do this already (chapter 3)!
 seldom used on low bit-error link (fiber, twisted pair)
 wireless links: high error rates, need link-layer reliability
• Q: why both link-level and end-end reliability?

Link Layer 5-13

Link layer services (more)
 flow control:
 pacing between adjacent sending and receiving nodes
 error detection:
 errors caused by signal attenuation, noise.
 receiver detects presence of errors: signals sender for
retransmission or drops frame
 error correction:
 receiver identifies and corrects bit error(s) without
resorting to retransmission
 half-duplex and full-duplex
 with half duplex, nodes at both ends of link can
transmit, but not at same time

Link Layer 5-14

Where is the link layer implemented?
 every host and router
 implemented in “adaptor” (aka
network interface card NIC) or
on a chip
 Ethernet card, 802.11 card; application
Ethernet chipset transport
network cpu memory
 implements link and link

physical layers host

 attaches to host system buses controller
bus
(e.g., PCI)
link
 combination of hardware, physical
physical
software, firmware transmission

network adapter
card

Link Layer 5-15

Adaptors communicating

datagram datagram

controller controller

sending host receiving host

datagram

frame

 sending side:  receiving side

 encapsulates datagram in  looks for errors, rdt, flow
link layer frame control, etc
 adds error checking bits,  extracts datagram, passes
rdt, flow control, etc. to upper layer

Link Layer 5-16

Link layer, LANs: outline
5.1 introduction, services 5.5 link virtualization:
5.2 error detection, MPLS
correction 5.6 data center networking
5.3 multiple access 5.7 a day in the life of a
protocols web request
5.4 LANs
 addressing, ARP
 Ethernet
 switches
 VLANS

Link Layer 5-17

Error detection and correction
EDC= Error Detection and Correction bits (redundancy)
D = Data protected by error checking, may include header fields

• Error detection not 100% reliable!

• protocol may miss some errors, but rarely
• larger EDC field yields better detection and correction

otherwise

Link Layer 5-18

Parity checking
single bit parity: two-dimensional bit parity:
 detect single bit  detect and correct single bit errors
errors

0 0

Link Layer 5-19

Internet checksum (review)
goal: detect “errors” (e.g., flipped bits) in transmitted packet
(note: used at transport layer only)

sender: receiver:
 treat segment contents as  compute checksum of
sequence of 16-bit received segment
integers  check if computed
 checksum: addition (1’s checksum equals checksum
complement sum) of field value:
segment contents  NO - error detected
 sender puts checksum  YES - no error detected.
value into UDP But maybe errors
checksum field nonetheless?

Link Layer 5-20

Cyclic redundancy check
 more powerful error-detection than Internet checksums
 view data bits, D, as a binary number
 choose r+1 bit pattern (generator), G
 goal: choose r CRC bits, R, such that
 <D,R> exactly divisible by G (modulo 2)
 receiver knows G, divides <D,R> by G. If non-zero remainder:
error detected!
 can detect all burst errors less than r+1 bits
 widely used in practice (Ethernet, 802.11 WiFi, ATM)

Link Layer 5-21

CRC example
want:
D.2r XOR R = nG
equivalently:
D.2r = nG XOR R
equivalently:
if we divide D.2r by
G, want remainder R
to satisfy:

D.2r
R = remainder[ ]
G

Link Layer 5-22

Link layer, LANs: outline
5.1 introduction, services 5.5 link virtualization:
5.2 error detection, MPLS
correction 5.6 data center networking
5.3 multiple access 5.7 a day in the life of a
protocols web request
5.4 LANs
 addressing, ARP
 Ethernet
 switches
 VLANS

Link Layer 5-23

Multiple access links, protocols
two types of “links”:
 point-to-point
 PPP for dial-up access
 point-to-point link between Ethernet switch, host
 broadcast (shared wire or medium)
 old-fashioned Ethernet
 upstream HFC
 802.11 wireless LAN

shared wire (e.g., shared RF shared RF humans at a

cabled Ethernet) (e.g., 802.11 WiFi) (satellite) cocktail party
(shared air, acoustical)

Link Layer 5-24

Multiple access protocols
 single shared broadcast channel
 two or more simultaneous transmissions  interference as
simultaneously received signals collide causing errors

multiple access protocol

 distributed algorithm that determines how nodes share channel,
i.e., determine when node can transmit
 communication about channel sharing must use channel itself!
 no out-of-band channel for coordination

Link Layer 5-25

An ideal multiple access protocol
given: broadcast channel of rate R bps
goal:
1. when one node wants to transmit, it can send at rate R.
2. when M nodes want to transmit, each can send at average
rate R/M
3. fully decentralized:
• no special node to coordinate transmissions
• no synchronization of clocks, slots
4. simple

Link Layer 5-26

MAC protocols: taxonomy
three broad classes:
 channel partitioning
 divide channel into smaller “pieces” (time slots, frequency, code)
 allocate piece to node for exclusive use
 random access
 channel not divided, allow collisions
 “recover” from collisions
 “taking turns”
 nodes take turns, but nodes with more to send can take longer
turns

Link Layer 5-27

Channel partitioning MAC protocols: TDMA
TDMA: time division multiple access
 access to channel in "rounds"
 each station gets fixed length slot (length = pkt
trans time) in each round
 unused slots go idle
 example: 6-station LAN, 1,3,4 have pkt, slots
2,5,6 idle

6-slot 6-slot
frame frame
1 3 4 1 3 4

Link Layer 5-28

Channel partitioning MAC protocols: FDMA
FDMA: frequency division multiple access
 channel spectrum divided into frequency bands
 each station assigned fixed frequency band
 unused transmission time in frequency bands go idle
 example: 6-station LAN, 1,3,4 have pkt, frequency bands 2,5,6
idle

time
frequency bands

FDM cable

Link Layer 5-29

Random access protocols
 when node has packet to send
 transmit at full channel data rate R.
 no a priori coordination among nodes
 two or more transmitting nodes ➜ “collision”,
 random access MAC protocol specifies:
 how to detect collisions
 how to recover from collisions (e.g., via delayed
retransmissions)
 examples of random access MAC protocols:
 slotted ALOHA
 ALOHA
 CSMA, CSMA/CD, CSMA/CA

Link Layer 5-30

Slotted ALOHA
assumptions: operation:
 all frames same size  when node obtains fresh
 time divided into same size frame, transmits in next slot
slots (time to transmit 1  if no collision: node can
frame) send new frame in next slot
 nodes start to transmit only  if collision: node
slot beginning retransmits frame in each
 nodes are synchronized subsequent slot with
 if 2 or more nodes transmit probability p until success
in slot, all nodes detect
collision

Link Layer 5-31

Slotted ALOHA
node 1 1 1 1 1

node 2 2 2 2

node 3 3 3 3

C E C S E C E S S

Pros: Cons:
 single active node can  collisions, wasting slots
continuously transmit at  idle slots
full rate of channel  nodes may be able to
 highly decentralized: only detect collision in less
slots in nodes need to be than time to transmit
in sync
packet
 simple  clock synchronization
Link Layer 5-32
Slotted ALOHA: efficiency

efficiency: long-run  max efficiency: find p*

fraction of successful slots that maximizes
(many nodes, all with [ ]
many frames to send)  for many nodes, take limit
of [ ] as N
 suppose: N nodes with goes to infinity, gives:
many frames to send, each max efficiency = 1/e = .37
transmits in slot with
probability p
at best: channel

!
 prob that given node has
success in a slot = used for useful
 prob that any node has a transmissions 37%
success = of time!

Link Layer 5-33

Slotted ALOHA: efficiency

efficiency: long-run  max efficiency: find p*

fraction of successful slots that maximizes
(many nodes, all with Np(1-p)N-1
many frames to send)  for many nodes, take limit
of Np*(1-p*)N-1 as N goes
 suppose: N nodes with to infinity, gives:
many frames to send, each max efficiency = 1/e = .37
transmits in slot with
probability p
at best: channel

!
 prob that given node has
success in a slot = p(1- used for useful
p)N-1 transmissions 37%
 prob that any node has a of time!
success = Np(1-p)N-1
Link Layer 5-34
Pure (unslotted) ALOHA
 unslotted Aloha: simpler, no synchronization
 when frame first arrives
 transmit immediately
 collision probability increases:
 frame sent at t0 collides with other frames sent in [t0-1,t0+1]

Link Layer 5-35

Pure ALOHA efficiency
P(success by given node) = P(node transmits) .
P(no other node transmits in [t0-1,t0] .
P(no other node transmits in [t0-1,t0]

= p . (1-p)N-1 . (1-p)N-1
= p . (1-p)2(N-1)

… choosing optimum p and then letting n

= 1/(2e) = .18

even worse than slotted Aloha!

Link Layer 5-36

CSMA (carrier sense multiple access)

CSMA: listen before transmit:

 if channel sensed idle: transmit entire frame
 if channel sensed busy, defer transmission

 human analogy: don’t interrupt others!

Link Layer 5-37

CSMA collisions spatial layout of nodes

 collisions can still occur:

propagation delay means
two nodes may not hear
other’s transmission
 collision: entire packet
transmission time
wasted
 distance & propagation
delay play role in in
determining collision
probability

Link Layer 5-38

CSMA/CD (collision detection)
CSMA/CD: carrier sensing, deferral as in CSMA
 collisions detected within short time
 colliding transmissions aborted, reducing channel wastage
 collision detection:
 easy in wired LANs: measure signal strengths, compare
transmitted, received signals
 difficult in wireless LANs: received signal strength
overwhelmed by local transmission strength
 human analogy: the polite conversationalist

Link Layer 5-39

CSMA/CD (collision detection)
spatial layout of nodes

Link Layer 5-40

Ethernet CSMA/CD algorithm
1. NIC receives datagram 4. If NIC detects another
from network layer, transmission while
creates frame transmitting, aborts and
2. If NIC senses channel sends jam signal
idle, starts frame 5. After aborting, NIC
transmission. Else if NIC enters binary
senses channel busy, waits (exponential) backoff:
until channel idle, then  after mth collision, NIC
transmits. chooses K at random
3. If NIC transmits entire from {0,1,2, …, 2m-1}.
frame without detecting NIC waits K·512 bit
another transmission, NIC times, returns to Step 2
is done with frame !  longer backoff interval
with more collisions
Link Layer 5-41
CSMA/CD efficiency
 tprop = max prop delay between 2 nodes in LAN
 ttrans = time to transmit max-size frame

1
efficiency 
1  5t prop /ttrans
 efficiency goes to 1
 as tprop goes to 0
 as ttrans goes to infinity
 better performance than ALOHA: and simple, cheap,
decentralized!

Link Layer 5-42

“Taking turns” MAC protocols
channel partitioning MAC protocols:
 share channel efficiently and fairly at high load
 inefficient at low load: delay in channel access, 1/N
bandwidth allocated even if only 1 active node!
random access MAC protocols
 efficient at low load: single node can fully utilize
channel
 high load: collision overhead
“taking turns” protocols
look for best of both worlds!

Link Layer 5-43

“Taking turns” MAC protocols
polling:
 master node “invites”
slave nodes to transmit data
in turn poll
 typically used with
“dumb” slave devices master
data
 concerns:
 polling overhead
 latency
 single point of slaves
failure (master)

Link Layer 5-44

“Taking turns” MAC protocols
token passing:
T
 control token passed
from one node to next
sequentially.
 token message
(nothing
 concerns: to send)
 token overhead T
 latency
 single point of failure
(token)

data
Link Layer 5-45
Cable access network
Internet frames,TV channels, control transmitted
downstream at different frequencies

cable headend

CMTS
…

splitter cable
cable modem … modem
termination system

ISP upstream Internet frames, TV control, transmitted

upstream at different frequencies in time slots

 multiple 40Mbps downstream (broadcast) channels

 single CMTS transmits into channels
 multiple 30 Mbps upstream channels
 multiple access: all users contend for certain upstream
channel time slots (others assigned)
Cable access network
cable headend MAP frame for
Interval [t1, t2]

Downstream channel i
CMTS
Upstream channel j

t1 t2 Residences with cable modems

Minislots containing Assigned minislots containing cable modem

minislots request frames upstream data frames

DOCSIS: data over cable service interface spec

 FDM over upstream, downstream frequency channels
 TDM upstream: some slots assigned, some have contention
 downstream MAP frame: assigns upstream slots
 request for upstream slots (and data) transmitted
random access (binary backoff) in selected slots
Link Layer 5-47
Summary of MAC protocols
 channel partitioning, by time, frequency or code
 Time Division, Frequency Division
 random access (dynamic),
 ALOHA, S-ALOHA, CSMA, CSMA/CD
 carrier sensing: easy in some technologies (wire), hard
in others (wireless)
 CSMA/CD used in Ethernet
 CSMA/CA used in 802.11
 taking turns
 polling from central site, token passing
 bluetooth, FDDI, token ring

Link Layer 5-48

Q1 Error detection/correction
 Can these schemes correct bit errors: Internet
checksums, two-dimendional parity, cyclic
redundancy check (CRC)
A. Yes, No, No
B. No, Yes, Yes
C. No, Yes, No
D. No, No, Yes
E. Ho, hum, ha

Data Link Layer 5-49

Q2 CRC vs Internet checksums
 Which of these is not true?
A. CRC’s are commonly used at the link layer
B. CRC’s can detect any bit error of up to r bits with
an r-bit EDC.
C. CRC’s are more resilient to bursty bit errors
D. CRC’s can not correct bit errors

Data Link Layer 5-50

Q3 Random access
 Consider an ALOHA network with N users that
transmit with probability p in slots just after a
collision. Assuming users have infinite data to
send, what is the probability that a slot is
successful (no collisions)?
A. Np
B. p(1-p)N-1
C. Np(1-p)N-1
D. C(N, N/2)p(1-p)N-1
E. Np/(1-p)

Data Link Layer 5-51

Q4 Random access
 Random access protocols achieve all four of the
properties below: True(A)/false(B)?
1. when one node wants to transmit, it can send at rate R.
2. when M nodes want to transmit, each can send at
average rate R/M
3. fully decentralized:
• no special node to coordinate transmissions
• no synchronization of clocks, slots
4. simple


Data Link Layer 5-52

Link layer, LANs: outline
5.1 introduction, services 5.5 link virtualization:
5.2 error detection, MPLS
correction 5.6 data center networking
5.3 multiple access 5.7 a day in the life of a
protocols web request
5.4 LANs
 addressing, ARP
 Ethernet
 switches
 VLANS

Link Layer 5-53

MAC addresses and ARP
 32-bit IP address:
 network-layer address for interface
 used for layer 3 (network layer) forwarding
 MAC (or LAN or physical or Ethernet) address:
 function: used ‘locally” to get frame from one interface to
another physically-connected interface (same network, in
IP-addressing sense)
 48 bit MAC address (for most LANs) burned in NIC ROM,
also sometimes software settable
 e.g.: 1A-2F-BB-76-09-AD

hexadecimal (base 16) notation

(each “number” represents 4 bits)

Link Layer 5-54

 LAN addresses and ARP
each adapter on LAN has unique LAN address

1A-2F-BB-76-09-AD

LAN
(wired or adapter
wireless)
71-65-F7-2B-08-53
58-23-D7-FA-20-B0

0C-C4-11-6F-E3-98

Link Layer 5-55

LAN addresses (more)
 MAC address allocation administered by IEEE
 manufacturer buys portion of MAC address space
(to assure uniqueness)
 analogy:
 MAC address: like Social Security Number
 IP address: like postal address
 MAC flat address ➜ portability
 can move LAN card from one LAN to another
 IP hierarchical address not portable
 address depends on IP subnet to which node is attached

Link Layer 5-56

 ARP: address resolution protocol
Question: how to determine
interface’s MAC address,
knowing its IP address? ARP table: each IP node (host,
router) on LAN has table
137.196.7.78
 IP/MAC address
mappings for some LAN
1A-2F-BB-76-09-AD
nodes:
137.196.7.23
137.196.7.14
< IP address; MAC address; TTL>
 TTL (Time To Live):
LAN time after which address
71-65-F7-2B-08-53 mapping will be
58-23-D7-FA-20-B0
forgotten (typically 20
min)
0C-C4-11-6F-E3-98
137.196.7.88

Link Layer 5-57

ARP protocol: same LAN
 A wants to send datagram
to B
 B’s MAC address not in A’s  A caches (saves) IP-to-
ARP table. MAC address pair in its
 A broadcasts ARP query ARP table until
packet, containing B's IP information becomes old
address (times out)
 dest MAC address = FF-  soft state: information that
FF-FF-FF-FF-FF times out (goes away)
 all nodes on LAN receive unless refreshed
ARP query  ARP is “plug-and-play”:
 B receives ARP packet,  nodes create their ARP
replies to A with its (B's) tables without intervention
MAC address from net administrator
 frame sent to A’s MAC
address (unicast)

Link Layer 5-58

 Addressing: routing to another LAN
walkthrough: send datagram from A to B via R
 focus on addressing – at IP (datagram) and MAC layer (frame)
 assume A knows B’s IP address
 assume A knows IP address of first hop router, R (how?)
 assume A knows R’s MAC address (how?)

A B
R
111.111.111.111
222.222.222.222
74-29-9C-E8-FF-55
49-BD-D2-C7-56-2A
222.222.222.220
1A-23-F9-CD-06-9B

111.111.111.112 111.111.111.110 222.222.222.221

CC-49-DE-D0-AB-7D E6-E9-00-17-BB-4B 88-B2-2F-54-1A-0F

Link Layer 5-59

Addressing: routing to another LAN
 A creates IP datagram with IP source A, destination B
 A creates link-layer frame with R's MAC address as dest, frame
contains A-to-B IP datagram
MAC src: 74-29-9C-E8-FF-55
MAC dest: E6-E9-00-17-BB-4B
IP src: 111.111.111.111
IP dest: 222.222.222.222

IP
Eth
Phy

A B
R
111.111.111.111
222.222.222.222
74-29-9C-E8-FF-55
49-BD-D2-C7-56-2A
222.222.222.220
1A-23-F9-CD-06-9B

111.111.111.112 111.111.111.110 222.222.222.221

CC-49-DE-D0-AB-7D E6-E9-00-17-BB-4B 88-B2-2F-54-1A-0F

Link Layer 5-60

Addressing: routing to another LAN
 frame sent from A to R
 frame received at R, datagram removed, passed up to IP

MAC src: 74-29-9C-E8-FF-55

MAC dest: E6-E9-00-17-BB-4B
IP src: 111.111.111.111
IP dest: 222.222.222.222
IP src: 111.111.111.111
IP dest: 222.222.222.222

IP IP
Eth Eth
Phy Phy

A B
R
111.111.111.111
222.222.222.222
74-29-9C-E8-FF-55
49-BD-D2-C7-56-2A
222.222.222.220
1A-23-F9-CD-06-9B

111.111.111.112 111.111.111.110 222.222.222.221

CC-49-DE-D0-AB-7D E6-E9-00-17-BB-4B 88-B2-2F-54-1A-0F

Link Layer 5-61

Addressing: routing to another LAN
 R forwards datagram with IP source A, destination B
 R creates link-layer frame with B's MAC address as dest, frame
contains A-to-B IP datagram

MAC src: 1A-23-F9-CD-06-9B

MAC dest: 49-BD-D2-C7-56-2A
IP src: 111.111.111.111
IP dest: 222.222.222.222
IP
IP Eth
Eth Phy
Phy

A B
R
111.111.111.111
222.222.222.222
74-29-9C-E8-FF-55
49-BD-D2-C7-56-2A
222.222.222.220
1A-23-F9-CD-06-9B

111.111.111.112 111.111.111.110 222.222.222.221

CC-49-DE-D0-AB-7D E6-E9-00-17-BB-4B 88-B2-2F-54-1A-0F

Link Layer 5-62

Addressing: routing to another LAN
 R forwards datagram with IP source A, destination B
 R creates link-layer frame with B's MAC address as dest, frame
contains A-to-B IP datagram

MAC src: 1A-23-F9-CD-06-9B

MAC dest: 49-BD-D2-C7-56-2A
IP src: 111.111.111.111
IP dest: 222.222.222.222
IP
IP Eth
Eth Phy
Phy

A B
R
111.111.111.111
222.222.222.222
74-29-9C-E8-FF-55
49-BD-D2-C7-56-2A
222.222.222.220
1A-23-F9-CD-06-9B

111.111.111.112 111.111.111.110 222.222.222.221

CC-49-DE-D0-AB-7D E6-E9-00-17-BB-4B 88-B2-2F-54-1A-0F

Link Layer 5-63

Addressing: routing to another LAN
 R forwards datagram with IP source A, destination B
 R creates link-layer frame with B's MAC address as dest, frame
contains A-to-B IP datagram
MAC src: 1A-23-F9-CD-06-9B
MAC dest: 49-BD-D2-C7-56-2A
IP src: 111.111.111.111
IP dest: 222.222.222.222

IP
Eth
Phy

A B
R
111.111.111.111
222.222.222.222
74-29-9C-E8-FF-55
49-BD-D2-C7-56-2A
222.222.222.220
1A-23-F9-CD-06-9B

111.111.111.112 111.111.111.110 222.222.222.221

CC-49-DE-D0-AB-7D E6-E9-00-17-BB-4B 88-B2-2F-54-1A-0F

Link Layer 5-64

Link layer, LANs: outline
5.1 introduction, services 5.5 link virtualization:
5.2 error detection, MPLS
correction 5.6 data center networking
5.3 multiple access 5.7 a day in the life of a
protocols web request
5.4 LANs
 addressing, ARP
 Ethernet
 switches
 VLANS

Link Layer 5-65

Ethernet
“dominant” wired LAN technology:
 cheap $20 for NIC
 first widely used LAN technology
 simpler, cheaper than token LANs and ATM
 kept up with speed race: 10 Mbps – 10 Gbps

Metcalfe’s Ethernet sketch

Link Layer 5-66
Ethernet: physical topology
 bus: popular through mid 90s
 all nodes in same collision domain (can collide with each
other)
 star: prevails today
 active switch in center
 each “spoke” runs a (separate) Ethernet protocol (nodes do
not collide with each other)

switch
star
bus: coaxial cable
Link Layer 5-67
Ethernet frame structure
sending adapter encapsulates IP datagram (or other
network layer protocol packet) in Ethernet frame
type
dest. source
preamble address address data CRC
(payload)

preamble:
 7 bytes with pattern 10101010 followed by one
byte with pattern 10101011
 used to synchronize receiver, sender clock rates

Link Layer 5-68

Ethernet frame structure (more)
 addresses: 6 byte source, destination MAC addresses
 if adapter receives frame with matching destination
address, or with broadcast address (e.g. ARP packet), it
passes data in frame to network layer protocol
 otherwise, adapter discards frame
 type: indicates higher layer protocol (mostly IP but
others possible, e.g., Novell IPX, AppleTalk)
 CRC: cyclic redundancy check at receiver
 error detected: frame is dropped

type
dest. source
preamble address address data CRC
(payload)

Link Layer 5-69

Ethernet: unreliable, connectionless
 connectionless: no handshaking between sending and
receiving NICs
 unreliable: receiving NIC doesnt send acks or nacks
to sending NIC
 data in dropped frames recovered only if initial
sender uses higher layer rdt (e.g., TCP), otherwise
dropped data lost
 Ethernet’s MAC protocol: unslotted CSMA/CD wth
binary backoff

Link Layer 5-70

802.3 Ethernet standards: link & physical layers
 many different Ethernet standards
 common MAC protocol and frame format
 different speeds: 2 Mbps, 10 Mbps, 100 Mbps, 1Gbps,
10G bps
 different physical layer media: fiber, cable

MAC protocol
application and frame format
transport
network 100BASE-TX 100BASE-T2 100BASE-FX
link 100BASE-T4 100BASE-SX 100BASE-BX
physical

copper (twister fiber physical layer

pair) physical layer
Link Layer 5-71
Link layer, LANs: outline
5.1 introduction, services 5.5 link virtualization:
5.2 error detection, MPLS
correction 5.6 data center networking
5.3 multiple access 5.7 a day in the life of a
protocols web request
5.4 LANs
 addressing, ARP
 Ethernet
 switches
 VLANS

Link Layer 5-72

Ethernet switch
 link-layer device: takes an active role
 store, forward Ethernet frames
 examine incoming frame’s MAC address,
selectively forward frame to one-or-more
outgoing links when frame is to be forwarded on
segment, uses CSMA/CD to access segment
 transparent
 hosts are unaware of presence of switches
 plug-and-play, self-learning
 switches do not need to be configured

Link Layer 5-73

Switch: multiple simultaneous transmissions
 hosts have dedicated, direct A
connection to switch
C’ B
 switches buffer packets
 Ethernet protocol used on each 6 1 2
incoming link, but no collisions;
full duplex 5 4 3
 each link is its own collision
domain B’ C
 switching: A-to-A’ and B-to-B’
can transmit simultaneously, A’
without collisions switch with six interfaces
(1,2,3,4,5,6)

Link Layer 5-74

Switch forwarding table
Q: how does switch know A’ A
reachable via interface 4, B’ B
C’
reachable via interface 5?
 A: each switch has a switch 6 1 2
table, each entry:
5 4 3
 (MAC address of host, interface
to reach host, time stamp) B’ C
 looks like a routing table!
A’
Q: how are entries created, switch with six interfaces
maintained in switch table? (1,2,3,4,5,6)
 something like a routing
protocol?
Link Layer 5-75
Switch: self-learning Source: A
Dest: A’

A A A’
 switch learns which hosts
can be reached through B
C’
which interfaces
 when frame received, 6 1 2
switch “learns”
location of sender: 5 4 3
incoming LAN segment
 records sender/location B’ C
pair in switch table
A’

MAC addr interface TTL

A 1 60 Switch table
(initially empty)

Link Layer 5-76

Switch: frame filtering/forwarding
when frame received at switch:

1. record incoming link, MAC address of sending host

2. index switch table using MAC destination address
3. if entry found for destination
then {
if destination on segment from which frame arrived
then drop frame
else forward frame on interface indicated by entry
}
else flood /* forward on all interfaces except arriving
interface */

Link Layer 5-77

Self-learning, forwarding: example Source: A
Dest: A’

A A A’
 frame destination, A’,
B
locaton unknown: flood C’

1
 destination A location 6 2

known:selectively send A A’
5 4 3
on just one link B’ C
A’ A

A’

MAC addr interface TTL

A 1 60 switch table
A’ 4 60 (initially empty)

Link Layer 5-78

Interconnecting switches
 switches can be connected together
S4

S1
S3
A S2
F
D I
B C
G H
E

Q: sending from A to G - how does S1 know to

forward frame destined to F via S4 and S3?
 A: self learning! (works exactly the same as in
single-switch case!)
Link Layer 5-79
Self-learning multi-switch example
Suppose C sends frame to I, I responds to C

S4

S1
S3
A S2
F
D I
B C
G H
E

 Q: show switch tables and packet forwarding in S1, S2, S3,

S4

Link Layer 5-80

Institutional network
mail server
to external
network
router web server

IP subnet

Link Layer 5-81

Switches vs. routers
application
transport
both are store-and-forward:
datagram network
 routers: network-layer frame link
devices (examine network- physical link frame
layer headers) physical
 switches: link-layer devices
(examine link-layer switch
headers)
network datagram
both have forwarding tables: link frame
 routers: compute tables physical
using routing algorithms, IP application
addresses transport
 switches: learn forwarding network
table using flooding, link
learning, MAC addresses physical

Link Layer 5-82

VLANs: motivation
consider:
 CS user moves office to
EE, but wants connect to
CS switch?
 single broadcast domain:
 all layer-2 broadcast
traffic (ARP, DHCP,
Computer
unknown location of
Science Electrical
Computer
Engineering
destination MAC
Engineering address) must cross
entire LAN
 security/privacy,
efficiency issues

Link Layer 5-83

 port-based VLAN: switch ports
VLANs grouped (by switch management
software) so that single physical
switch ……
Virtual Local
1 7 9 15
Area Network 2 8 10 16

switch(es) supporting
VLAN capabilities can … …
be configured to Electrical Engineering Computer Science
define multiple virtual (VLAN ports 1-8) (VLAN ports 9-15)

LANS over single … operates as multiple virtual switches

physical LAN
infrastructure. 1 7 9 15
2 8 10 16

… …

Electrical Engineering Computer Science

(VLAN ports 1-8) (VLAN ports 9-16)

Link Layer 5-84

 Port-based VLAN
router
 traffic isolation: frames
to/from ports 1-8 can only
reach ports 1-8
 can also define VLAN based on
MAC addresses of endpoints, rather
than switch port 1 7 9 15

2 8 10 16

 dynamic membership: ports

can be dynamically assigned … …
among VLANs Electrical Engineering Computer Science
(VLAN ports 1-8) (VLAN ports 9-15)

 forwarding between VLANS: done

via routing (just as with separate
switches)
 in practice vendors sell combined
switches plus routers

Link Layer 5-85

VLANS spanning multiple switches
1 7 9 15 1 3 5 7

2 8 10 16 2 4 6 8

… …

Electrical Engineering Computer Science Ports 2,3,5 belong to EE VLAN

(VLAN ports 1-8) (VLAN ports 9-15) Ports 4,6,7,8 belong to CS VLAN

 trunk port: carries frames between VLANS defined over

multiple physical switches
 frames forwarded within VLAN between switches can’t be vanilla
802.1 frames (must carry VLAN ID info)
 802.1q protocol adds/removed additional header fields for frames
forwarded between trunk ports

Link Layer 5-86

802.1Q VLAN frame format
type

preamble dest. source data (payload) CRC

address address 802.1 frame

type
dest. source
preamble
address address
data (payload) CRC 802.1Q frame

2-byte Tag Protocol Identifier Recomputed

(value: 81-00) CRC

Tag Control Information (12 bit VLAN ID field,

3 bit priority field like IP TOS)

Link Layer 5-87

Link layer, LANs: outline
5.1 introduction, services 5.5 link virtualization:
5.2 error detection, MPLS
correction 5.6 data center networking
5.3 multiple access 5.7 a day in the life of a
protocols web request
5.4 LANs
 addressing, ARP
 Ethernet
 switches
 VLANS

Link Layer 5-88

Multiprotocol label switching (MPLS)
 initial goal: high-speed IP forwarding using fixed
length label (instead of IP address)
 fast lookup using fixed length identifier (rather than
shortest prefix matching)
 borrowing ideas from Virtual Circuit (VC) approach
 but IP datagram still keeps IP address!

PPP or Ethernet
MPLS header IP header remainder of link-layer frame
header

label Exp S TTL

20 3 1 5
Link Layer 5-89
MPLS capable routers
 a.k.a. label-switched router
 forward packets to outgoing interface based only on
label value (don’t inspect IP address)
 MPLS forwarding table distinct from IP forwarding tables
 flexibility: MPLS forwarding decisions can differ
from those of IP
 use destination and source addresses to route flows to same
destination differently (traffic engineering)
 re-route flows quickly if link fails: pre-computed backup
paths (useful for VoIP)

Link Layer 5-90

MPLS versus IP paths

R6
D
R4 R3
R5
A
R2

 IP routing: path to destination IP router

determined by destination address
alone

Link Layer 5-91

MPLS versus IP paths
entry router (R4) can use different MPLS
routes to A based, e.g., on source address
R6
D
R4 R3
R5
A
R2

 IP routing: path to destination IP-only

determined by destination address router
alone
 MPLS routing: path to destination can MPLS and
IP router
be based on source and dest. address
 fast reroute: precompute backup routes in
case of link failure
Link Layer 5-92
MPLS signaling
 modify OSPF, IS-IS link-state flooding protocols to
carry info used by MPLS routing,
 e.g., link bandwidth, amount of “reserved” link bandwidth
 entry MPLS router uses RSVP-TE signaling protocol
to set up MPLS forwarding at downstream routers

RSVP-TE
R6
D
R4
R5 modified
link state A
flooding

Link Layer 5-93

MPLS forwarding tables
in out out
label label dest
interface
10 A 0 in out out
12 D 0 label label dest
interface
8 A 1 10 6 A 1
12 9 D 0

R6
0 0
D
1 1
R4 R3
R5
0 0
A
R2 in outR1 out
label label dest
in out out
interface
label label dest 6 - A 0
interface
8 6 A 0
Link Layer 5-94
Link layer, LANs: outline
5.1 introduction, services 5.5 link virtualization:
5.2 error detection, MPLS
correction 5.6 data center networking
5.3 multiple access 5.7 a day in the life of a
protocols web request
5.4 LANs
 addressing, ARP
 Ethernet
 switches
 VLANS

Link Layer 5-95

Data center networks
 10’s to 100’s of thousands of hosts, often closely
coupled, in close proximity:
 e-business (e.g. Amazon)
 content-servers (e.g., YouTube, Akamai, Apple, Microsoft)
 search engines, data mining (e.g., Google)
 challenges:
 multiple applications, each
serving massive numbers of
clients
 managing/balancing load,
avoiding processing,
networking, data bottlenecks
Inside a 40-ft Microsoft container,
Chicago data center
Link Layer 5-96
Data center networks
load balancer: application-layer
routing
 receives external client requests
 directs workload within data center
Internet  returns results to external client (hiding data
center internals from client)
Border router
Load Load
balancer Access router
balancer

Tier-1 switches
B

A C Tier-2 switches

TOR switches

Server racks

1 2 3 4 5 6 7 8
Link Layer 5-97
Data center networks
 rich interconnection among switches, racks:
 increased throughput between racks (multiple routing paths
possible)
 increased reliability via redundancy

Tier-1 switches

Tier-2 switches

TOR switches

Server racks

1 2 3 4 5 6 7 8
Link layer, LANs: outline
5.1 introduction, services 5.5 link virtualization:
5.2 error detection, MPLS
correction 5.6 data center networking
5.3 multiple access 5.7 a day in the life of a
protocols web request
5.4 LANs
 addressing, ARP
 Ethernet
 switches
 VLANS

Link Layer 5-99

Synthesis: a day in the life of a web request
 journey down protocol stack complete!
 application, transport, network, link
 putting-it-all-together: synthesis!
 goal: identify, review, understand protocols (at all
layers) involved in seemingly simple scenario:
requesting www page
 scenario: student attaches laptop to campus network,
requests/receives www.google.com

Link Layer5-100
 A day in the life: scenario

browser DNS server

Comcast network
68.80.0.0/13

school network
68.80.2.0/24

web page

web server Google’s network

64.233.169.105 64.233.160.0/19

Link Layer5-101
A day in the life… connecting to the Internet
DHCP DHCP  connecting laptop needs to
UDP
DHCP
DHCP IP
get its own IP address, addr
DHCP Eth of first-hop router, addr of
Phy DNS server: use DHCP
DHCP
 DHCP request encapsulated
in UDP, encapsulated in IP,
DHCP
DHCP
DHCP UDP
encapsulated in 802.3
DHCP IP Ethernet
DHCP Eth router
Phy (runs DHCP)  Ethernet frame broadcast
(dest: FFFFFFFFFFFF) on
LAN, received at router
running DHCP server
 Ethernet demuxed to IP
demuxed, UDP demuxed to
DHCP
Link Layer5-102
A day in the life… connecting to the Internet
DHCP DHCP  DHCP server formulates
DHCP UDP DHCP ACK containing
DHCP IP client’s IP address, IP
DHCP Eth address of first-hop router
Phy for client, name & IP
address of DNS server
 encapsulation at DHCP
DHCP DHCP server, frame forwarded
DHCP UDP (switch learning) through
DHCP IP LAN, demultiplexing at
DHCP Eth router client
Phy (runs DHCP)
DHCP
 DHCP client receives
DHCP ACK reply

Client now has IP address, knows name & addr of DNS

server, IP address of its first-hop router

Link Layer5-103
A day in the life… ARP (before DNS, before HTTP)
DNS DNS  before sending HTTP request, need
DNS UDP IP address of www.google.com:
DNS
ARP
IP DNS
ARP query Eth
Phy  DNS query created, encapsulated in
UDP, encapsulated in IP,
encapsulated in Eth. To send frame
ARP
to router, need MAC address of
ARP reply Eth
Phy router interface: ARP
router
 ARP query broadcast, received by
(runs DHCP) router, which replies with ARP
reply giving MAC address of
router interface
 client now knows MAC address
of first hop router, so can now
send frame containing DNS
query
Link Layer5-104
A day in the life… using DNS DNS
DNS UDP DNS server
DNS IP
DNS DNS DNS Eth
DNS UDP DNS Phy
DNS IP
DNS Eth
Phy
DNS
Comcast network
68.80.0.0/13

router
 IP datagram forwarded from
(runs DHCP) campus network into comcast
 IP datagram containing DNS network, routed (tables created by
query forwarded via LAN RIP, OSPF, IS-IS and/or BGP
switch from client to 1st hop routing protocols) to DNS server
router  demux’ed to DNS server
 DNS server replies to client
with IP address of
www.google.com
Link Layer5-105
A day in the life…TCP connection carrying HTTP
HTTP
HTTP
SYNACK
SYN TCP
SYNACK
SYN IP
SYNACK
SYN Eth
Phy

 to send HTTP request, client

first opens TCP socket to
web server
router  TCP SYN segment (step 1 in 3-
(runs DHCP)
SYNACK
SYN TCP way handshake) inter-domain
SYNACK
SYN IP routed to web server
SYNACK
SYN Eth
Phy  web server responds with TCP
SYNACK (step 2 in 3-way
web server handshake)
64.233.169.105  TCP connection established!

Link Layer5-106
A day in the life… HTTP request/reply
HTTP
HTTP HTTP  web page finally (!!!) displayed
HTTP
HTTP TCP
HTTP
HTTP IP
HTTP
HTTP Eth
Phy

 HTTP request sent into TCP

socket
router  IP datagram containing HTTP
HTTP HTTP (runs DHCP)
HTTP TCP
request routed to
HTTP IP www.google.com
HTTP Eth  web server responds with
Phy HTTP reply (containing web
page)
web server
64.233.169.105
 IP datagram containing HTTP
reply routed back to client
Link Layer5-107
Chapter 5: Summary
 principles behind data link layer services:
 error detection, correction
 sharing a broadcast channel: multiple access
 link layer addressing
 instantiation and implementation of various link
layer technologies
 Ethernet
 switched LANS, VLANs
 virtualized networks as a link layer: MPLS
 synthesis: a day in the life of a web request

Link Layer5-108
Chapter 5: let’s take a breath
 journey down protocol stack complete (except
PHY)
 solid understanding of networking principles,
practice
 ….. could stop here …. but lots of interesting
topics!
 wireless
 multimedia
 security
 network management

Link Layer5-109
DATACENTER NETWORK DESIGNS

Data Link Layer

5-110
 Scaling a LAN network
 Self-learning Ethernet switches work great at small
scales, but buckle at larger scales
• Broadcast overhead of self-learning linear in the total
number of interfaces
• Broadcast storms possible in non-tree topologies
 Goals
• Scalability to a very large number of machines
• Isolation of unwanted traffic from unrelated subnets
• Ability to accommodate general types of workloads (Web,
database, MapReduce, scientific computing, etc.)

111
Typical DC network components
 rich interconnection among switches, racks:
 increased throughput between racks (multiple routing paths
possible)
 increased reliability via redundancy

Tier-1 or core
switches

Tier-2 or
aggregation
switches

TOR switches

Server racks

1 2 3 4 5 6 7 8
 DC network design questions
 Core and aggregation switches much faster than ToR
switches
 How much faster should core and aggregation switches
need to be than ToR switches?
 How many ports do core/aggregation switches need to
support for a given number of ToR switch ports?
 How many cables need to be run in total for a N
machine datacenter?
 What bisection bandwidth can be achieved?
Q: Why can’t we just build a single BIG switch to
interconnect all machines?
113
 DC network topologies
 Fat-tree (used ambiguously to mean Clos as well as a
simple hierarchical design)
 Clos family
 Hypercube
 Torus

114
 Why simpler hierarchies not good enough?
 High cost
 High oversubscription (ratio of worst-case aggregate
bandwidth among end-hosts to bisection bandwidth)

115
 Fat tree topology
 Core branches, i.e., those near the top of the hierarchy,
are fatter or higher in capacity

116
Example: uniform Clos topology [UCSD]

[UCSD] A Scalable Commodity Data Center Network Architecture

117
 Clos family
 Ingress, intermediate, and egress switches where each
stage’s links form a bipartite graph

118
VL2: Clos case study (Microsoft)

119
VL2: Addressing and routing

120
 Valiant load balancing
 Randomization for efficient, load-balanced routing [VLB]

[VLB] Valiant Load-Balancing: Building Networks That Can Support All Traffic Matrices

121
VL2: Directory for AA<->LA mappings

122
BCube: relies on more server ports

123
Other topologies from “supercomputing”

Hypercube

124
 Optical in data centers
 Optical switching (100’s of Gbps) faster than traditional
switches (40-160Gbps).
 Optical cheaper per 10Gbps port
 But optical circuit establishment delay high
• MEMS (Micro-electro mechanical systems) reconfiguration
time is ~10ms
 Optical enhanced data center designs migrate heavy
flows (elephants) to optical pathways

125
 Energy usage numbers
 Typical US household: ~1000kWh per month or ~30kW
 Typical desktop computer: 80-250 W
 Typical 1U rack mounted server: ~300W (can be a few
thousand W for high-end servers)
 Switches and networking equipment?

126
 Switch power consumption
 Generally small fraction (5-25%) of servers in typical
topologies

127
 Techniques to reduce energy
 Dynamic voltage and frequency scaling (DVFS): reduces
CV2f by reducing voltage V
• Generally not power-proportional, i.e., power does not
proportionally go down with decreased usage
 Shutting down (“consolidating”) servers and parts of
network: widely studied by cautiously used if at all in
practice

128