Network Programming with Go Language: Essential Skills for Programming, Using and Securing Networks with Open Source Google Golang 2nd Edition Jan Newmarch pdf download
Network Programming with Go Language: Essential Skills for Programming, Using and Securing Networks with Open Source Google Golang 2nd Edition Jan Newmarch pdf download
https://ebookmeta.com/product/network-programming-with-go-
language-essential-skills-for-programming-using-and-securing-
networks-with-open-source-google-golang-2nd-edition-jan-newmarch/
https://ebookmeta.com/product/network-programming-with-go-
language-2nd-edition-ronald-petty/
https://ebookmeta.com/product/software-development-with-go-cloud-
native-programming-using-golang-with-linux-and-docker-nanik-
tolaram/
https://ebookmeta.com/product/salesforce-platform-governance-
method-a-guide-to-governing-changes-development-and-enhancements-
on-the-salesforce-platform-1st-edition-lee-harding-2/
Legal studies for VCE, access and justice 14th Edition
Lisa Filippin
https://ebookmeta.com/product/legal-studies-for-vce-access-and-
justice-14th-edition-lisa-filippin/
https://ebookmeta.com/product/a-beginners-guide-to-paper-
wildflowers-learn-to-make-43-beautiful-paper-flowers-
over-250-full-size-templates-emiko-yamamoto/
https://ebookmeta.com/product/overcoming-addiction-an-islamic-
approach-to-recovery-1st-edition-tayba-foundation/
https://ebookmeta.com/product/all-i-want-for-christmas-1st-
edition-poppy-parkes/
https://ebookmeta.com/product/finding-pandora-the-complete-
collection-2nd-edition-e-rachael-hardcastle/
Double Crossed A Review of the Most Extreme Exercise
Program 2nd Edition Dr Sean M Wells
https://ebookmeta.com/product/double-crossed-a-review-of-the-
most-extreme-exercise-program-2nd-edition-dr-sean-m-wells/
Jan Newmarch and Ronald Petty
Network Programming with Go
Language
Essential Skills for Programming, Using and
Securing Networks with Open Source Google
Golang
2nd ed.
Dr. Jan Newmarch
Oakleigh, VIC, Australia
Ronald Petty
San Francisco, CA, USA
The publisher, the authors and the editors are safe to assume that the
advice and information in this book are believed to be true and accurate
at the date of publication. Neither the publisher nor the authors or the
editors give a warranty, expressed or implied, with respect to the
material contained herein or for any errors or omissions that may have
been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.
This Apress imprint is published by the registered company APress
Media, LLC, part of Springer Nature.
The registered company address is: 1 New York Plaza, New York, NY
10004, U.S.A.
I dedicate this to my family.
Preface to the Second Edition
While an age has passed in Internet years, Go remains a primary
destination for programmers. Go conquered the container technology
space. It continues to find affection in Cloud Native development. Go
strives to remain true to itself, backward compatible, yet adding new
language features like Generics. Tooling improvements such as Fuzzing
allow for more secure application development.
Go has changed, and so has this book. The first edition used Go 1.8;
we are now on Go 1.18. The code has been updated to reflect this new
reality. The examples have been largely developed to show a particular
feature of Go networking without forcing complexity like managing
several projects or packages scattered across the book; the associated
repository can be found here
https://github.com/Apress/network-prog-with-go-2e.
The first version of this book assumed familiarity with Go, and that
remains in this edition. We expand slightly what we are willing to
discuss in this book with the inclusion of more third-party modules,
tools, and techniques. Jan was correct to keep the focus on Go and not
to be distracted with the ecosystem at large.
If you desire to learn about implementing networking concepts with
Go, I hope this book serves you well.
As a point of comparison, what follows is Jan’s original preface,
reflecting Go in 2017.
Preface to the First Edition
It’s always fun to learn a new programming language, especially when it
turns out to be a major one. Prior to the release of Go in 2009, I was
teaching a Master’s level subject in network programming at Monash
University. It’s good to have a goal when learning a new language, but
this time, instead of building yet another wine cellar program, I decided
to orient my lecture notes around Go instead of my (then) standard
delivery vehicle of Java.
The experiment worked well: apart from the richness of the Java
libraries that Go was yet to match, all the programming examples
transferred remarkably well, and in many cases were more elegant than
the original Java programs.
This book is the result. I have updated it as Go has evolved and as
new technologies such as HTTP/2 have arisen. But if it reads like a
textbook, well, that is because it is one. There is a large body of
theoretical and practical concepts involved in network programming
and this book covers some of these as well as the practicalities of
building systems in Go.
In terms of language popularity, Go is clearly rising. It has climbed
to 16th in the TIOBE index, is 18th in the PYPL (Popularity of
Programming Language), and is 15th in the RedMonk Programming
Language rankings. It is generally rated as one of the fastest growing
languages.
There is a growing community of developers both of the core
language and libraries and of the independent projects. I have tried to
limit the scope of this book to the standard libraries only and to the
“sub-repositories” of the Go tree. While this eliminates many excellent
projects that no doubt make many programming tasks easier,
restricting the book to the official Go libraries provides a clear bound.
This book assumes a basic knowledge of Go. The focus is on using
Go to build network applications, not on the basics of the language.
Network applications are different than command-line applications, are
different than applications with a graphical user interface, and so on. So
the first chapter discusses architectural aspects of network programs.
The second chapter is an overview of the features of Go that we use in
this book. The third chapter on sockets covers the Go version of the
basics underlying all TCP/IP systems. Chapters 4, 5, and 6 are more
unusual in network programming books. They cover the topics of what
representations of data will be used, how a network interaction will
proceed, and for text, which language formats are used. Then in Chapter
7, we look at the increasingly important topic of security. In Chapter 8,
we look at one of the most common application layer protocols in use,
HTTP. The next four chapters are about topics related to HTTP and
common data formats carried above HTTP – HTML and XML. In
Chapter 13, we look at an alternative approach to network
programming, remote procedure calls. Chapters 14 and 15 consider
further aspects of network programming using HTTP.
Any source code or other supplementary material referenced by the
author in this book is available to readers on GitHub. For more detailed
information, please visit
https://github.com/Apress/network-prog-with-go-2e.
Acknowledgments
I want to share my appreciation for Jan Newmarch for collaborating on
this book. This project has offered me a tremendous sense of
achievement and allowed me to cross a much-anticipated item off my
bucket list. I would also like to thank Eldon Alameda for his thoughtful
approach at letting me know when I am off the mark and for providing
me with solid advice.
Additionally, I owe gratitude to my partners at Apress, both Steve
Anglin for the opportunity and Mark Powers for the guidance to help
see this through. Thank you to my colleagues at RX-M, including Randy
Abernethy, Christopher Hanson, Andrew Bassett, and Anita Wu. Our
work over the years has allowed for my participation in a project such
as this book.
Finally, I want to thank my wife Julie and daughter Charlotte. Julie’s
capacity to manage the world while I hide out on a computer is
unmatched and most appreciated. Charlotte’s energy, abilities, and
creativity inspire me to become better every day.
—Ronald Petty
Table of Contents
Chapter 1:Architectural Layers
Protocol Layers
ISO OSI Protocol
OSI Layers
TCP/IP Protocol
Some Alternative Protocols
Networking
Gateways
Host-Level Networking
Packet Encapsulation
Connection Models
Connection Oriented
Connectionless
Communications Models
Message Passing
Remote Procedure Call
Distributed Computing Models
Client-Server System
Client-Server Application
Server Distribution
Communication Flows
Synchronous Communication
Asynchronous Communication
Streaming Communication
Publish/Subscribe
Component Distribution
Gartner Classification
Three-Tier Models
Fat vs.Thin
Middleware Model
Middleware Examples
Middleware Functions
Continuum of Processing
Points of Failure
Acceptance Factors
Thoughts on Distributed Computing
Transparency
Access Transparency
Location Transparency
Migration Transparency
Replication Transparency
Concurrency Transparency
Scalability Transparency
Performance Transparency
Failure Transparency
Eight Fallacies of Distributed Computing
Fallacy: The Network Is Reliable
Fallacy: Latency Is Zero
Fallacy: Bandwidth Is Infinite
Fallacy: The Network Is Secure
Fallacy: Topology Doesn’t Change
Fallacy: There Is One Administrator
Fallacy: Transport Cost Is Zero
Fallacy: The Network Is Homogeneous
Conclusion
Chapter 2:Overview of the Go Language
Types
Slices and Arrays
Maps
Pointers
Functions
Structures
Methods
Multithreading
Packages
Modules
Type Conversion
Statements
GOPATH
Running Go Programs
Standard Libraries
Error Values
Conclusion
Chapter 3:Socket-Level Programming
The TCP/IP Stack
IP Datagrams
UDP
TCP
Internet Addresses
IPv4 Addresses
IPv6 Addresses
IP Address Type
Using Available Documentation and Examples
The IPMask Type
Basic Routing
The IPAddr Type
Host Canonical Name and Addresses Lookup
Services
Ports
The TCPAddr Type
TCP Sockets
TCP Client
A Daytime Server
Multithreaded Server
Controlling TCP Connections
Timeout
Staying Alive
UDP Datagrams
Server Listening on Multiple Sockets
The Conn, PacketConn, and Listener Types
Raw Sockets and the IPConn Type
Conclusion
Chapter 4:Data Serialization
Structured Data
Mutual Agreement
Self-Describing Data
Encoding Packages
ASN.1
ASN.1 Daytime Client and Server
JSON
A Client and A Server
The Gob Package
A Client and A Server
Encoding Binary Data As Strings
Protocol Buffers
Installing and Compiling Protocol Buffers
The Generated personv3.pb.go File
Using the Generated Code
Conclusion
Chapter 5:Application-Level Protocols
Protocol Design
Why Should You Worry?
Version Control
The Web
Message Format
Data Format
Byte Format
Character Format
A Simple Example
A Stand-Alone Application
The Client-Server Application
The Client Side
Alternative Presentation Aspects
The Server Side
Protocol:Informal
Text Protocol
Server Code
Client Code
Textproto Package
State Information
Application State Transition Diagram
Client-State Transition Diagrams
Server-State Transition Diagrams
Server Pseudocode
Conclusion
Chapter 6:Managing Character Sets and Encodings
Definitions
Character
Character Repertoire/Character Set
Character Code
Character Encoding
Transport Encoding
ASCII
ISO 8859
Unicode
UTF-8, Go, and Runes
UTF-8 Client and Server
ASCII Client and Server
UTF-16 and Go
Little-Endian and Big-Endian
UTF-16 Client and Server
Unicode Gotchas
ISO 8859 and Go
Other Character Sets and Go
Conclusion
Chapter 7:Security
ISO Security Architecture
Functions and Levels
Mechanisms
Data Integrity
Symmetric Key Encryption
Public Key Encryption
X.509 Certificates
TLS
A Basic Client
Server Using a Self-Signed Certificate
Conclusion
Chapter 8:HTTP
URLs and Resources
i18n
HTTP Characteristics
Versions
HTTP/0.9
HTTP/1.0
HTTP 1.1
HTTP Major Upgrades
HTTP/2
HTTP/3
Simple User Agents
The Response Type
The HEAD Method
The GET Method
Configuring HTTP Requests
The Client Object
Proxy Handling
Simple Proxy
Authenticating Proxy
HTTPS Connections by Clients
Servers
File Server
Handler Functions
Bypassing the Default Multiplexer
HTTPS
Conclusion
Chapter 9:Templates
Inserting Object Values
Using Templates
Pipelines
Defining Functions
Variables
Conditional Statements
The html/template Package
Conclusion
Chapter 10:A Complete Web Server
Browser Site Diagram
Browser Files
Basic Server
The listFlashCards Function
The manageFlashCardsFunction
The Chinese Dictionary
The Dictionary Type
Flashcard Sets
Fixing Accents
The ListWords Function
The showFlashCards Function
Presentation on the Browser
Running the Server
Conclusion
Chapter 11:HTML
The html/template Package
Tokenizing HTML
XHTML/HTML
JSON
Conclusion
Chapter 12:XML
Unmarshalling XML
Marshalling XML
Parsing XML
The StartElement Type
The EndElement Type
The CharData Type
The Comment Type
The ProcInst Type
The Directive Type
XHTML
HTML
Conclusion
Chapter 13:Remote Procedure Call
Go’s RPC
HTTP RPC Server
HTTP RPC Client
TCP RPC Server
TCP RPC Client
Matching Values
JSON
JSON RPC Server
JSON RPC Client
Conclusion
Chapter 14:REST
URIs and Resources
Representations
REST Verbs
The GET Verb
The PUT Verb
The DELETE Verb
The POST Verb
No Maintained State (That Is, Stateless)
HATEOAS
Representing Links
Transactions with REST
The Richardson Maturity Model
Flashcards Revisited
URLs
ServeMux (The Demultiplexer)
Content Negotiation
GET /
POST /
Handling Other URLs
The Complete Server
Client
Using REST or RPC
Conclusion
Chapter 15:WebSockets
WebSockets Server
The golang.org/x/net/websocket Package
The Message Object
The JSON Object
The Codec Type
WebSockets over TLS
WebSockets in an HTML Page
The github.com/gorilla/websocket Package
Echo Server
Echo Client
Conclusion
Chapter 16:Gorilla
Middleware Pattern
Standard Library ServeMux Examples
Customizing Muxes
gorilla/mux
Why Should We Care
Gorilla Handlers
Additional Gorilla Examples
gorilla/rpc
gorilla/schema
gorilla/securecookie
Conclusion
Chapter 17:Testing
Simple and Broken
httptest Package
Below HTTP
Leveraging the Standard Library
Conclusion
Appendix A:Fuzzing
Fuzzing in Go
Fuzzing Failures
Conclusion
Appendix B:Generics
A Filtering Function Without Generics
Refactor Using Generics
Custom Constraints
Using Generics on Collections
How Not to Use Generics?
Conclusion
Index
About the Authors
Jan Newmarch
was head of ICT (higher education) at
Box Hill Institute before retiring, and still
is adjunct professor at Canberra
University, and adjunct lecturer in the
School of Information Technology,
Computing and Mathematics at Charles
Sturt University. He is interested in more
aspects of computing than he has time to
pursue, but the major thrust over the last
few years has developed from user
interfaces under UNIX into Java, the Web,
and then into general distributed
systems. Jan developed a number of
publicly available software systems in these areas. For the last few
years, he had been looking at sound for Linux systems and
programming the Raspberry Pi’s GPU. He is now exploring aspects of
the IoT and Cyber Security. He lives in Melbourne, Australia, and enjoys
the food and culture there, but is not so impressed by the weather.
OSI Layers
The function of each layer from bottom to top is as follows:
The Physical layer conveys the bit stream using electrical, optical, or
radio technologies.
The Data Link layer puts the information packets into network
frames for transmission across the Physical layer and back into
information packets.
The Network layer provides switching and routing technologies.
The Transport layer provides transparent transfer of data between
end systems and is responsible for end-to-end error recovery and
flow control.
The Session layer establishes, manages, and terminates connections
between applications.
The Presentation layer provides independence from differences in
data representation (e.g., encryption).
The Application layer supports application and end-user processes.
A layer in the OSI model often maps to a modern protocol; for
example, the IP from TCP/IP maps to the Network layer, also known as
layer 3 (Physical is layer 1). The Application layer, layer 7, maps to
HTTP. Some protocols like HTTPS seem to blend layers, 5 (Session) and
6 (Presentation). No model is perfect; alternatives exist to the OSI
model that maps closer to a given reality, such as the TCP/IP protocol
model.
TCP/IP Protocol
While the OSI model was being argued, debated, partly implemented,
and fought over, the DARPA Internet research project was busy building
the TCP/IP protocols. These have been immensely successful and have
led to The Internet (with capitals). This is a much simpler stack, as
shown in Figure 1-2.
Connection Oriented
A single connection is established for the session. Two-way
communications flow along the connection. When the session is over,
the connection is broken. The analogy is to a phone conversation. An
example is TCP.
Connectionless
In a connectionless system, messages are sent independent of each
other. Ordinary mail is the analogy. Connectionless messages may
arrive out of order. Messages do not have an impact on each other. An
example is the IP protocol. UDP is a connectionless protocol above IP
and is often used as an alternative to TCP, as it is much lighter weight.
Connectionless is also known as unconnected or stateless.
Connection-oriented transports may be established on top of
connectionless ones – TCP over IP. Connectionless transports may be
established on top of connection-oriented ones – HTTP over TCP.
Messages over a connection-oriented transport protocol have some
kind of relation, for example, a sequence number used to keep order.
Having state allows for functionality and optimizations; it also has an
associated cost of storage and computing.
There can be variations on these. For example, a session might
enforce messages arriving but might not guarantee that they arrive in
the order sent. However, these two are the most common.
Connection models are not the only way a protocol can vary. One
often desired feature is reliability; this is where the protocol has logic to
fix some types of errors; for example, TCP resends a missing packet. It’s
pretty common to assume connection-oriented protocols are reliable;
this is not always the case (e.g., MPLS). Additional features of a network
protocol could include message boundary management, delivery
ordering, error checking, flow control, etc. These features can exist in
one protocol layer and not another, partly why there are so many
network protocol stacks.
Sometimes, these features are reworked throughout the protocol
stack. An example of this kind of feature rework is with HTTP/3. In
HTTP/2, reliability is provided using TCP at layer 4. In HTTP/3, TCP is
being replaced with UDP, which is not reliable. Instead, reliability will
be provided with another protocol known as QUIC. While QUIC is
considered a Transport layer, like TCP or UDP, it works on top of UDP.
As you can see, the layer model is not an exact science.
Communications Models
In a distributed system, there will be many components (i.e., processes)
running that have to communicate with each other. There are two
primary models for this, message passing and remote procedure calls.
In the context of networking, these models allow interprocess (and/or
thread) communication with intent to invoke behavior on the remote
process.
Message Passing
Some languages are built on the principle of message passing.
Concurrent languages (and tools) often use such a mechanism, and the
most well-known example is probably the UNIX pipeline. The UNIX
pipeline is a pipeline of bytes, but this is not an inherent limitation:
Microsoft’s PowerShell can send objects along its pipelines, and
concurrent languages such as Parlog can send arbitrary logic data
structures in messages between concurrent processes. Recent
languages such as Go have mechanisms for message passing (between
threads).
Message passing is a primitive mechanism for distributed systems.
Set up a connection and pump some data down it. At the other end,
figure out what the message was and respond to it, possibly sending
messages back. This is illustrated in Figure 1-4.
Figure 1-4 The message passing communications model
Event-driven systems act in a similar manner. At a low level, the
programming language node.js runs an event loop waiting for I/O
events, dispatching handlers for these events and responding. At a
higher level, most user interface systems use an event loop waiting for
user input, while in the networking world, Ajax uses the
XMLHttpRequest to send and receive requests.
Minka lennähti ylös. Nyt hän ei purrut, vaan kietoi kätensä Barbron
kaulaan.
"Te olette sen näköinen, kuin olisi teillä syli täysi riemua", sanoi
tohtori.
"Onko se ainoastaan?"
"J.P. Jacobsen?"
"Jäättekö Tukholmaan?"
"Niin mainion iloiseksi, sillä minä saisin kuulla juuri sitä, mikä on
minun mieleeni."
"Ei, ei, mutta jos minä laulan, niin se tapahtuu teidän tähtenne,
Barbro, siksi että te olette se kevään ihana, heleä hedelmäkukka,
joka ei ole vuosikausiin tuoksunut minun ylettyvilläni. 'Olen lempinyt,
elänyt ja laulanut lauluani', kuten sanotaan. Siitä on jo aikoja. Jos
otan luutun taas käteeni, niin se johtuu siitä, että uusi kevät on
kosketellut sen kieliä."
Lucia-juhla.
Odottamaton kohtaus.
Minka oli kenties se, joka oli muuttunut enimmin. Hän ei ollut
ainoastaan oppinut puhumaan melkein täydellisesti ruotsia, vaan
myöskin koko hänen olennostaan oli karissut paljon muukalaisuutta.
Koreat valekivi-helyt ja kiiltolanka-käädyt hän oli pannut pois, sillä oli
olemassa "eräs", joka piti hänestä enemmän ilman niitä. Työtä ja
järjestystä hän vieroi samoin kuin ennenkin, mutta joskus pisti
hänen päähänsä yrittää.
"Onpa siinä hauskat kannet", sanoi Maud, kun Barbro oli pistänyt
kirjan nopeasti pois, minkä Maudin terävä silmä kuitenkin keksi.
"Mikä ukko?"
"Johan nyt jotakin", puhkesi Maud sanomaan, "et kai sinä ole niin
hupsu, että välität mokomasta 'le petit dieu' roskasta?" Hän
mätkäytti lahjakirjan pöydälle, istahti tuolille, heilautti toisen jalan
ristiin toisen päälle, ojensi käsivartensa ikäänkuin luutuksi, jota oli
soittavinaan, ja alkoi laulaa:
"Ce que femme veut, Dieu le veut. Minä olen tuottanut itselleni
kavaljeerin kotoa. Taikka oikeastaan hän tulee näinä päivinä. Hän on
nuorin veljeni, somin poika, minkä tunnen."
Kun hän nyt näki tuommoisen, ehkä liiankin pyylevän niskan, niin
tuntui kuin hänen aurinkoinen elämänilonsa olisi alkanut kurkistaa
harmaiden ajatuspilvien lomasta. Ja niin pian kuin hissi oli ehtinyt
ylös, koetti hän ottaa selville, pettikö niska hänet vai ei.
"Valitsenko minä?"