Go in Practice

Copyright

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact

       Special Sales Department

       Manning Publications Co.

       20 Baldwin Road

       PO Box 761

       Shelter Island, NY 11964

       Email: 

orders@manning.com

©2016 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.

Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.

Development editor: Susanna Kline

Technical development editors: Ivan Kirkpatrick, Kim Shrier,

Glenn Burnside, Alain Couniot

Review editor: Aleksandar Dragosavljevic

Project editor: Karen Gulliver

Copy editor: Sharon Wilkey

Proofreader: Melody Dolab

Technical Proofreader: James Frasché

Typesetter: Dottie Marsico

Cover designer: Marija Tudor

ISBN 9781633430075

Printed in the United States of America

1 2 3 4 5 6 7 8 9 10 – EBM – 21 20 19 18 17 16

Brief Table of Contents

Copyright

Brief Table of Contents

Table of Contents

Foreword

Preface

Acknowledgments

About this Book

About the Authors

About the Cover Illustration

1. Background and fundamentals

Chapter 1. Getting into Go

Chapter 2. A solid foundation

Chapter 3. Concurrency in Go

2. Well-rounded applications

Chapter 4. Handling errors and panics

Chapter 5. Debugging and testing

3. An interface for your applications

Chapter 6. HTML and email template patterns

Chapter 7. Serving and receiving assets and forms

Chapter 8. Working with web services

4. Taking your applications to the cloud

Chapter 9. Using the cloud

Chapter 10. Communication between cloud services

Chapter 11. Reflection and code generation

Index

List of Figures

List of Listings

Table of Contents

Copyright

Brief Table of Contents

Table of Contents

Foreword

Preface

Acknowledgments

About this Book

About the Authors

About the Cover Illustration

1. Background and fundamentals

Chapter 1. Getting into Go

1.1. What is Go?

1.2. Noteworthy aspects of Go

1.2.1. Multiple return values

1.2.2. A modern standard library

1.2.3. Concurrency with goroutines and channels

1.2.4. Go the toolchain—more than a language

1.3. Go in the vast language landscape

1.3.1. C and Go

1.3.2. Java and Go

1.3.3. Python, PHP, and Go

1.3.4. JavaScript, Node.js, and Go

1.4. Getting up and running in Go

1.4.1. Installing Go

1.4.2. Working with Git, Mercurial, and version control

1.4.3. Exploring the workspace

1.4.4. Working with environment variables

1.5. Hello, Go

1.6. Summary

Chapter 2. A solid foundation

2.1. Working with CLI applications, the Go way

2.1.1. Command-line flags

2.1.2. Command-line frameworks

2.2. Handling configuration

Technique 3 Using configuration files

Technique 4 Configuration via environment variables

2.3. Working with real-world web servers

2.3.1. Starting up and shutting down a server

2.3.2. Routing web requests

2.4. Summary

Chapter 3. Concurrency in Go

3.1. Understanding Go’s concurrency model

3.2. Working with goroutines

Technique 10 Using goroutine closures

Technique 11 Waiting for goroutines

Technique 12 Locking with a mutex

3.3. Working with channels

Technique 13 Using multiple channels

Technique 14 Closing channels

Technique 15 Locking with buffered channels

3.4. Summary

2. Well-rounded applications

Chapter 4. Handling errors and panics

4.1. Error handling

Technique 16 Minimize the nils

Technique 17 Custom error types

Technique 18 Error variables

4.2. The panic system

4.2.1. Differentiating panics from errors

4.2.2. Working with panics

4.2.3. Recovering from panics

4.2.4. Panics and goroutines

4.3. Summary

Chapter 5. Debugging and testing

5.1. Locating bugs

5.1.1. Wait, where is my debugger?

5.2. Logging

5.2.1. Using Go’s logger

5.2.2. Working with system loggers

5.3. Accessing stack traces

Technique 26 Capturing stack traces

5.4. Testing

5.4.1. Unit testing

5.4.2. Generative testing

5.5. Using performance tests and benchmarks

Technique 29 Benchmarking Go code

Technique 30 Parallel benchmarks

Technique 31 Detecting race conditions

5.6. Summary

3. An interface for your applications

Chapter 6. HTML and email template patterns

6.1. Working with HTML templates

6.1.1. Standard library HTML package overview

6.1.2. Adding functionality inside templates

6.1.3. Limiting template parsing

6.1.4. When template execution breaks

6.1.5. Mixing templates

6.2. Using templates for email

Technique 38 Generating email from templates

6.3. Summary

Chapter 7. Serving and receiving assets and forms

7.1. Serving static content

Technique 39 Serving subdirectories

Technique 40 File server with custom error pages

Technique 41 Caching file server

Technique 42 Embedding files in a binary

Technique 43 Serving from an alternative location

7.2. Handling form posts

7.2.1. Introduction to form requests

7.2.2. Working with files and multipart submissions

7.2.3. Working with raw multipart data

7.3. Summary

Chapter 8. Working with web services

8.1. Using REST APIs

8.1.1. Using the HTTP client

8.1.2. When faults happen

8.2. Passing and handling errors over HTTP

8.2.1. Generating custom errors

8.2.2. Reading and using custom errors

8.3. Parsing and mapping JSON

Technique 53 Parsing JSON without knowing the schema

8.4. Versioning REST APIs

Technique 54 API version in the URL

Technique 55 API version in content type

8.5. Summary

4. Taking your applications to the cloud

Chapter 9. Using the cloud

9.1. What is cloud computing?

9.1.1. The types of cloud computing

9.1.2. Containers and cloud-native applications

9.2. Managing cloud services

9.2.1. Avoiding cloud provider lock-in

9.2.2. Dealing with divergent errors

9.3. Running on cloud servers

9.3.1. Performing runtime detection

9.3.2. Building for the cloud

9.3.3. Performing runtime monitoring

9.4. Summary

Chapter 10. Communication between cloud services

10.1. Microservices and high availability

10.2. Communicating between services

10.2.1. Making REST faster

10.2.2. Moving beyond REST

10.3. Summary

Chapter 11. Reflection and code generation

11.1. Three features of reflection

Technique 66 Switching based on type and kind

Technique 67 Discovering whether a value implements an interface

Technique 68 Accessing fields on a struct

11.2. Structs, tags, and annotations

11.2.1. Annotating structs

11.2.2. Using tag annotations

11.3. Generating Go code with Go code

Technique 70 Generating code with go generate

11.4. Summary

Index

List of Figures

List of Listings

Foreword

When I heard that Matt Farina and Matt Butcher were starting a new book on Go, I was excited. Both have been key contributors in the Go ecosystem for years, and have extensive work experience and backgrounds that flavor the prose in this book with the spice of past learnings. The book is intended as a spiritual successor to Go in Action, taking you beyond the basics that we introduced there and into more practical learning.

The book is broken into four easily digestible parts, each with a different focus. Part 1 is a refresher on key Go concepts. If you’re in a hurry and comfortable with your Go skills, you can safely skip this section, but I discourage that. In reviewing the final manuscript, I found nuggets of such value that I think everyone would benefit from these chapters.

Part 2 dives into the mechanics of managing a Go application in the real world. The chapter on errors is one of the best treatises on Go errors I’ve ever read, and the chapter on debugging and testing provides useful information on that crucial middle step of application development that takes your application from proof of concept to reliable production system.

In part 3, you’ll learn about ways to create user interfaces for your application. The chapter on templates is an excellent guide to what many find to be a complicated part of Go’s ecosystem. You’ll see practical ways to reuse your templates and make your web interfaces more dry. The examples alone are worth the price of the book, as it’s difficult to find examples of template usage that can be easily mapped to a real-world application. Later, you’ll see how to create and consume a standards-compliant REST API and learn the tricks to properly versioning that API.

The final section of the book moves into the interoperability layer that’s required in nearly every application today. You’ll dive deep into cloud infrastructure and see where Go fits in the cloud-computing model. You’ll finish with great coverage of microservices and service-to-service communication patterns.

Whether you’re just coming to Go or you’ve been writing Go applications for years, this book has vital knowledge that will help you take your application development to the next level. The authors do a great job of presenting complex information with a unified voice and in a manner that’s easy to digest. I’m excited for the publication of this book and the value that it brings to the Go community. I hope that you’ll enjoy reading it as much as I have.

—BRIAN KETELSEN

CO-AUTHOR OF GO IN ACTION

CO-FOUNDER OF GOPHER ACADEMY

Preface

When we first started using Go, we saw a language with a lot of potential. We wanted to build applications with it. But it was a new language, and many companies are wary of introducing a new programming language.

This is especially true in the enterprise, where Go has the potential to have a huge impact. New languages are challenged to be trusted, accepted, and adopted. There are hundreds of thousands of developers in businesses where leaders need to be swayed to try a new language and developers need to learn it well enough to build applications and see a benefit.

Open source projects, conferences, training, and books all help to make a programming language more palatable. We wanted to write a book that teaches Go in order to help the Go community, help those trying to learn Go or to convince their organizations’ leadership, and help us in the companies that we work for and with.

When we first started the book, it was targeted squarely at cloud development with Go. Go is a language built for the cloud, and we’ve spent years working in cloud computing. Once we started working with Manning Publications, we saw an opportunity to expand beyond the cloud, into more useful and helpful patterns. And so the book shifted from being cloud-focused to pattern-focused. Yet it still retains its cloud roots.

Go in Practice is our attempt to help developers move from being familiar with the language to being productive with it. To help the community of developers grow, while helping organizations write better software.

Acknowledgments

We’ve spent about two years writing this book, but none of the effort would have been possible without the commitment of our families. They’ve supported us through the early mornings, late nights, and weekends when we were focused on writing. They were there as we were fixated on solving problems, even when we weren’t sitting down to write.

Good code is never created in a vacuum. We’re also grateful to the women and men of the Go community who have so generously given their time to create a great language, great libraries, and a thriving ecosystem. It has been exciting to be a part of such a diverse, burgeoning community of developers. In particular, Rob Pike, Brian Ketelsen, and Dave Cheney all reached out to us early in our Go learning process. They’re admirable ambassadors of the language. Special thanks to Brian for contributing the foreword to the book and for endorsing our work.

We appreciate the many individuals who gave time and effort to the creation of this book. It has been an arduous process, and thanks to many careful readers, including our MEAP readers, we found and corrected numerous mistakes.

We’d like to thank everyone at Manning, especially our development editor, Susanna Kline; our technical development editors, Ivan Kirkpatrick, Kim Shrier, Glenn Burnside, and Alain Couniot; and our technical proofreader, James Frasché; as well as everyone who worked on our book behind the scenes. Thanks also to the many reviewers who took the time to read our manuscript at various stages of its development and who provided invaluable feedback: Anthony Cramp, Austin Riendeau, Brandon Titus, Doug Sparling, Ferdinando Santacroce, Gary A. Stafford, Jim Amrhein, Kevin Martin, Nathan Davies, Quintin Smith, Sam Zaydel, and Wes Shaddix.

Finally, we owe a debt of gratitude to the Glide community, which has grown with us as we worked to build a top-tier package manager for Go. Thank you for your support.

Matt Butcher

I began writing this book at Revolv, continued when Google/Nest acquired us, and finished at Deis. Thanks to all three for supporting the writing of this book. Thanks to Brian Hardock, Cristian Cavalli, Lann Martin, and Chris Ching, all of whom served as early sounding boards. Matt Boersma provided helpful feedback for several chapters. Kent Rancourt and Aaron Schlesinger each inspired particular code examples in this book. Matt Fisher, Sivaram Mothiki, Keerthan Mala, Helgi Þorbjörnsson (yes, Helgi, I copied and pasted that), Gabe Monroy, Chris Armstrong, Sam Boyer, Jeff Bleiel, Joshua Anderson, Rimas Mocevicius, Jack Francis, and Josh Lane all (wittingly or unwittingly) influenced specific portions of this book. The impact of Michelle Noorali and Adam Reese cannot be understated; I’ve learned a lot watching a couple of Ruby developers master Go. And thanks to Angie, Annabelle, Claire, and Katherine for their unflagging support and understanding.

Matt Farina

I would like to thank Kristin, my beautiful and amazing wife, along with our wonderful daughters, Isabella and Aubrey, for their love and support.

I wrote this book while working at Hewlett Packard Enterprise, formerly Hewlett-Packard. Working at HPE has taught me invaluable lessons while providing me with the opportunity to work alongside and learn from those far wiser than myself. Specifically, I need to thank Rajeev Pandey, Brian Aker, Steve McLellan, Erin Handgen, Eric Gustafson, Mike Hagedorn, Susan Balle, David Graves, and many others. They have affected the way I write and operate applications, and that has shown up in these chapters in subtle ways.

There have been many others who influenced portions of this book, sometimes without realizing it. Thanks to Tim Pletcher, Jason Buberel, Sam Boyer, Larry Garfield, and all those I may have forgotten who had a positive influence.

Finally, I want to thank Matt Butcher. I never imagined authoring books until you suckered me into it. Thanks!

About this Book

Go in Practice is a book about practical development using the Go programming language. Developers already familiar with the basics of Go will find patterns and techniques for creating Go applications. Chapters are organized around central themes (for example, chapter 10, Communicating between cloud services), but then explore a variety of techniques related to that theme.

How the book is organized

The 11 chapters are divided into four parts.

Part 1, Background and fundamentals, provides a foundation for building applications. Chapter 1 provides the background of Go for those not already familiar with it or those with a passing understanding who would like to learn more. Building console applications and servers is the topic of chapter 2, and concurrency in Go is the topic of chapter 3.

Part 2, Well-rounded applications, contains chapters 4 and 5. These chapters cover errors, panics, debugging, and testing. The goal of this section is to build applications you trust that handle problems well.

Part 3, An interface for your applications, contains three chapters with topics ranging from generating HTML and to serving assets to providing and working with APIs. Many Go applications provide web applications and REST APIs for interaction. These chapters cover patterns to aid in their construction.

Part 4, Taking your applications to the cloud, contains the remaining chapters, which focus on cloud computing and generating code. Go is a language built with cloud needs in mind. This section showcases patterns that enable working with those services and operating applications, sometimes as microservices, in them. It also covers generating code and metaprogramming.

There are 70 techniques explored in the book, each with its own Problem, Solution, and Discussion sections.

Code conventions and downloads

All source code in the book is presented in a mono-spaced typeface like this, which sets it off from the surrounding text. In many listings, the code is annotated to point out key concepts, and numbered bullets are sometimes used in the text to provide additional information about the code.

Source code for the examples in the book is available for download from the publisher’s website at www.manning.com/books/go-in-practice and from GitHub at github.com/Masterminds/go-in-practice.

Author Online Forum

The purchase of Go in Practice includes free access to a private web forum run by Manning Publications, where you can make comments about the book, ask technical questions, and receive help from the authors and from other users. To access the forum and subscribe to it, point your web browser to www.manning.com/books/go-in-practice. This page provides information on how to get on the forum after you’re registered, what kind of help is available, and the rules of conduct on the forum.

Manning’s commitment to our readers is to provide a venue where a meaningful dialogue between individual readers and between readers and the authors can take place. It’s not a commitment to any specific amount of participation on the part of the authors, whose contribution to the forum remains voluntary (and unpaid). We suggest you try asking the authors some challenging questions lest their interest stray!

The Author Online forum and the archives of previous discussions will be accessible from the publisher’s website as long as the book is in print.

About the Authors

MATT BUTCHER is an architect at Deis, where contributing to open source projects is his day job. He has written several books and dozens of articles. Matt holds a PhD in philosophy and teaches in the Computer Science department at Loyola University Chicago. Matt is passionate about building strong teams and developing elegant solutions to complex problems.

MATT FARINA is a Principal Engineer in the Advanced Technology Group at Hewlett Packard Enterprise. He is an author, speaker, and regular contributor to open source software who has been developing software for over a quarter century. He likes to solve problems for regular people by creating solutions using both the latest technology and the mundane that can be easily overlooked.

About the Cover Illustration

The figure on the cover of Go in Practice is captioned Habit of the Wife of a Russian Peasant in 1768. The illustration is taken from Thomas Jefferys’ A Collection of the Dresses of Different Nations, Ancient and Modern, published in London between 1757 and 1772. The title page states that these are hand-colored copperplate engravings, heightened with gum arabic.

Thomas Jefferys (1719–1771) was called Geographer to King George III. An English cartographer, he was the leading map supplier of his day. He engraved and printed maps for government and other official bodies and produced a wide range of commercial maps and atlases, especially of North America. His work as a map maker sparked an interest in local dress customs of the lands he surveyed and mapped, and these are brilliantly displayed in his four-volume collection.

Fascination with faraway lands and travel for pleasure were relatively new phenomena in the late 18th century, and collections such as this one were popular, introducing the tourist as well as the armchair traveler to the inhabitants of other countries. The diversity of the drawings in Jefferys’ volumes speaks vividly of the uniqueness and individuality of the world’s nations some 200 years ago. Dress codes have changed since then, and the diversity by region and country, so rich at the time, has faded away. It’s now often hard to tell the inhabitant of one continent from another. Perhaps, trying to view it optimistically, we’ve traded a cultural and visual diversity for a more varied personal life, or a more varied and interesting intellectual and technical life.

At a time when it’s hard to tell one computer book from another, Manning celebrates the inventiveness and initiative of the computer business with book covers based on the rich diversity of regional life of two centuries ago, brought back to life by Jefferys’ pictures.

Part 1. Background and fundamentals

This opening part of the book provides some background about Go and a foundation for building applications. Chapter 1 starts with an overview of Go for those not already familiar with it.

Chapters 2 and 3 move into base components for an application. Chapter 2 provides the foundation for building an application, including working with console applications and servers, and handling configuration. Chapter 3 focuses on using goroutines. Goroutines are one of the more powerful and useful elements in Go. They’re regularly used in Go applications, and you’ll see them through the rest of this book.

Chapter 1. Getting into Go

This chapter covers

Introducing Go

Understanding where Go fits into the language landscape

Getting up and running in Go

The way we build and run software is changing. Innovation has swept in, disrupting long-standing assumptions about the computing environments that software runs in. To fully take advantage of these innovations, you need languages and tools that support them at their core.

When most mainstream programming languages and supporting toolchains were developed, they were designed for single-core processing. That’s what we had. Now desktop computers, servers, and even our phones have processors with multiple cores. Running software with operations taking place concurrently can happen anywhere.

Toolchains around building applications have changed. Increased functionality and complexity in software requires environments that can build and execute the code rapidly and efficiently. Testing larger and more complicated codebases needs to happen quickly so it doesn’t become a development blocker. Many applications are developed using libraries. Libraries and their versions are managed differently, thanks to solutions to disk-space problems that hampered this in the past.

The way infrastructure and software are delivered has changed. Using colocated servers, managing your own hardware, or getting simple virtual private servers used to be the norm. Standing up a service at scale often meant you needed an investment in running your own hardware, including load balancers, servers, and storage. Getting everything ordered, assembled, and connected to the world would take weeks or months. Now it’s available in a matter of seconds or minutes via the cloud.

This chapter introduces the Go programming language for those not already familiar with it. In this chapter, you’ll learn about the language, the toolchain that accompanies it, where Go fits into the landscape of languages, and how to install Go and get it running.

1.1. What is Go?

Go, sometimes referred to as golang to make it easier to find on the web, is a statically typed and compiled open source programming language initially developed by Google. Robert Griesemer, Rob Pike, and Ken Thompson were attempting to create a language for modern systems programming that solved real-world problems they encountered while building large systems at scale.

Instead of attempting to attain theoretical pureness, these designers engineered Go around real-world practical situations. It’s inspired by a host of languages that came before it, including C, Pascal, Smalltalk, Newsqueak, C#, JavaScript, Python, Java, and many others.

Go isn’t the typical statically typed and compiled language. The static typing has features that make it feel dynamic, and the compiled binaries have a runtime that includes garbage collection. The design of the language took into account the types of projects that Google would need to use it for: large codebases operating at scale and being developed by large developer teams.

At its core, Go is a programming language defined by a specification that can be implemented by any compiler. The default implementation is shipped via the go tool. But Go is more than a programming language. As figure 1.1 illustrates, layers are built on top of the language.

Figure 1.1. The layers of Go

Developing applications requires more than a programming language—for example, testing, documentation, and formatting. Each of these areas needs tools to support it. The go tool that’s used to compile applications also provides functionality to support these elements. It’s a toolchain for application development. One of the most notable aspects of the toolchain is package management. Out of the box, the programming language Go and the go toolchain provide for packages. A built-in package system, along with a common toolchain for the essential elements of development, has enabled an ecosystem to form around the programming language.

One of the defining characteristics of Go is its simplicity. When Griesemer, Pike, and Thompson were originally designing the language, a feature didn’t go in until all three agreed that it should be a feature of the language. This style of decision-making, along with their years of experience, led to a simple but powerful language. It’s simple enough to keep in your head yet powerful enough to write a wide variety of software.

An example of this philosophy