Rust for Rustaceans: Idiomatic Programming for Experienced Developers

Rust for Rustaceans

Idiomatic Programming for Experienced Developers

by Jon Gjengset

RUST FOR RUSTACEANS. Copyright © 2022 by Jon Gjengset.

All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher.

First printing

25 24 23 22 21 1 2 3 4 5 6 7 8 9

ISBN-13: 978-1-7185-0185-0 (print)

ISBN-13: 978-1-7185-0186-7 (ebook)

Publisher: William Pollock

Managing Editor: Jill Franklin

Production Manager and Editor: Rachel Monaghan

Developmental Editor: Liz Chadwick

Cover Illustrator: James L. Barry

Interior Design: Octopod Studios

Technical Reviewer: David Tolnay

Copyeditor: Rachel Head

Compositor: Maureen Forys, Happenstance Type-O-Rama

Proofreader: Sadie Barry

For information on book distributors or translations, please contact No Starch Press, Inc. directly:

No Starch Press, Inc.

245 8th Street, San Francisco, CA 94103

phone: 1.415.863.9900; info@nostarch.com

www.nostarch.com

Library of Congress Control Number: 2021944983

No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other product and company names mentioned herein may be the trademarks of their respective owners. Rather than use a trademark symbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark.

The information in this book is distributed on an As Is basis, without warranty. While every precaution has been taken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in it.

V2

About the Author

Jon Gjengset has worked in the Rust ecosystem since the early days of Rust 1.0, and built a high-performance relational database from scratch in Rust over the course of his PhD at MIT. He’s been a frequent contributor to the Rust toolchain and ecosystem, including the asynchronous runtime Tokio, and maintains several popular Rust crates, such as hdrhistogram and inferno. Jon has been teaching Rust since 2018, when he started live-streaming intermediate-level Rust programming sessions. Since then, he’s made videos that cover advanced topics like async and await, pinning, variance, atomics, dynamic dispatch, and more, which have been received enthusiastically by the Rust community.

About the Technical Reviewer

David Tolnay is a prolific, well-known, and respected contributor in the Rust ecosystem who maintains some of the most widely used Rust libraries, including syn, serde, and anyhow. He is also a member of the Rust library team.

Foreword

Dear reader,

In the course of your experience with Rust so far, it’s likely that you have noticed a knowledge gap between what your existing learning resources have prepared you for versus what you see from the folks making the top tier of widely used Rust libraries and applications.

Libraries that do very well are commonly powered by a co-occurrence of taste and dedication on the authors’ part: feeling what to build, and building the thing (it’s that simple). This book teaches neither of those things.

However, it’s been my experience that taste emerges from a deep comfort with the fundamental pieces. It’s here I feel this book will be helpful to you. I don’t consider it a coincidence that pretty much all of the household name open source Rust library developers understand everything in this book—even when it’s not the case that they use every single thing from the book in every single library.

In this book you will find a level of nuance and tradeoffs and opinions that does not arise from introductory material. Structs are structs, and we have no need to have an opinion about structs. But infinitely flexible macro APIs (Chapter 7), the judicious application of unsafe code (Chapter 9), effective testing that speeds you up rather than slowing you down (Chapter 6)—someone who’s digested The Rust Programming Language (a.k.a. The Book, https://doc.rust-lang.org/book/) but not much beyond that is generally going to have a hard time manifesting what they know into high-quality or innovative projects, but this book takes you to the starting point to begin building your personal taste in highly polished Rust development. You will take what you read here and get it wrong a bunch of times, and get it right a couple, and get better.

I encourage you to seize upon that starting point consciously. I want you to be free to think that we got something wrong in this book; that the best current guidance in here is missing something, and that you can accomplish something over the next couple years that is better than what anybody else has envisioned. That’s how Rust and its ecosystem have gotten to this point.

David Tolnay

Preface

One of the goals listed on the Rust 2018 roadmap was to develop teaching resources to better serve intermediate Rustaceans—those who aren’t beginners but also aren’t compiler experts looking to design a new iteration of the borrow checker. That call inspired me to start live-streaming coding sessions where I implemented real systems in Rust in real time—not toy projects or long-winded introductions to basic concepts, but libraries and tools I would actually use for my research. My thinking was that Rust newcomers needed to see an experienced Rust programmer go through the whole development process, including design, debugging, and iteration, in order to understand how to think in Rust. While a beginner could attempt the same things themselves, it’d likely be far slower and frustrating since they would also be learning the language along the way.

Many developers said that my videos provided a good way to learn to use Rust for real, which was very exciting. However, over the years, it also became clear that the videos weren’t for everyone, or for every situation. Some developers prefer to be more in control over their own learning and would rather have a teaching resource they can consume at their own pace. Others just need to understand a particular topic better, or find out how a specific feature works or is best used, and for those situations, a six-hour coding video isn’t that helpful. I wanted to make sure that intermediate resources were available for those people and situations as well, which is what ultimately made me decide to write this book. My aim was to distill all that time spent teaching intermediate Rust by example into solid textual explanations of the most important intermediate topics.

I realized early on that the book would complement the videos, not replace them. I remain convinced that the best way to quickly gain experience in a language, barring actively working with it yourself daily, is to watch someone experienced use it. But in my time writing this book, I’ve also found that this format works incredibly well as a comprehensive, by-topic reference that collects lots of knowledge in one place, which is where coding videos fall terribly short. The coding sessions help develop your Rust experience, intuition, and taste. The book teaches you the theory, mechanisms, and idioms of the language. And ultimately, a developer needs all of the above to truly excel at what they do.

Now, many many words and iterations later, what you have in front of you is my attempt at plugging another hole in the set of intermediate Rust teaching resources. I hope that you find it useful and that we’re now one step closer to fulfilling that roadmap goal!

Acknowledgments

Having never written a book before, I knew little of what to expect from the process. I naively assumed that it would be like writing a sequence of blog posts, or perhaps like writing extensive documentation, but writing a book has been a whole different ballgame. I’ve spent countless hours researching, planning, writing, rewriting, discarding, rethinking, and editing. And I’m incredibly grateful for the support and patience my girlfriend, Talia, has shown throughout the many late nights I spent working on this project—without you, the writing experience would have been so much bleaker.

This book could never have happened without the incredible Rust developers and community, who have developed a language and ecosystem I continue to find a joy to interact with and am inspired to help spread and teach to the best of my ability. The same goes for the amazing people who have watched my streams over the years; I don’t think I would ever have ended up in a position to write this book in the first place without your ongoing support, encouragement, and endless curiosity.

This book also would not have been half as good had it not been for David Tolnay, who to my great delight agreed to be the book’s technical reviewer. David was obviously invaluable in finding errors in both theory and code, but it was his vast experience, attention to detail, and penchant for pedagogy that truly made a mark. His thoughtful and insightful comments sometimes made me decide to rewrite entire sections, but always in ways that made them immeasurably better than they had been before.

The same goes for my editor, Liz Chadwick, and the rest of the publishing team at No Starch. It was so fun to see Liz’s journey through the book’s development; she picked up Rust along the way while reading, and I was thrilled when her comments showed that she truly followed along with the intermediate material. The discussions whenever there was something she didn’t follow were always illuminating, and resulted in more accessible and thorough explanations.

I’d also be remiss not to mention Steve Klabnik and Carol Nichols, the authors of The Rust Programming Language, which was my first introduction to Rust. This book is, at least in my mind, very much a sequel to their book and could not exist without the extraordinary job they’ve done of making the fundamentals of Rust so easily accessible and well explained.

Finally, I want to give a nod to you. Yes, you! Writing this book has been a very long process, and it’s partially the outpouring of support and encouragement from the people wanting to read it that has kept me going throughout it all. And it’s people like you who pick up this book, whether virtually or physically, with a desire to improve your own understanding and skills that drive me to keep contributing to the collection of Rust teaching resources as best I can.

Thank you all!

Introduction

In any language, the gap between what the introductory material teaches you and what you know after years of hands-on experience is always wide. Over time, you build familiarity with idioms, develop better mental models for core concepts, learn which designs and patterns work and which do not, and discover useful libraries and tools in the surrounding ecosystem. Taken together, this experience enables you to write better code in less time.

With this book, I’m hoping to distill years of my own experience writing Rust code into a single, easy-to-digest resource. Rust for Rustaceans picks up where The Rust Programming Language (the Rust book) leaves off, though it’s well suited to any Rust programmer that wants to go beyond the basics, wherever you learned the trade. This book delves deeper into concepts such as unsafe code, the trait system, no_std code, and macros. It also covers new areas like asynchronous I/O, testing, embedded development, and ergonomic API design. I aim to explain and demystify these more advanced and powerful features of Rust and to enable you to build faster, more ergonomic, and more robust applications going forward.

What’s in the Book

This book is written both as a guide and as a reference. The chapters are more or less independent, so you can skip directly to topics that particularly interest you (or are currently causing you headaches), or you can read the book start to finish for a more holistic experience. That said, I do recommend that you start by reading Chapters 1 and 2, as they lay the foundation for the later chapters and for many topics that will come up in your day-to-day Rust development. Here’s a quick breakdown of what you’ll find in each chapter:

Chapter 1, Foundations, gives deeper, more thorough descriptions of fundamental Rust concepts like variables, memory, ownership, borrowing, and lifetimes that you’ll need to be familiar with to follow the remainder of the book.

Chapter 2, Types, similarly provides a more exhaustive explanation of types and traits in Rust, including how the compiler reasons about them, their features and restrictions, and a number of advanced applications.

Chapter 3, Designing Interfaces, covers how to design APIs that are intuitive, flexible, and misuse-resistant, including advice on how to name things, how to use the type system to enforce API contracts, and when to use generics versus trait objects.

Chapter 4, Error Handling, explores the two primary kinds of errors (enumerated and opaque), when the use of each is appropriate, and how each of these are defined, constructed, propagated, and handled.

Chapter 5, Project Structure, focuses on the non-code parts of a Rust project, such as Cargo metadata and configuration, crate features, and versioning.

Chapter 6, Testing, details how the standard Rust testing harness works and presents some testing tools and techniques that go beyond standard unit and integration tests, such as fuzzing and performance testing.

Chapter 7, Macros, covers both declarative and procedural macros, including how they’re written, what they’re useful for, and some of their pitfalls.

Chapter 8, Asynchronous Programming, gives an introduction to the difference between synchronous and asynchronous interfaces and then delves into how asynchrony is represented in Rust both at the low level of Future and Pin and at the high level of async and await. The chapter also explains the role of an asynchronous executor and how it makes the whole async machinery come together.

Chapter 9, Unsafe Code, explains the great powers that the unsafe keyword unlocks and the great responsibilities that come with those powers. You’ll learn about common gotchas in unsafe code as well as tools and techniques you can use to reduce the risk of incorrect unsafe code.

Chapter 10, Concurrency (and Parallelism), looks at how concurrency is represented in Rust and why it can be so difficult to get right in terms of both correctness and performance. It covers how concurrency and asynchrony are related (but not the same), how concurrency works when you get closer to the hardware, and how to stay sane while trying to write correct concurrent programs.

Chapter 11, Foreign Function Interfaces, teaches you how to make Rust cooperate nicely with other languages and what FFI primitives like the extern keyword actually do.

Chapter 12, Rust Without the Standard Library, is all about using Rust in situations where the full standard library isn’t available, such as on embedded devices or other constrained platforms, where you’re restricted to what the core and alloc modules provide.

Chapter 13, The Rust Ecosystem, doesn’t cover a particular Rust subject but instead aims to give broader guidance about working in the Rust ecosystem. It contains descriptions of common design patterns, advice on staying up to date on additions to the language and best practices, tips on useful tools and other useful trivia I’ve accumulated over the years that isn’t otherwise described in any single place.

The book has a website at https://rust-for-rustaceans.com with links to resources from the book, future errata, and the like. You’ll also find that information at the book’s page on the No Starch Press website at https://nostarch.com/rust-rustaceans/.

And now, with all that out of the way, there’s only one thing left to do:

fn main() {

1

Foundations

As you dive into the more advanced corners of Rust, it’s important that you ensure you have a solid understanding of the fundamentals. In Rust, as in any programming language, the precise meaning of various keywords and concepts becomes important as you begin to use the language in more sophisticated ways. In this chapter, we’ll walk through many of Rust’s primitives and try to define more clearly what they mean, how they work, and why they are exactly the way that they are. Specifically, we’ll look at how variables and values differ, how they are represented in memory, and the different memory regions a program has. We’ll then discuss some of the subtleties of ownership, borrowing, and lifetimes that you’ll need to have a handle on before you continue with the book.

You can read this chapter from top to bottom if you wish, or you can use it as a reference to brush up on the concepts that you feel less sure about. I recommend that you move on only when you feel completely comfortable with the content of this chapter, as misconceptions about how these primitives work will quickly get in the way of understanding the more advanced topics, or lead to you using them incorrectly.

Talking About Memory

Not all memory is created equal. In most programming environments, your programs have access to a stack, a heap, registers, text segments, memory-mapped registers, memory-mapped files, and perhaps nonvolatile RAM. Which one you choose to use in a particular situation has implications for what you can store there, how long it remains accessible, and what mechanisms you use to access it. The exact details of these memory regions vary between platforms and are beyond the scope of this book, but some are so important to how you reason about Rust code that they are worth covering here.

Memory Terminology

Before we dive into regions of memory, you first need to know about the difference between values, variables, and pointers. A value in Rust is the combination of a type and an element of that type’s domain of values. A value can be turned into a sequence of bytes using its type’s representation, but on its own you can think of a value more like what you, the programmer, meant. For example, the number 6 in the type u8 is an instance of the mathematical integer 6, and its in-memory representation is the byte 0x06. Similarly, the str Hello world is a value in the domain of all strings whose representation is its UTF-8 encoding. A value’s meaning is independent of the location where those bytes are stored.

A value is stored in a place, which is the Rust terminology for a location that can hold a value. This place can be on the stack, on the heap, or in a number of other locations. The most common place to store a value is a variable, which is a named value slot on the stack.

A pointer is a value that holds the address of a region of memory, so the pointer points to a place. A pointer can be dereferenced to access the value stored in the memory location it points to. We can store the same pointer in more than one variable and therefore have multiple variables that indirectly refer to the same location in memory and thus the same underlying value.

Consider the code in Listing 1-1, which illustrates these three elements.

let x = 42;

let y = 43;

let var1 = &x;

let mut var2 = &x;

1 var2 = &y;

Listing 1-1: Values, variables, and pointers

Here, there are four distinct values: 42 (an i32), 43 (an i32), the address of x (a pointer), and the address of y (a pointer). There are also four variables: x, y, var1, and var2. The latter two variables both hold values of the pointer type, because references are pointers. While var1 and var2 store the same value initially, they store separate, independent copies of that value; when we change the value stored in var2 1, the value in var1 does not change. In particular, the = operator stores the value of the right-hand side expression in the place named by the left-hand side.

An interesting example of where the distinction between variables, values, and pointers becomes important is in a statement such as:

let string = Hello world;

Even though we assign a string value to the variable string, the actual value of the variable is a pointer to the first character in the string value Hello world, and not the string value itself. At this point you might say, But hang on, where is the string value stored, then? Where does the pointer point? If so, you have a keen eye—we’ll get to that in a second.

Note

Technically, the value of string also includes the string’s length. We’ll talk about that in Chapter 2 when we discuss wide pointer types.

Variables in Depth

The definition of a variable I gave earlier is broad and unlikely to be all that useful in and of itself. As you encounter more complex code, you’ll need a more accurate mental model to help you reason through what the programs are really doing. There are many such models that we can make use of. Describing them all in detail would take up several chapters and is beyond the scope of this book, but broadly speaking, they can be divided into two categories: high-level models and low-level models. High-level models are useful when thinking about code at the level of lifetimes and borrows, while low-level models are good for when you are reasoning about unsafe code and raw pointers. The models for variables described in the following two sections will suffice for most of the material in this book.

High-Level Model

In the high-level model, we don’t think of variables as places that hold bytes. Instead, we think of them just as names given to values as they are instantiated, moved, and used throughout a program. When you assign a value to a variable, that value is from then on named by that variable. When a variable is later accessed, you can imagine drawing a line from the previous access of that variable to the new access, which establishes a dependency relationship between the two accesses. If the value in a variable is moved, no lines can be drawn from it anymore.

In this model, a variable exists only so long as it holds a legal value; you cannot draw lines from a variable whose value is uninitialized or has been moved, so effectively it isn’t there. Using this model, your entire program consists of many of these dependency lines, often called flows, each one tracing the lifetime of a particular instance of a value. Flows can fork and merge when there are branches, with each split tracing a distinct lifetime for that value. The compiler can check that at any given point in your program, all flows that can exist in parallel with each other are compatible. For example, there cannot be two parallel flows with mutable access to a value. Nor can there be a flow that borrows a value while there is no flow that owns the value. Listing 1-2 shows examples of both of these cases.

let mut x; // this access would be illegal, nowhere to draw the flow from: // assert_eq!(x, 42); 1 x = 42; // this is okay, can draw a flow from the value assigned above: 2 let y = &x; // this establishes a second, mutable flow from x: 3 x = 43; // this continues the flow from y, which in turn draws from x. // but that flow conflicts with the assignment to x! 4 assert_eq!(*y, 42);

Listing 1-2: Illegal flows that the borrow checker will catch

First, we cannot use x before it is initialized, because we have nowhere to draw the flow from. Only when we assign a value to x can we draw flows from it. This code has two flows: one exclusive (&mut) flow from 1 to 3, and one shared (&) flow from 1 through 2 to 4. The borrow checker inspects every vertex of every flow and checks that no other incompatible flows exist concurrently. In this case, when the borrow checker inspects the exclusive flow at 3, it sees the shared flow that terminates at 4. Since you cannot have an exclusive and a shared use of a value at the same time, the borrow checker (correctly) rejects the code. Notice that if 4 was not there, this code would compile fine! The shared flow would terminate at 2, and when the exclusive flow is checked at 3, no conflicting flows would exist.

If a new variable is declared with the same name as a previous one, they are still considered distinct variables. This is called shadowing—the later variable shadows the former by the same name. The two variables coexist, though subsequent code no longer has a way to name the earlier one. This model matches roughly how the compiler, and the borrow checker in particular, reasons about your program, and is actually used internally in the compiler to produce efficient code.

Low-Level Model

Variables name memory locations that may or may not hold legal values. You can think of a variable as a value slot. When you assign to it, the slot is filled, and its old value (if it had one) is dropped and replaced. When you access it, the compiler checks that the slot isn’t empty, as that would mean the variable is uninitialized or its value has been moved. A pointer to a variable refers to the variable’s backing memory and can be dereferenced to get at its value. For example, in the statement let x: usize, the variable x is a name for a region of memory on the stack that has room for a value the size of a usize, though it does not have a well-defined value (its slot is empty). If you assign a value to that variable, such as with x = 6, that region of memory will then hold the bits representing the value 6. &x does not change when you assign to x. If you declare multiple variables with the same name, they still end up with different chunks of memory backing them. This model matches the memory model used by C and C++, and many other low-level languages, and is useful for when you need to reason explicitly about memory.

Note

In this example, we ignore CPU registers and treat them as an optimization. In reality, the compiler may use a register to back a variable instead of a region of memory if no memory address is needed for that variable.

You may find that one of these matches your previous model better than the other, but I urge you to try to wrap your head around both of them. They are both equally valid, and both are simplifications, like any useful mental model has to be. If you are able to consider a piece of code from both of these perspectives, you will find it much easier to work through complicated code segments and understand why they do or do not compile and work as you expect.

Memory Regions

Now that you have a grip on how we refer to memory, we need to talk about what memory actually is. There are many different regions of memory, and perhaps surprisingly, not all of them are stored in the DRAM of your computer. Which part of memory you use has a significant impact on how you write your code. The three most important regions for the purposes of writing Rust code are the stack, the heap, and static memory.

The Stack

The stack is a segment of memory that your program uses as scratch space for function calls. Each time a function is called, a contiguous chunk of memory called a frame is allocated at the top of the stack. Near the bottom of the stack is the frame for the main function, and as functions call other functions, additional frames are pushed onto the stack. A function’s frame contains all the variables within that function, along with any arguments the function takes. When the function returns, its stack frame is reclaimed.

The bytes that make up the values of the function’s local variables are not immediately wiped, but it’s not safe to access them as they may have been overwritten by a subsequent function call whose frame overlaps with the reclaimed one. And even if they haven’t been overwritten, they may contain values that are illegal to use, such as ones that were moved when the function returned.

Stack frames, and crucially the fact that they eventually disappear, are very closely tied to the notion of lifetimes in Rust. Any variable stored in a frame on the stack cannot be accessed after that frame goes away, so any reference to it must have a lifetime that is at most as long as the lifetime of the frame.

The Heap

The heap is a pool of memory that isn’t tied to the current call stack of the program. Values in heap memory live until they are explicitly deallocated. This is useful when you want a value to live beyond the lifetime of the current function’s frame. If that value is the function’s return value, the calling function can leave some space on its stack for the called function to write that value into before it returns. But if you want to, say, send that value to a different thread with which the current thread may share no stack frames at all, you can store it on the heap.

The heap allows you to explicitly allocate contiguous segments of memory. When you do so, you get a pointer to the start of that segment of memory. That memory segment is reserved for you until you later deallocate it; this process is often referred to as freeing, after the name of the corresponding function in the C standard library. Since allocations from the heap do not go away when a function returns, you can allocate memory for a value in one place, pass the pointer to it to another thread, and have that thread safely continue to operate on that value. Or, phrased differently, when you heap-allocate memory, the resulting pointer has an unconstrained lifetime—its lifetime is however long your program keeps it alive.

The primary mechanism for interacting with the heap in Rust is the Box type. When you write Box::new(value), the value is placed on the heap, and what you are given back (the Box) is a pointer to that value on the heap. When the Box is eventually dropped, that memory is freed.

If you forget to deallocate heap memory, it will stick around forever, and your application will eventually eat up all the memory on your machine. This is called leaking memory and is usually something you want to avoid. However, there are some cases where you explicitly want to leak memory. For example, say you have a read-only configuration that the entire program should be able to access. You can allocate that on the heap and explicitly leak it with Box::leak to get a 'static reference to it.

Static Memory

Static memory is really a catch-all term for several closely related regions located in the file your program is compiled into. These regions are automatically loaded into your program’s memory when that program is executed. Values in static memory live for the entire execution of your program. Your program’s static memory contains the program’s binary code, which is usually mapped as read-only. As your program executes, it walks through the binary code in the text segment instruction by instruction and jumps around whenever a function is called. Static memory also holds the memory for variables you declare with the static keyword, as well as certain constant values in your code, like strings.

The special lifetime 'static, which gets its name from the static memory region, marks a reference as being valid for as long as static memory is around, which is until the program shuts down. Since a static variable’s memory is allocated when the program starts, a reference to a variable in static memory is, by definition, 'static, as it is not deallocated until the program shuts down. The inverse is not true—there can be 'static references that do not point to static memory—but the name is still appropriate: once you create a reference with a static lifetime, whatever it points to might as well be in static memory as far as the rest of the program is concerned, as it can be used for however long your program wishes.

You will encounter the 'static lifetime much more often than you will encounter truly static memory (through the static keyword, for example) when working with Rust. This is because 'static often shows up in trait bounds on type parameters. A bound like T: 'static indicates that the type parameter T is able to live for however long we keep it around for, up to and including the remaining execution of the program. Essentially, this bound requires that T is owned and self-sufficient, either in that it does not borrow other (non-static) values or that anything it does borrow is also 'static and thus will stick around until the end of the program. A good example of 'static as a bound is the std::thread::spawn function that creates a new thread, which requires that the closure you pass it is 'static. Since the new thread may outlive the current thread, the new thread