Introduction

10 PRINT "HELLO"
20 GOTO 10

In April, 1984, my father bought a computer for his home office, a Luxor ABC-802, with a Z80 CPU, 64 kilobytes of RAM, a yellow-on-black screen with 80 by 25 text mode, or about 160 by 75 pixels in graphics mode, and two floppy drives. It had BASIC in its ROM, and came with absolutely no games. If I wanted to play with it, I had to learn how to program, and write my own games. I learned BASIC, and over the next few years would learn Pascal, C, and more. I had found my passion. I was 14 years old and I knew what I wanted to do when I grew up.

When I was learning how to program, I thought it was important to really understand how computers work, how programming languages work, and how various tools like text editors work. I wanted to hone my craft and produce the finest code humanly possible. I was wrong.

This essay is a condensation of what I wish I had been told after I had learned the basics of how to code. Instead, I was told, in person and in magazine articles, that by the time I would be twenty five years old, I'd be too old to work as a programmer. It was critical that I learn as many algorithms, data structures, and languages as quickly as possible. Programming was, after all, a young man's game, and required being able to stay up all night, every night, to crank out more code than anyone else. That was the only way to succeed. It turns out that none of that was true: not the part about youth, nor the part of missing sleep, and especially not the part about gender.

As I write this essay, it will soon be forty years to the day since I first wrote computer code. I've managed to support myself by developing software, and I still write code every day. There is nothing else I would rather do for a living. I can't point at enormous successes and impressive feats, but I hope that surviving for decades in the industry gives me sufficient credentials to speak about software development.

This essay discusses some of the things I've learned about how to successfully build software. These are things I've learned from my own experience; I'm not a researcher, and there are few references to sources, and this is largely not supported by evidence. I'm basing this essay on my own experience, and if you disagree, that's fine.

My goal in this essay is to get the reader to think, to research, to learn, to ponder. My goal is not to tell the reader how to think, what to think, how things are, or to give the answer to every question about every aspect of the process of building software.

The core skills

Interesting and significant software is beyond the capacity of any one person to build alone in a reasonable time frame. This means that the fundamental, crucial, core skills in building software are communication and collaboration.

It's not enough to know how computers work, how to use programming languages, to know algorithms and data structures, or how to use the varied tools involved in software construction. You also need to know how to talk with other people to learn what software to build, what it must do, how much effort is acceptable, how to manage the work, and many more things. You have know how to work with others to build something together that's bigger than any of you. If you and your team can do it well, it'll be bigger than the sum of you. Team work can be a force multiplier.

These are difficult skills to learn. I've found them much more difficult than any technical part of building software.

Even in the projects where I'm the only person, there are at least three people involved: past me, present me, and future me.

• Past me is a lazy and careless slob who always leaves a mess.
• Present me does superb work, but has to cope with all the stupid stuff done by past me, and needs to placate future me.
• Future me is an egotistic and opinionated snob for whom nothing is ever good enough.

Getting these three people to get along well enough to get anything done is a constant struggle for me. I use my journal to cope. I write there what I do, why, and what the result is. I try to write so that I'll understand it, even if I'm tired, or have just experienced a short term neuralizer mind wipe a la Men in Black. I also plan explicit iterations, with specific tasks, and I use the GTD system to track things that still need to done.

On productivity

What productivity means is not well defined for software development: time is easy to measure, but the output is not. Most people have at least a vague idea what productivity means to them. However, there are known factors that impact productivity---these are customarily ignored.

To do good work you need to take care of yourself. You have to sleep well. You have to eat well. If you're tired, or have too much stress, you'll make mistakes and bad decisions. These result in you doing bad work. That's not a moral problem, but you may have to fix your own mistakes. You'll enjoy life, and work, more if you do good work.

Hillel Wayne has talked about sleep and has sources. Recommended reading:

• https://www.hillelwayne.com/talks/ese/
• https://www.hillelwayne.com/talks/what-we-know-we-dont-know/

You also need to have an environment that helps you do good work. Is it quiet? Are there interruptions? Is the furniture comfortable, and unlikely to hurt you in the long run? Do you have access to necessities, such as a toilet and tea? Can you take a break from your work area? Can you go for a walk: walking helps thinking. This is an incomplete list.

Further, you need to manage yourself and your work in ways that suit you. The best way to do this depends on who you are, your preferences and experiences, and the kind of work you do. There is no single solution that works for everyone all the time.

For myself, I've found applying the David Allen Getting Things Done system has worked well, but whatever you do, you need to know what you need to achieve, today and in the near future, and you need to arrange things so that you can concentrate on one thing at a time. Note, however, that I use the GTD system partly to know that I can slack off; I do not aim to be as productive as possible at all times.

It may be worth tracking what you've done. Reflecting on that can give you a sense of achievement, and lets you see the progress you make. This can be a motivation and morale booster.

Multi-tasking is fine for a computer, but your brain can only think about one thing at a time and has really miserable context switching.

On governance

When a group collaborates, one of the things they need to establish early on is governance: basically, how does the group make decisions, and how do they change them? The decisions range from the fundamental to the mundane: Who is in the group? Is it a democracy, or hierarchy, or some other structure? Who gets to have a say? What beverages does the group get? What font should the website use?

Governance is difficult, but it's easier when it's explicit. Uncertainty about responsibility and power results in confusion and quarrel, and these can tear a group apart.

All groups eventually have conflicts. Managing this and resolving differences is nominally the job of management, but really it falls on everyone. This requires skills that seem to be rarely taught to programmers, which is a shame.

Recommended reading:

• Contributor Covenant
• The Tyranny Of Structurelessness by Jo Freeman

On politics and ethics

Software development is always a political and ethical act. Whenever you're building software, you have to make a myriad of decisions:

• what shall the software do?
• what resources will using it require?
• how shall it be used?
• what abilities shall using it require?
• what things will be easy to do?
• what things will be hard to do?
• will the software empower its users, or force them to do things for someone else's benefit?

All these decisions have consequences, which will favor some people over other people. Hence, they have political and ethical aspects, which need to be considered.

As an example, if you make something that can be used with a cheap phone, you enable most people in the world to use it. If you make it so that it requires expensive hardware, you exclude many people, especially poor people. Sometimes that is inevitable, and inherent in the problem you're building software to solve, but it's still a choice.

There may not be a right or wrong choice for a particular decision, but there are always consequences. You have to consider them, and decide if they're acceptable, and for whom.

In general, the software you build will reflect some values. Make sure you know you agree with them.

Recommended reading:

• Tool Safety by Leonard Richardson

On ethics and software freedom

Free and open source software is the ethical kind of software. Software freedom is essential for the well-being of people whose lives are affected by the use of computers, whether they are using computers directly or someone else is using computers to do things that matter to other people.

I have been writing free software since 1986, when I first read the GNU Manifesto on a BBS. (The term "open source" is a synonym: I do not care to debate the differences between the two.)

Much of my career has involved building open source software. My preference for free and open source over non-free software is clear, and well documented.

Entire libraries' worth of text has been produced over the past three decades about the ethics of free and open source software versus other kinds, and specifics of licenses. I'm not going to make a summary of arguments here, or even point at recommended reading.

I wanted to make my position clear, in case it matters to the reader. However, very little of this essay, including the topic of ethics, is related to software freedom and applies equally to the development non-free software.

On diversity and quality

Human rights are fundamentally important. Treating other people well is the right thing to do. All of this is of paramount priority, whatever you do. If you don't agree with this, I have no hope for you.

In a software development context, I've found that the most crucial thing for building high quality software is to have diversity of thought among the people contributing to it. The more different kinds of thinking is brought into the project, the more likely the decisions will work for more circumstances, for more people, and deal with more things going wrong.

Diversity of thought comes from different kinds of people coming from different backgrounds and having lived different kinds of lives. Sometimes differences are visible externally, sometimes not.

When in doubt, choose different. If you exclude people based on them being unlike you, you will likely be choosing poorly.

Diversity doesn't guarantee success. Nothing guarantees success. However, uniformity guarantees you get one-sided answers. Sometimes that's enough, but often it's not.

Collaboration and communication can certainly be more challenging in a diverse setting. Do not fear this. Treat challenges as an opportunity to learn, to become better at what you do.

People will make mistakes, and you too will make them. It's a good policy to be benign and kind when others make mistakes, and expect that from others in return. Punishing others for mistakes will leave you alone.

On maintenance

It is widely considered, in the software industry, that most of the cost of software production comes after the initial release, in the so called maintenance phase. The initial development might take a year or two, maintenance will take decades.

One would think that this would lead to development practices, and architectural decision, and everything else to be optimized for lower maintenance costs. Unfortunately, the economic and other incentives favor the opposite. In most companies, the event horizon tends to be a quarter year, or a full year at most. It's not considered acceptable to spend a lot more effort now to save most of the effort in the long term.

Even for personal hobby projects, there is often a strong urge to get something working as soon as possible rather than taking one's time to do things well, but at least that's an internal urge, not external pressure. Even so, the end result is the same: software that's harder to change later on, when bugs are found, requirements change, or the software needs to be adapted to changes in the surrounding system or ecosystem.

I am not alone in seeing the problem. I don't think it can be solved, unless the economic incentives are radically changed. It's on individual developers to try to reduce maintenance costs surreptitiously, when they can, as a kind of preemptive guerrilla maintenance.

Big questions for projects

I find that it's important, but also helpful, to answer a few questions at the beginning of every new project. The answers don't have to be final, and it's OK to change them at any time: if you don't adjust your thinking when you learn new things, you're not good at what you do. The answers can even be inane, as long as they're honest. The point to is to get you thinking about the questions at all.

• For whom are you building the software? Whose opinions about it matter?

  This tells you some of the stakeholders in the project. There will likely be more stakeholders later on, but it's a start. Knowing who the stakeholders are lets you concentrate on their needs. This helps make the project a success.

  The stakeholders might be the end users, or might be the CEO. If you think it's the end users, but it's actually the CEO, you will make decisions that make the CEO unhappy, and the project will fail, and you won't understand why. And vice versa.

• Why are you building the software?

  "It's my job" might be a very practical answer. It's an answer that means that you need to re-evaluate everything if the chance of getting money is reduced. A different answer might be more about passion, or mission, and this means your reaction to changing circumstances is going to be different. There's no wrong answer here, but be sure to be honest, at least to yourself.

• What should the software do, in broad strokes? Also, what should it not do?

  This is about knowing if you're building a word processor or a spreadsheet. It's not about detailed requirements or acceptance criteria.

• How should the software work, in broad strokes?

  Again, big picture view. Are you making a command line tool, a web service, an operating system, a mobile app, or something else? What type and size of hardware?

• What's important and what is just nice to have?

  This gets into requirements and acceptance criteria. You need to know them to build something good. Write them down. Also write down how you verify you meet them.

Planning and estimating

Detailed planning beyond the very near future is difficult and usually fails. This includes estimating how long the work will take. I avoid doing this.

Planning and estimation are far from useless and should not be neglected. I plan and I estimate, but within the limits of what I've learned to be realistic. Specifically, I find it nearly impossible to make detailed plans beyond a one or two week iteration. By detailed plans I mean figuring out tasks at a granularity of what can be done in one sitting.

Something surprising usually happens that ruins the plans beyond a couple of weeks. It might be management changing priorities, or the client changing their mind, or the company going bankrupt, or being bought. It might be something else, but usually something happens. If an iteration is short, you don't lose too much if all your planning has to be thrown away.

You can also react more easily if you need to change course.

I stress that I'm talking about detailed planning here. It's fine to make plans for what to do next month or next quarter, as long as everyone involved knows that these plans are likely to have to change, and the plans are kept suitably high level.

An analogy: if you're planning a road trip across the country, you will probably want to plan what cities to drive through. You probably don't want plan every stop to fill up on fuel or to eat. You want to leave enough flexibility that you can change route if there's an accident, or weather, or something else that may come as a surprise on the way.

Iteration works. Nothing else seems to work for large software projects. At least I've never seen any other approach work.

For me, iterations work better if they're focused. Having an explicit, clear goal helps cut down on scope creep. It concentrates work to aim at the iteration goal, rather than on what seems like a good idea in the moment.

When I plan for an iteration, I break things into small tasks that can be finished in one sitting. My estimates for tasks fall into four buckets: up to a quarter hour, full hour, four hours, or too long. I prefer the shorter tasks: they're easier to estimate, and easier to do, and usually break fewer things. Anything that's too long needs to be broken down further.

Some tasks are impossible to estimate. Debugging is a typical example. How long will it take you to realize the Ethernet cable is broken, or an operating system update broke things, or find the cause of some other unforeseen problem? Debugging happens, and you had best leave enough slack in your planning to cope with some troubleshooting.

Interruptions also happen. Sometimes you have no option but to react and respond to them immediately. Leave room for that.

Express yourself in writing

Your memory is fleeting and fallible. You will forget details. You will forget important stuff. You will remember things wrongly. When you collaborate, you and others will disagree on what was said, agreed upon, and decided.

I was once in a meeting, with four people, plus the CTO. The CTO forbade us to take notes: it seems the fad of the week was that note taking is what makes meetings a waste of time. The meeting took two hours. Afterwards, the four of us had about eight different opinions of what had been decided. No follow-up actions were ever taken.

Write things down. It seems like a stupid chore, tedious, boring, and archaic, but it's the way organizations remember. Writing things down is a super power that is easily overlooked.

Your team should keep meeting minutes, covering at least the important decisions, and what actions everyone should do after the meeting. Keep them short enough that they're easy write, and to read. Archive them somewhere everyone can look at them at any time. This helps people who weren't there, perhaps because they were on vacation, or only join a year later.

Apropos meetings, learn how to have good meetings. What works for me is an agenda set well ahead of time, with supporting materials. Prepare for the meeting so that you don't waste everyone's time. Have a chair who keeps discussion on track, and time boxed. Your process for a good meeting might be different, but you should find one.

On doing work

When making a change, make only one change at a time. If you can, split the change you're making into smaller partial changes. Small changes are easier to understand and less likely to be catastrophic.

Automate away friction: running tests, making a release, packaging, delivery, deployment, etc. Do this from as early on as feasible. Set up a pipeline where you can make a change and make sure the software still works and willing users can start using the changed software. The smoother you can make this pipeline, the easier it will be to build the software.

Use version control for every project. Practice continuous integration by merging changes to the main line of development often.

Develop an automated test suite that you trust: if tests pass, you can make a release, or deploy to production. This usually means starting the test suite early on in the project. Make sure the automated tests cover all aspects you care about: the goal is to make sure that what you give to others works they way it's intended. Update production code and test code together.

Run the test suite many times in a row to identify flaky tests. Do not suffer flaky tests.

Do stress or load testing, even if you only do it in simple ways. You don't know your software and system can handle 10,000 concurrent users until you've done so. Mine didn't.

In one job, we had a system that had worked fine in production for a while. We deployed it to a customer who had a lot more traffic. Our software failed. The problem turned out that we ran out of TCP port numbers: we made an HTTP request for each incoming message, but we didn't re-use the underlying TCP connection. There was so much traffic that all possible port numbers were used, and then everything stalled. The fix was to re-use the TCP connection, a change that took about one line, and then everything worked fine. We would have found this ourselves had we done even the simplest load testing.

If possible, use the software yourself. You'll understand your users better.

Watch other people use the software. You'll understand your software's sharp corners better. True story: it's hard for people who've not been involved in the development of software to use the software, if it has three different names for the same thing. Seeing someone struggle with this makes it painfully obvious.

Treat testers as friends who help you find when you've made a mistake. They're not out to humiliate you. Nobody is so perfect that they never make mistakes or overlook details or forget entire features. Don't ask me how I know.

Advice on coding

Simple, obvious code is easier to write, easier to get to work, easier to understand, and easier to change without breaking it. Simplify things as much as you can, but no more than that. Sacrifice simplicity at the altar of performance only when you have proven the performance gains are worth it.

Complexity is the enemy you think is a friend.

Coupling and cohesion are still important concepts.

In every project, conceptual clarity is important. Also, keep your terminology consistent. Confusion lurks where clarity hides. With great confusion comes great annoyance and the certainty of bugs.

Speed is a feature to users, but not always for development, as too much haste leaves too little time to think. Functionality can be a misfeature.

Recommended reading:

• A Philosophy of Software Design by John Ousterhout, which I've not yet read, only skimmed, but which people I respect keep recommending to me.

Developing a career

You can choose to be a deep expert on something very specific, or to be a generalist, or some mix. Choose wisely. There may not be any wrong choice, but every choice has consequences.

Be humble. Be Nanny, not Granny. People may respect the powerful witch more, but they like the kind one better.

Be open and honest. Treat others fairly. You don't have to believe in karma for it to work, so make it work for you, not against you.

Help and lift up others. But at the same time, don't allow others to abuse or take advantage of you. You don't need to accept bullshit. Set your boundaries.

Ask for help when you need it, or when you get stuck. Accept help when offered.

I am not the right person to talk about developing a career, but when I've done the above, things have usually ended up going well.

Takeaway

Take care of yourself. Sleep. Eat. Exercise. Rest. Relax. Take care of other people, as best you can. People are important. Software is just fun.

Acknowledgments

Thank you to Brennen Bearnes, Richard Braakman, Tyler Cipriani, Greg Grossmeier, Hackhörnchen, Soile Mottisenkangas, Daniel Silverstone, and Enrico Zini, and for reviewing drafts of this essay. Any errors are mine.