Developing Microservices With Node - Js - Sample Chapter
Developing Microservices With Node - Js - Sample Chapter
ee
$ 39.99 US
25.99 UK
P U B L I S H I N G
David Gonzalez
Developing Microservices
with Node.js
pl
C o m m u n i t y
E x p e r i e n c e
D i s t i l l e d
Developing Microservices
with Node.js
Learn to develop efficient and scalable microservices for
server-side programming in Node.js using this hands-on guide
Sa
m
David Gonzalez
services for a number of years, trying to find solutions for the right level of
abstraction and learning how to get the right balance between too concrete
and too abstract.
He studied in Spain, but soon moved to the wider and more interesting market
of Dublin, where he has been living since 2011. David is currently working as an
independent consultant in the FinTech sector. The URL to his Linkedin account is
https://ie.linkedin.com/in/david-gonzalez-737b7383.
He loves experimenting with new technologies and paradigms in order to get the
broader picture of the complex world of software development.
Preface
This book is a hands-on guide to start writing microservices using Node.js and
the most modern frameworks, especially Seneca and PM2. In the chapters, we will
discuss how to design, build, test, and deploy microservices using the best practices.
We will also discuss a valuable lesson: how to make the right level of compromise
in order to avoid over-designing and get the business requirements aligned with the
technical solutions.
Preface
Chapter 6, Testing and Documenting Node.js Microservices, introduces using the main
frameworks for testing on Node.js: Mocha and Chai. We will also make use of Sinon
(another framework) to mock services. Swagger will be our choice for documenting
microservices.
Chapter 7, Monitoring Microservices, covers using PM2 to monitor our microservices.
We will use it along with Keymetrics to get the maximum benefit of such an
advanced tool.
Chapter 8, Deploying Microservices, explains how, using PM2, we are going to learn
to deploy microservices in our different environments, managing our ecosystem of
applications with a single command and reducing the overhead introduced by the
microservices architecture. We will also discuss Docker, one of the most advanced
systems to deploy applications not only in Node.js, but also in general applications.
Microservices Architecture
Microservices are becoming more and more popular. Nowadays, pretty much
every engineer on a green field project should be considering using microservices
in order to improve the quality of the systems they build. They should know the
architectural principles involving such systems. We will expose the difference
between microservices and Service-Oriented Architecture (SOA). We will also
introduce a great platform to write microservices, Node.js, which will allow us to
create high-performing microservices with very little effort.
In this chapter, you will learn about microservices from the architectural point
of view:
Microservice-oriented architectures
Key benefits
Why Node.js?
[1]
Microservices Architecture
Monolithic software
The natural trend for new high-tech companies such as Amazon or Netflix is building
their new software using microservices, which is the ideal scenario: they get a huge
advantage of microservices-oriented software (through out this book, you will learn
how) in order to scale up their new products without a big effort. The problem is
that not all companies can plan their software upfront. Instead of planning, these
companies build software based on the organic growth experienced: few software
components group business flows by affinity. It is not rare to see companies with two
big software components: the user-facing website and the internal administration
tools. This is usually known as a monolithic software architecture.
Some of these companies face big problems when trying to scale the engineering
teams. It is hard to coordinate teams that build, deploy, and maintain a single
software component. Clashes on releases and reintroduction of bugs are a common
problem that drains a large chunk of energy from the teams. One of the solution
to this problem (it comes with benefits) is to split the monolithic software into
microservices so that the teams are able to specialize in a few smaller modules and
autonomous and isolated software components that can be versioned, updated, and
deployed without interfering with the rest of the systems of the company.
Splitting the monolith into microservices enables the engineering team to create
isolated and autonomous units of work that are highly specialized in a given task
such as sending e-mails, processing card payments, and so on.
[2]
Chapter 1
A microservice is an autonomous unit of work that can execute one task without
interfering with other parts of the system, similar to what a job position is to a
company. This has a number of benefits that can be used in favor of the engineering
team in order to help scale the systems of a company.
Nowadays, hundreds of systems are built using microservices-oriented architectures,
as follows:
Netflix: This is one of the most popular streaming services, it has built an
entire ecosystem of applications that collaborate in order to provide a reliable
and scalable streaming system used across the globe.
Spotify: This is one of the leading music streaming services in the world,
it has built this application using microservices. Every single widget of the
application (which is a website exposed as a desktop app using Chromium
Embedded Framework) is a different microservice that can be updated
individually.
Microservice-oriented architectures
Microservices-oriented architectures have some particularities that makes them
desirable for any mid/large-sized company that wants to keep their IT systems
resilient and in scale up/down-ready status.
How is it better?
They are not the holy grail of software engineering, but, when handled with care,
they become the perfect approach to solve most of the big problems faced by
tech-dependent companies.
It is important to keep the key principles of the microservices-oriented architecture's
design in mind, such as resilience, composability, elasticity, and so on; otherwise,
you could end up with a monolithic application split across different machines that
produces problems rather than an elegant solution.
Shortcomings
There is also some criticism around microservices-oriented architectures, as
they introduce some problems to deal with, such as latency, traceability, and
configuration management that are not present with monolithic-based software.
Some of the problems are described as follows:
Microservices Architecture
In general, engineers should try to evaluate the pros and cons of this approach
and make a decision on whether to use microservices or not in order to fit the
business needs.
Microservices-oriented architectures have some particularities that need to be taken
into consideration. When a software engineer is writing monolithic software, there
are some problems that are completely overlooked due to the nature of the software
being built.
For example, imagine that our software needs to send e-mails. In a monolithic
software, we would just add the functionality to the core of the application. We
might even choose to create a dedicated module to deal with e-mails (which seems
like a good idea). Now, imagine that we are creating a microservice and, instead of
adding a functionality to a big software artifact, we create a dedicated service that
can be deployed and versioned independently. In this case, we will have an extra
step that we didn't have to take into consideration, the network latency, to reach the
new microservice.
In the preceding example, no matter what approach (monolithic or microservices)
you are taking to build the software, is not a big deal; for example, if an e-mail is lost,
it is not the end of the world. As per definition, the e-mail delivery is not guaranteed,
so our application will still work, although we might receive a few complaints from
our customers.
They are smart endpoints that contain the business logic and communicate
using simple channels and protocols.
[4]
Chapter 1
This simple interface can be understood by everyone, but it is the key when moving
towards microservices. We have encapsulated all the business knowledge behind
an interface so that we could theoretically switch the payment provider without
affecting the rest of the applicationthe implementation details are hidden from the
outer world.
The following is what we know until now:
We know the method name, therefore, we know how to invoke the service
[5]
Microservices Architecture
We have created a highly cohesive and loosely coupled business unit. Let's justify
this affirmation in the following:
Highly cohesive: All the code inside the payments module will do only one
thing, that is, deal with payments and all the aspects of calling a third-party
service (connection handling, response codes, and so on), such as a debit
card processor.
Loosely coupled: What happens if, for some reason, we need to switch
to a new payment processor? Is there any information bleeding out of the
interface? Would we need to change the calling code due to changes in the
contract? The answer is no. The implementation of the payment service
interface will always be a modular unit of work.
The following diagram shows how a system composed of many components gets
one of them (payment service) stripped out into a microservice:
[6]
Chapter 1
Once this module is implemented, we will be able to process the payments and
our monolithic application will have another functionality that could be a good
candidate to extract into a microservice.
Now, we can rollout new versions of the payment service, as long as the interface
does not change, as well as the contract with the rest of the world (our system
or third parties), hasn't changed. That is why it is so important to keep the
implementation independent from interfacing, even though the language does not
provide support for interfaces.
We can also scale up and deploy as many payment services as we require so that
we can satisfy the business needs without unnecessarily scaling the rest of the
application that might not be under pressure.
Downloading the example code
You can download the example code files for this book from your
account at http://www.packtpub.com. If you purchased this book
elsewhere, you can visit http://www.packtpub.com/support
and register to have the files e-mailed directly to you.
You can download the code files by following these steps:
Select the book for which you're looking to download the code
files.
You can also download the code files by clicking on the Code Files
button on the book's webpage at the Packt Publishing website. This
page can be accessed by entering the book's name in the Search box.
Please note that you need to be logged in to your Packt account.
Once the file is downloaded, please make sure that you unzip or
extract the folder using the latest version of:
[7]
Microservices Architecture
As you can see, it is a human readable protocol that does not need to be explained in
order to be understood.
Nowadays, it is broadly understood that HTTP is not confined to be used in the Web,
and as it was designed, it is now used as a general purpose protocol to transfer data
from one endpoint to another. HTTP is all you need for the communication between
microservices: a protocol to transfer data and recover from transmission errors
(when possible).
In the past few years, especially within the enterprise world, there has been an effort
to create smart communication mechanisms such as BPEL. BPEL stands for Business
Process Execution Language, and instead of focusing on communication actions, it
focuses on actions around business steps.
This introduces some level of complexity in the communication protocol and makes
the business logic of the application bleed into it from the endpoints, causing some
level of coupling between the endpoints.
[8]
Chapter 1
The business logic should stay within the endpoints and not bleed into the
communication channel so that the system can be easily tested and scaled. The
lesson learned through the years is that the communication layer must be a plain
and simple protocol that ensures the transmission of the data and the endpoints
(microservices).These endpoints should embed into their implementation the fact
that a service could be down for a period of time (this is called resilience, we will talk
about this later in this chapter) or the network could cause communication issues.
HTTP usually is the most used protocol when building microservices-oriented
software but another interesting option that needs to be explored is the use of queues,
such as Rabbit MQ and Kafka, to facilitate the communication between endpoints.
The queueing technology provides a clean approach to manage the communication
in a buffered way, encapsulating the complexities of acknowledging messages on
highly transactional systems.
Decentralization
One of the major cons of monolithic applications is the centralization of everything
on a single (or few) software components and databases. This, more often than not,
leads to huge data stores that needs to be replicated and scaled according to the
needs of the company and centralized governance of the flows.
Microservices aim for decentralization. Instead of having a huge database, why not
split the data according to the business units explained earlier?
Some of the readers could use the transactionality as one of the main reasons for not
doing it. Consider the following scenario:
1. A customer buys an item in our microservices-oriented online shop.
2. When paying for the item, the system issues the following calls:
1. A call to the financial system of the company to create a transaction
with the payment.
2. A call to the warehouse system to dispatch the book.
3. A call to the mailing system to subscribe the customer to the
newsletter.
[9]
Microservices Architecture
In a monolithic software, all the calls would be wrapped in a transaction, so if, for
some reason, any of the calls fails, the data on the other calls won't be persisted
in the database.
When you learn about designing databases, one of the first and the most important
principles are summarized by the ACID acronym:
[ 10 ]
Chapter 1
[ 11 ]
Microservices Architecture
Technology alignment
When building a new software, there is always a concept that every developer
should keep in mind: standards.
Standards guarantee that your service will be technologically independent so that
it will be easy to build the integrations using a different programming language
or technologies.
One of the advantages of modeling a system with microservices is that we can
choose the right technology for the right job so that we can be quite efficient when
tackling problems. When building monolithic software, it is fairly hard to combine
technologies like we can do with microservices. Usually, in a monolithic software,
we are tied to the technology that we choose in the beginning.
Java Remote Method Invocation (RMI) is one example of the non-standard protocols
that should be avoided if you want your system to be open to new technologies. It is
a great way of connecting software components written in Java, but the developers
will struggle (if not fail) to invoke an RMI method using Node.js. This will tie our
architecture to a given language, which from the microservices point of view, will kill
one of the most interesting advantages: technology heterogeneity.
Chapter 1
The microservices should be as small as needed, but keep in mind that every
microservice adds an overhead to the operations team that needs to manage a new
service. Try to answer the question how small is too small? in terms of manageability,
scalability, and specialization. The microservice should be small enough to be
managed and scaled up (or down) quickly without affecting the rest of the system,
by a single person; and it should do only one thing.
As a general rule, a microservice should be small
enough to be completely rewritten in a sprint.
Key benefits
In the previous topic, we talked about what a microservices-oriented architecture is.
I also exposed the design principles that I have learned from experience, as well as
showed a few benefits of this type of architecture.
Now, it is time to outline these key benefits and show how they will help us to
improve the quality of our software, as well as be able to quickly accommodate the
new business requirements.
Resilience
Resilience is defined in Wikipedia as the ability of a system to cope with change. I like
to think about resilience as the ability of a system to gracefully recover from an exception
(transitory hardware failure, unexpectedly high network latency, and so on) or a
stress period without affecting the performance of the system once the situation has
been resolved.
Although it sounds simple, when building microservices-oriented software, the
source of problems broadens due to the distributed nature of the system, sometimes
making it hard (or even impossible) to prevent all abnormal situations.
Resilience is the ability to gracefully recover from errors. It also adds another level of
complexity: if one microservice is experiencing problems, can we prevent a general
failure? Ideally, we should build our system in a way that the service response is
degraded instead of resulting in a general failure, although this is not always easy.
[ 13 ]
Microservices Architecture
Scalability
Nowadays, one of the common problems in companies is the scalability of the
systems. If you have worked on a monolithic software before, I am sure that you have
experienced capacity problems at some point, alongside the growth of the company.
Usually, these problems are not across all the layers or subsystems of the application.
There is always a subsystem or service that performs significantly slower than the
rest, causing the entire application to fail if it is not able to cope with the demand.
The following diagram describes how a microservice can be scaled up (two mailing
services) without interfering with the rest of the system:
[ 14 ]
Chapter 1
If our insurance system was a microservice-oriented software, the only thing needed
to resolve the high demand for quote calculations would've been to spawn more
instances of the microservice (or microservices) responsible for their calculation. Please
bear in mind that scaling up services could add an overhead for operating them.
Technology heterogeneity
The world of software is changing every few months. New languages are coming
to the industry as a de facto standard for a certain type of systems. A few years
ago, Ruby on Rails appeared at the scene and rose as one of the most used web
frameworks for new projects in 2013. Golang (a language created by Google) is
becoming a trend nowadays as it combines huge performance with an elegant
and simple syntax that can be learned by anyone with some experience in another
programming language in a matter of days.
In the past, I have used Python and Java as successful alternatives to write
microservices.
Java especially, since Spring Boot was released, is an attractive technology stack to
write agile (to write and operate) microservices.
Django, is also a powerful framework on Python to write microservices. Being very
similar to Ruby on Rails, it automates database migrations and makes the creation of
CRUD (Create Read Update Delete) services an incredibly easy task.
Node.js took the advantage of a well-known language, JavaScript, to create a new
server-side stack that is changing the way engineers create new software.
So, what is wrong in combining all of them? In all fairness, it is an advantage: we can
choose the right tool for the right job.
Microservices-oriented architectures enable you to do it, as long as the integration
technologies are standard. As you learned before, a microservice is a small and
independent piece of software that can operate by itself.
[ 15 ]
Microservices Architecture
The following diagram shows how the microservices hide data storage/gathering,
having only the communication points in commonmaking them a good example
of low coupling (one service implementation change won't interfere with any other
service):
We have talked about performance earlier. There are always parts of our systems
that are under more pressure than others. With modern multicore CPUs, parallel
(concurrent) programming could solve some of these performance issues, however,
Node.js is not a good language to parallelize tasks. We could choose to rewrite the
microservice under pressure using a more appropriate language, such as Erlang, to
manage concurrency in a more elegant way. It should take no more than two weeks
to do it.
[ 16 ]
Chapter 1
Replaceability
Replaceability is the ability to change one component of a system without interfering
with how the system behaves.
When talking about software, replaceability comes along with low coupling. We
should be writing our microservices in a way that the internal logic will not be
exposed to the calling services so that the clients of a given service do not need
to know about how it is implemented, just the interface. Let's take a look at the
following example. It is written in Java as we only need to see the interface to
identify the pitfalls:
public interface GeoIpService {
/**
* Checks if an IP is in the country given by an ISO code.
**/
boolean isIn(String ip, String isoCode) throws
SOAPFaultException;
}
[ 17 ]
Microservices Architecture
Independence
No matter how hard we try, the human brain is not designed to solve complex
problems. The most efficient mode of functioning for the human brain is
one thing at the time so that we break down complex problems into smaller ones.
Microservices-oriented architectures should follow this approach: all the services
should be independent and interact through the interface up to a point that they can
be developed by different groups of engineers without any interaction, aside from
agreeing the interfaces. This will enable a company adopting microservices to scale
up, or down, the engineering teams, depending on the business needs, making the
business agile in responding to peaks of demand or periods of quietness.
Easy to deploy
Microservices should be easy to deploy.
Being software developers, we are well aware that a lot of things could go wrong,
preventing a software from being deployed.
Microservices, as stated before, should be easy to deploy for a number of reasons, as
stated in the following list:
Small amount of business logic (remember the two weeks re-write from scratch
rule of thumb) leading into simpler deployments.
[ 18 ]
Chapter 1
[ 19 ]
Microservices Architecture
Another difference between microservices and SOA is the technologies used for
interconnecting and writing the services.
J2EE is a technology stack that was designed to write SOA architectures as it
enforced enterprise standards. Java Naming and Directory Interface, Enterprise
Java Beans, and Enterprise Service Bus (ESB) were the ecosystems where SOA
applications were built and maintained. Although ESB is a standard, very few
engineers who graduated after 2005 have heard about ESB, even fewer have used it,
and nowadays the modern frameworks such as Ruby on Rails do not even consider
such complex pieces of software.
On the other hand, microservices enforce the use of standards (such as HTTP) that
are broadly known and broadly interoperable. We can choose the right language or
tool to build a component (microservice) following one of the key benefits explained
earlier in this chapter, in the Technology heterogeneity section.
Aside from the technology stack and the size of the services, there is an even bigger
difference between SOA and microservices: the domain model. Earlier in this
chapter, we have talked about decentralization. Decentralization of the governance,
but, moreover, decentralization of the data. In a microservices-based software, every
microservice should store its own data locally, isolating the domain models to a
single service; whereas, on an SOA oriented-software, the data is usually stored in a
few big databases and the services share the domain models.
Why Node.js?
A few years ago, I didn't believe in Node.js. To me, it was a trend more than a
real tool to solve problems JavaScript in the server? That didn't look right. In all
fairness, I didn't even like JavaScript. Then, the modern frameworks such as jQuery
or Angular.js came to the rescue. They solved one of the problems, which was the
cross-browser compatibility. Where before we needed to factor in at least three
different browsers, after jQuery all this logic was nicely encapsulated in a library
so that we didn't need to worry about compatibility as long as we followed the
jQuery documentation.
Then, JavaScript became more popular. Suddenly, all the internal tools were written
with Single-Page Application (SPA) frameworks with a heavy usage of JavaScript,
therefore, the majority of developers nowadays, one way or another, are proficient
in JavaScript.
[ 20 ]
Chapter 1
Then, someone decided to take JavaScript out of the browser, which was a great
idea. Rhino, Node.js, and Nashorn are examples of runtimes that can execute
standalone JavaScript. Some of them can even interact with the Java code, enabling
the developer to import Java classes into a JavaScript program, which gives you the
access to an endless set of frameworks already written in Java.
Let's focus on Node.js. Node.js is the perfect candidate for microservices-oriented
architectures for a number of reasons, as stated in the following list:
Easy to scale
Highly testable
Easy to deploy
These reasons, along with others that we will develop in the following chapters,
make Node.js the perfect candidate for building solid microservices.
API aggregation
Seneca is the framework that I have chosen for development in the following
chapters. One of the most attractive characteristics of Seneca is API aggregation.
API aggregation is an advanced technique to compose an interface by aggregating
different functionalities (plugins, methods, and so on) to it.
Let's take a look at the following example:
var express = require('express');
var app = express();
app.get('/sayhello', function (req, res) {
res.send('Hello World!');
});
app.get('/saygoodbye', function(req, res) {
res.send('Bye bye!');
});
var server = app.listen(3000, function () {
var host = server.address().address;
var port = server.address().port;
console.log('App listening at http://%s:%s', host, port);
});
[ 21 ]
Microservices Architecture
The preceding example uses Express, a very popular web framework for Node.js.
This framework is also built around the API aggregation technique. Let's take a look
at the fourth and seventh lines. In these lines, the developer registers two methods
that are to be executed when someone hits the URLs /sayhello and /saygoodbye
with a GET request. In other words, the application is composed of different smaller
and independent implementations that are exposed to the outer world on a single
interface, in this case, an app listening on the 3000 port.
In the following chapters, I will explain why this property is important and how to
take advantage of it when building (and scaling) microservices.
Chapter 1
Summary
In this chapter, we studied the key concepts around microservices, as well as the best
practices to be followed when designing high-quality software components towards
building robust and resilient software architectures that enable us to respond quickly
to the business needs.
You have also learned the key benefits such as the possibility of using the right
language for the right service (technology heterogeneity) on the microservices-oriented
architectures as well as some of the pitfalls that could make our life harder, such as
the overhead on the operational side caused by the distributed nature of the
microservices-oriented architectures.
Finally, we discussed why Node.js is a great tool for building microservices, as well
as how we could benefit from JavaScript to build high-quality software components
through techniques like API aggregation.
In the following chapters, we will be developing the concepts discussed in this
chapter, with code examples and further explanation about the topics I have
learned over the years.
As explained before, we will focus on the V8 version of JavaScript, but I will
provide some hints on how to easily write upgradeable components to embrace
ECMAScript 6.
[ 23 ]
www.PacktPub.com
Stay Connected: